A Polytomous Extension of the Generalized Distance Discriminating Method

Abstract

This article proposes a generalized distance discriminating method for test with polytomous response (GDD-P). The new method is the polytomous extension of an item response theory (IRT)-based cognitive diagnostic method, which can identify examinees’ ideal response patterns (IRPs) based on a generalized distance index. The similarities between observed response patterns and IRPs for polytomous response situation are measured by the index of GDD-P, and the attribute patterns can be recognized via the relationship between attribute patterns and IRPs. Feasible designs about polytomous Q-matrix and scoring items for polytomous response are also discussed. In simulation, the classification accuracy of the GDD-P method for the test with polytomous response was investigated, and results indicated that the proposed method had promising performance in recognizing examinees’ attribute patterns.

Keywords

IRT-based cognitive diagnosis generalized distance discrimination for polytomous response polytomous Q-matrix item scoring method classification accuracy

Cognitive diagnosis assessment aims at providing valuable information about examinees’ cognitive strengths and weaknesses at the skill, attribute, or competence level. The introduction of the Q-matrix (Tatsuoka, 1983) to a test has made it possible to establish the relationship between test items and attributes. An examinee’s latent information reflected by an attribute pattern can be inferred through a pattern recognition procedure based on the Q-matrix. Although dozens of cognitive diagnostic models (CDMs) or approaches have been proposed during the last 30 years, most of them are developed for dichotomous response. However, in practical applications, tests often include polytomous items (i.e., items that are scored polytomously). If researchers use existent CDMs or methods for dichotomous response to analyze such test data, information might be lost due to the process of reducing polytomous to dichotomous scores. To address this particular testing need, it is necessary for researchers to develop and use cognitive diagnostic methods for polytomous response.

Although many CDMs or approaches have been proposed and some of them have been summarized from different aspects (e.g., de la Torre, 2011; Embretson & Reise, 2000; Fu & Li, 2007; Haertel, 1989; Maris, 1999; Nichols, Chipman, & Brennan, 1995), it is difficult to determine which method provides the most reliable diagnostic information for examinees. A cognitive diagnostic method for practical application should diagnose an examinee’s attribute pattern with relatively high precision. The accuracy of examinees’ pattern recognition can therefore be considered as criteria for measuring the precision of a CDM, especially for a new proposed approach. Regarding this, the generalized distance discriminating (GDD) method for dichotomous response (Sun, Zhang, Xin, & Bao, 2011) has been shown to have good performance using simulation data. The method constructs a discriminating index based on item response theory (IRT) to recognize examinees’ attribute patterns from their dichotomous responses. Specifically, the GDD method as an IRT-based method offers a new way of doing cognitive diagnosis. For CDMs, which are widely used for cognitive diagnosis, attribute patterns can be estimated based on a specific item response function that reflects a direct relationship between the attribute patterns and the item responses. For IRT-based procedures such as the Rule Space Method (RSM; Tatsuoka, 1983, 1995), Attribute Hierarchy Method (AHM; Leighton, Gierl, & Hunka, 2004) and GDD, attribute patterns are not estimated directly. The ideal response patterns (IRPs) are identified from the observed response patterns (ORPs) using an IRT-based discriminating index; thereafter, the attribute patterns can be determined by examining the corresponding relationship between the IRPs and the attribute patterns as indicated by the Q-matrix.

Given the relatively high classification accuracy of the GDD method, this study extends the method to make it appropriate for polytomous response. The development of the new method denoted as GDD-P, where P stands for polytomous response, is given in the next section. The design of the Q-matrix with polytomous elements and an item scoring method for polytomous response are discussed in the section titled “Q-matrix Design and Scoring Method.” Although Fu (2005) proposed a method for scoring polytomous items in the context of the polytomous extension of the fusion model, this is by no means the only possible approach to scoring polytomous items. In this work, the authors define an item scoring method for describing the correspondence relationship between the attribute patterns and IRPs for both the dichotomous Q-matrix and polytomous Q-matrix. Following the specific design of Q-matrix with polytomous elements and the scoring method, the submatrix of the polytomous Q-matrix for guaranteeing a one-to-one mapping between the attribute patterns and the IRPs is also discussed. Based on the theoretical part of the GDD-P method and the designs related with Q-matrix and item scoring, three simulation studies are conducted in the section titled “Simulation.” In the “A Real Data Analysis” section, the authors provide a real data analysis by implementing the GDD-P method using dichotomous Q-matrix and polytomous Q-matrix. They conclude the article in the last section by discussing the practical implications of the proposed method.

GDD Method for Polytomous Response

GDD Method

Sun et al. (2011) proposed the GDD method to identify each examinee’s IRP for a cognitive diagnostic test with dichotomous items. Attribute patterns of these examinees can be recognized based on the relationship between IRPs and attribute patterns, which is determined by the Q-matrix (Tatsuoka, 1983) of the test with an assumption that the underlying process is conjunctive (e.g., Fu & Li, 2007). The core idea of the method is to use a generalized distance index to identify each examinee’s IRP. The index is based on the item response function of an IRT model. The distance definition generalizes the Hamming distance by adding the response probabilities of examinees as adjustment factors. Below is an overview of the generalized distance index for dichotomous response.

Assume N examinees respond to a cognitive diagnostic test, and the total number of IRPs inferred from the Q-matrix is T. Let vector $Y_{i} = (Y_{i 1}, \dots, Y_{iJ})$ and $I_{t} = (I_{1}^{(t)}, \dots, I_{J}^{(t)})$ denote examinee i’s ORP and tth IRP (i = 1, . . . , N; t = 1, . . . , T), respectively. The item response for every item of the test is either 0 or 1. A generalized distance index for item j, which measures the similarity between $Y_{ij}$ and $I_{j}^{(t)}$ , is defined as

d (Y_{i j}, I_{j}^{(t)}) = {\begin{matrix} 0, i f Y_{i j} = I_{j}^{(t)} \\ 1 - P_{j_{0}} (θ_{i}), i f Y_{i j} = 1_{} a n d_{} I_{j}^{(t)} = 0 \\ 1 - P_{j_{1}} (θ_{i}), i f Y_{i j} = 0_{} a n d_{} I_{j}^{(t)} = 1 \end{matrix},

where $P_{j_{0}} (θ_{i})$ and $P_{j_{1}} (θ_{i})$ can be specified as an item response function of any IRT model appropriate for dichotomous response. The idea of the definition can be explained as follows:

Case 1. If $Y_{ij} = I_{j}^{(t)}$ , $d (Y_{ij}, I_{j}^{(t)})$ is equal to zero, which indicates a perfect match between the ideal and observed response.

Case 2. If $Y_{ij} = 1$ and $I_{j}^{(t)} = 0$ , then $d (Y_{ij}, I_{j}^{(t)}) = 1 - P_{j_{0}} (θ_{i})$ is the probability of correctly answering item j with the ability θ_i. To illustrate its meaning, assume two examinees, who both got the correct answer on item j, have different abilities: θ_i > θ_l. Because examinee i has a higher probability of getting the correct answer (i.e., $P_{j_{1}} (θ_{i}) > P_{j_{1}} (θ_{l}) \Leftrightarrow P_{j_{0}} (θ_{i}) < P_{j_{0}} (θ_{l})$ ) than examinee l, examinee i is less likely to have gotten the answer by guessing than examinee l. Therefore, the difference between the two examinees is reflected by $d (Y_{ij}, I_{j}^{(t)}) > d (Y_{lj}, I_{j}^{(t)})$ .

Case 3. If $Y_{ij} = 0$ and $I_{j}^{(t)} = 1$ , then $1 - P_{j_{1}} (θ_{i}) = P_{j_{0}} (θ_{i})$ . Similar to the previous example, if θ_i > θ_l holds, examinee i is more likely to have gotten the wrong answer by slipping than examinee l, and therefore $d (Y_{ij}, I_{j}^{(t)}) < d (Y_{lj}, I_{j}^{(t)})$ .

The generalized distance between the vectors $Y_{i}$ and $I_{t}$ is defined as

d (Y_{i}, I_{t}) = \sum_{j = 1}^{J} d (Y_{ij}, I_{j}^{(t)}) .

It represents the sum of the generalized distances across the J items. Therefore, each examinee’s latent IRP can be identified by calculating all the T generalized distance indices and following the rule $min_{t = 1, \dots, T .} {d (Y_{i}, I_{t})}$ . In practical application, the ability parameter $θ_{i}$ of examinee i in Equation 1 should be replaced with its estimate ${\hat{θ}}_{i}$ based on the examinee’s ORP.

In Sun et al.’s (2011) work, a Monte Carlo simulation study was conducted to compare the examinee classification accuracy of three cognitive diagnosis methods: GDD, RSM, and AHM. The latter two methods are chosen for the comparison because they are also both IRT-based methods. The simulation data were generated from the DINA (deterministic inputs, noisy “and” gate) model (Junker & Sijtsma, 2001), which was viewed as the baseline to check the performance among GDD, RSM, and AHM. Sixteen conditions representing the combinations of four kinds of item slippage probabilities and four types of attribute hierarchies (Leighton et al., 2004) were considered. The item parameters of both the DINA and unidimensional IRT models were estimated using the expectation-maximization (EM) algorithm (e.g., de la Torre, 2009; Woodruff & Hanson, 1996). The results showed that GDD and DINA models performed almost equally with respect to classification accuracy, and both performed better than the RSM and AHM.

Upon closer examination, it is found that the GDD method and DINA model share a similarity in how examinee classification is carried out. As shown in Equation 1, if $I_{j}^{(t)} = 0$ and $Y_{ij} = 1$ , $1 - P_{j_{0}} (θ_{i}) = P_{j_{1}} (θ_{i})$ is the guessing probability of examinee i on item j; if $I_{j}^{(t)} = 1$ and $Y_{ij} = 0$ , $1 - P_{j_{1}} (θ_{i}) = P_{j_{0}} (θ_{i})$ is the slipping probability of examinee i on item j. So minimizing Equation 2 is equivalent to minimizing the summation of some guessing and slipping probabilities. As for DINA model, g_j and s_j usually represent the guessing and slipping probabilities of item j. If $I_{j}^{(t)} = 0$ , the probability of $Y_{ij} = 0$ is equal to 1−g_j; if $I_{j}^{(t)} = 1$ , the probability of $Y_{ij} = 1$ is equal to 1−s_j. After the item parameters of the DINA model are estimated, the attribute pattern of an examinee can be identified by using the MAP (maximum a posterior) method. The posterior probability of an attribute pattern is the product of g_j, s_j, (1 −g_j), or (1 − s_j) for j = 1, . . . , J if the prior probabilities of all the attribute patterns are identical. Because of the constraints, g_j < 1 − s_j for j = 1, . . . , J, the attribute pattern of an examinee should be estimated as the one that has the largest product of (1−g_j) or (1−s_j) for all the items, which is similar to minimizing Equation 2. This may explain why the GDD method performs similarly to the DINA model in identifying examinees’ attribute patterns.

Definition of Generalized Distance Index for Polytomous Response

In the polytomous context, the symbols $Y_{i}$ and $I_{t}$ are still used to represent examinee i’s ORP and tth IRP, respectively. For a polytomous item j, the possible scores are denoted as 0, 1, . . . , m_j, where m_j represents the maximum score on item j. A generalized distance for polytomous response for item j, which measures the similarity between $Y_{ij}$ and $I_{j}^{(t)}$ , can be defined as

d (Y_{ij}, I_{j}^{(t)}) = | Y_{ij} - I_{j}^{(t)} | \times (1 - P_{I_{j}^{(t)}} (θ_{i})),

where $P_{I_{j}^{(t)}} (θ_{i})$ represents the probability of getting a response of $I_{j}^{(t)}$ on item j for examinee i with ability θ_i. The form of $P_{I_{j}^{(t)}} (θ_{i})$ can be specified by an IRT model (e.g., two-parameter logistic model [2PLM], generalized partial credit model [GPCM]). The generalized distance for polytomous response for item j, $d (Y_{ij}, I_{j}^{(t)})$ , can be computed as follows. If examinee i’s observed response Y_ij is equal to the ideal response for the same item (i.e., $I_{j}^{(t)}$ ), then let $d (Y_{ij}, I_{j}^{(t)}) = 0$ . Otherwise, calculate a rough measure of the similarity between two responses Y_ij and I_j^(t), $| Y_{ij} - I_{j}^{(t)} |$ . Next, calculate $P_{I_{j}^{(t)}} (θ_{i})$ to adjust the similarity measure in Step 2. Hence, $| Y_{ij} - I_{j}^{(t)} | \times (1 - P_{I_{j}^{(t)}} (θ_{i}))$ takes into consideration as an adjustment of the hamming distance to reflect both the difference between Y_ij and I_j^(t) and the effect of examinee i’s latent ability θ_i. $P_{I_{j}^{(t)}} (θ_{i})$ is calculated and used to multiply the hamming distance.

For the polytomous response situation, whenever Y_ij≠I_j^(t), Equation 3 represents the distance between the ideal and observed responses weighted by the “unlikeliness” of the ideal response given the examinee’s ability. $P_{I_{j}^{(t)}} (θ_{i})$ should be specified as a polytomous IRT model (e.g., Samejima’s, 1969, graded response model [GRM]). The ability parameter θ_i of examinee i in Equation 3 can still be replaced by its estimate ${\hat{θ}}_{i}$ based on the examinee’s ORP. The generalized distance between Y_i and I_t, $d (Y_{i}, I_{t})$ , is defined as in Equation 2. Also, $min_{t = 1, \dots, T .} {d (Y_{i}, I_{t})}$ is used as the rule to identify examinee i’s IRP for the polytomous response situation. The method introduced in this section is called the GDD-P method, where P represents the polytomous nature of the items. Following the definition of Equation 1, the GDD method can be treated as a special case of the GDD-P method.

In comparing the IRT-based methods for cognitive diagnosis (i.e., RSM, AHM, GDD, and GDD-P), it can be found that these methods are all based on the match or lack thereof between the ORPs and IRPs. Attribute patterns can be recognized by these methods in two steps. First, an examinee’s IRP is identified by applying an IRT-based discriminating index. Second, the attribute pattern behind the IRP is determined based on the information contained in the Q-matrix. Note that the form of Q-matrix determines the form of attribute patterns and the item scoring method determines how the attribute patterns correspond to the IRPs. In other words, if one would like to use an IRT-based method like GDD-P, the specific attribute patterns and how they map to the IRPs should be known. This implies that the design of the Q-matrix and the item scoring method play a critical role in the pattern recognition procedure of an IRT-based method. For this reason, some feasible Q-matrix designs and scoring item methods are discussed in the next section.

Q-matrix Design and Scoring Method

Design of Q-Matrix

A polytomous Q-matrix is defined as a Q-matrix that can assume nonnegative integers as its elements. To differentiate Tatsuoka’s (1983) Q-matrix, which can only contain dichotomous values, from the polytomous Q-matrix, the authors refer to the former as the dichotomous Q-matrix in this article. Note that the dichotomous Q-matrix could not sufficiently reflect the different difficulty levels of an attribute in different items, whereas the polytomous Q-matrix can. A test for probing a comprehensive problem-solving process often requires that the grain size of an attribute should not be too small. Therefore, it is necessary to design the polytomous Q-matrix to characterize the large grain sizes of the attributes. Table 1 is as an example of the polytomous Q-matrix. The test contains 10 items and three independent attributes, A1, A2, and A3. The attributes A2 and A3 are probed to different extents. Because the maximum elements corresponding to A1, A2, and A3 in Q-matrix are 1, 2, and 3, respectively, there is a total number of 24 attribute patterns (i.e., 2 × 3 × 4 = 24; see Table 2). The number indicates some examinees could possess the same attributes but to different extents.

Table 1.

Polytomous Q-Matrix for the Test With Three Attributes.

Attribute	Item
	1	2	3	4	5	6	7	8	9	10
A1	1	0	0	0	0	0	1	1	0	1
A2	0	1	2	0	0	0	0	2	2	2
A3	0	0	0	1	2	3	2	0	2	3

Table 2.

Attribute Patterns for the Test With Three Attributes.

Attribute	Attribute pattern
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
A1	0	1	0	0	0	0	0	1	1	1	1	1	0	0	0	0	0	0	1	1	1	1	1	1
A2	0	0	1	2	0	0	0	1	2	0	0	0	1	2	1	2	1	2	1	2	1	2	1	2
A3	0	0	0	0	1	2	3	0	0	1	2	3	1	1	2	2	3	3	1	1	2	2	3	3

Item Scoring Method

A feasible item scoring method is introduced here to show how attribute patterns are mapped to the IRPs for a polytomous response test. The scoring method can work with both the dichotomous and the polytomous Q-matrices. Let vector $α = (α_{1}, \dots, α_{K})$ denote an attribute pattern with K attributes. If $Q = {q_{k j}}_{K \times J}$ with J items is designed to be polytomous, an ideal response on item j of the pattern α can be designed as

S_{j} (α) = \sum_{k = 1}^{K} q_{k j} \times I_{{α_{k} \geq q_{k j}}},

where I represents an indicator function.

When a dichotomous Q-matrix is used, $S_{j} (α)$ is equal to the number of attributes mastered by α among the attributes being probed by item j. For instance, if a test probes three attributes and an item requires all the three attributes, as in, q _j = (1, 1, 1), an ideal score on this item should be 0, 1, 2, or 3. Furthermore, if the three attributes are independent, the attribute patterns corresponding to the ideal score of 2 should be α= (1, 1, 0), α = (1, 0, 1) or α= (0, 1, 1). If the three attributes have the divergent hierarchy of A1 → A2 and A1 → A3, it can be assumed that the prerequisite relationships are such that an examinee’s nonmastery of the basic attribute, A1, will lead to his nonmastery of any other advanced attributes, A2 and A3. In this hierarchy, neither A2 nor A3 can be mastered unless A1 is first mastered. This means the attribute pattern corresponding to an ideal score of 2 could be either α = (1, 1, 0) or α = (1, 0, 1), and cannot be α = (0, 1, 1). When a polytomous Q-matrix such as that in Table 1 is used, an ideal score for Item 10 is 0, 1, 2, 3, 4, 5, or 6 (see Table 3). If the hierarchy A1 → A2 → A3 exists among the attributes and the mastery of more advanced attributes presupposes the mastery of more basic attributes, the total number of attribute patterns would decrease from 24 to 10. The ideal scores for Item 10 become 0, 1, 3, or 6. Note that in practice, the GDD-P method can be successfully applied with other feasible item scoring methods if the mapping from the attribute patterns to the IRPs is determined explicitly.

Table 3.

Ideal Response Patterns for the Test With Three Attributes.

	Ideal response pattern
Item	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
1	0	1	0	0	0	0	0	1	1	1	1	1	0	0	0	0	0	0	1	1	1	1	1	1
2	0	0	1	1	0	0	0	1	1	0	0	0	1	1	1	1	1	1	1	1	1	1	1	1
3	0	0	0	2	0	0	0	0	2	0	0	0	0	2	0	2	0	2	0	2	0	2	0	2
4	0	0	0	0	1	1	1	0	0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
5	0	0	0	0	0	2	2	0	0	0	2	2	0	0	2	2	2	2	0	0	2	2	3	3
6	0	0	0	0	0	0	3	0	0	0	0	3	0	0	0	0	3	3	0	0	0	0	3	3
7	0	1	0	0	0	2	2	1	1	1	3	3	0	0	2	2	2	2	1	1	3	3	3	3
8	0	1	0	2	0	0	0	1	3	1	1	1	0	2	0	2	0	2	1	3	1	3	1	3
9	0	0	0	2	0	2	2	0	2	0	2	0	0	2	2	4	0	4	0	2	2	4	2	4
10	0	1	0	2	0	0	3	1	3	1	1	4	0	2	0	2	3	5	1	3	1	3	4	6

Note: The order of 24 ideal response patterns listed here is consistent with that of 24 attribute patterns in Table 2.

Sufficiency of Polytomous Q-Matrix

A critical issue about the polytomous Q-matrix and item scoring method introduced is ensuring under which conditions the attribute patterns under those designs are identifiable. The identifiability of attribute patterns for the dichotomous Q-matrix and conjunctive item scoring method has been explored. Chiu, Douglas, and Li (2009) proved that if the attributes are all independent for dichotomous Q-matrix, the IRPs for different attribute patterns are different if all the single-attribute items are included in the Q-matrix. Ding, Yang, and Wang (2010) pointed out that for dichotomous Q-matrix with a strong attribute prerequisite relationship (Leighton et al., 2004), one-to-one mapping exists between attribute patterns and IRPs if and only if the dichotomous Q-matrix contains the reachability matrix (e.g., Tatsuoka, 1995) as its submatrix. Based on these results, for the polytomous Q-matrix with the item scoring method introduced above, a matrix denoted by R (defined in the following) can be used as the submatrix of the polytomous Q-matrix to guarantee a one-to-one mapping between the attribute patterns and the IRPs.

Matrix R for Polytomous Q-Matrix

If an attribute hierarchy of a test is defined by domain experts and the test is designed to probe different levels of the attributes, the matrix R for constituting the polytomous Q-matrix can be generated following these steps. First, initially consider that every attribute takes only dichotomous values (i.e., 0 or 1), and obtain the reachability matrix, which is denoted by R’, for a specific attribute hierarchy. Second, denote the value representing the highest level of attribute k as w_k (k = 1, . . . , K) and replicate the patterns which probe this attribute in R’ for w_k times. Then, replace the kth element of these patterns with 1, . . . ,w_k. The new patterns are used to compose the columns of matrix R. This procedure should be applied to every attribute to produce R. For instance, as shown in Table 1, if three independent attributes in a test, A1, A2, and A3, are designed to be probed at one, two, and three levels, respectively (i.e., in an attribute pattern, the value of A1 is 0 or 1; that of A2 is 0, 1, or 2; and that of A3 is 0, 1, 2, or 3), the submatrix composed of the first six columns of the polytomous Q-matrix given in Table 1 is the matrix R. However, the preceding matrix R generated is not unique in guaranteeing the identification of the attribute patterns from polytomous Q-matrix. As such, the authors introduced another method of generating other submatrices to play the same role as R. To differentiate from the matrix R, the new submatrix is denoted by G_R.

Matrix G_R for Polytomous Q-Matrix

The method for generating G_R could be based on matrix R and follows three steps. First, find two columns of matrix R, v₁ and v₂, such that the elements of the two vectors for every attribute are equal or have a minimum value of zero. Second, a new vector v₃ is obtained by taking the maximum value of v₁ and v₂ for every attribute. Third, G_R can be generated by replacing v₁ or v₂ of matrix R with v₃. Different vector selections in the first step will generate different G_R. Following the example above, if vectors v₁ = (1, 0, 0) and v₂ = (0, 2, 0) of matrix R shown in Table 4 are used, a new vector v₃ = (1, 2, 0) can be used to replace either v₁ or v₂ of R to generate two different G_R, as shown in Table 4.

Table 4.

Matrix R and Possible G_Rs for the Three-Attribute Test.

		Item
	Attribute	1	2	3	4	5	6
Matrix R	A1	1	0	0	0	0	0
	A2	0	1	2	0	0	0
	A3	0	0	0	1	2	3
Matrix G_R,1	A1	1	0	1	0	0	0
	A2	0	1	2	0	0	0
	A3	0	0	0	1	2	3
Matrix G_R,2	A1	1	0	0	0	0	0
	A2	2	1	2	0	0	0
	A3	0	0	0	1	2	3

Simulation

In this section, three studies were conducted to investigate the classification performance of the GDD-P method for polytomous items under the scoring method introduced in the previous section. Studies 1 and 2 focused on the condition that strong prerequisite structures existed among the attributes. Study 3 focused on the condition that the attribute dependence and distribution of attribute patterns followed the rule suggested by de la Torre and Douglas (2004). The correct pattern classification rate (CPCR) and average attribute match rate (AAMR) were used as the evaluation criteria for the GDD-P method for these studies. These statistics are defined as

CPCR = \frac{1}{N} \sum_{i = 1}^{N} I_{{α_{i} = {\tilde{α}}_{i}}}

and

AAMR = \frac{1}{K \times N} \sum_{i = 1}^{N} \sum_{k = 1}^{K} I_{{α_{ik} = {\tilde{α}}_{ik}}},

where N is the sample size, K is number of attributes, $α_{i} = (α_{i 1}, \dots, α_{iK})$ and ${\tilde{α}}_{i} = ({\tilde{α}}_{i 1}, \dots, {\tilde{α}}_{iK})$ represent examinee i’s estimated attribute pattern and true attribute pattern, respectively. The computer code for the simulation study was written in R language.

Study 1

Condition 1: Data From Dichotomous Q-Matrix and Four Attribute Structures

In this study, K = 7 attributes were assumed and the attributes followed four structures. As shown by Figure 1, the attributes under the first three structures—linear, convergent, and divergent (Leighton et al., 2004)—were dependent; the attributes under the last structure were independent. For the first three structures, an assumption that is same as that in the examples in the “Item Scoring Method” section is set: An attribute could be mastered only if all of its prerequisite attributes are also mastered. The Q-matrix for each structure was designed to be composed of J = 35 items. Under each structure, the reachability matrix (see Table 5) was used as a submatrix in a dichotomous Q-matrix and the remaining 28 columns were randomly sampled from all possible attribute patterns except the all-zero pattern.

Figure 1.

Attribute structures for Study 1.

Table 5.

Reachability Matrix for Study 1.

	Item
Attribute	1	2	3	4	5	6	7
Linear
A1	1	1	1	1	1	1	1
A2	0	1	1	1	1	1	1
A3	0	0	1	1	1	1	1
A4	0	0	0	1	1	1	1
A5	0	0	0	0	1	1	1
A6	0	0	0	0	0	1	1
A7	0	0	0	0	0	0	1
Independent
A1	1	0	0	0	0	0	0
A2	0	1	0	0	0	0	0
A3	0	0	1	0	0	0	0
A4	0	0	0	1	0	0	0
A5	0	0	0	0	1	0	0
A6	0	0	0	0	0	1	0
A7	0	0	0	0	0	0	1
Convergent
A1	1	0	1	0	0	0	1
A2	0	1	1	0	0	0	1
A3	0	0	1	0	0	0	1
A4	0	0	0	1	0	1	1
A5	0	0	0	0	1	1	1
A6	0	0	0	0	0	1	1
A7	0	0	0	0	0	0	1
Divergent
A1	1	1	1	1	1	1	1
A2	0	1	0	1	1	0	0
A3	0	0	1	0	0	1	1
A4	0	0	0	1	0	0	0
A5	0	0	0	0	1	0	0
A6	0	0	0	0	0	1	0
A7	0	0	0	0	0	0	1

To generate the true attribute patterns of N = 1,000 examinees, the number of examinees possessing each attribute pattern can be determined first. As different attribute patterns correspond to different IRPs, the total scores of IRPs were used to design the frequencies of those examinees’ true attribute patterns. Specifically, by anchoring zero and the full score to the ability values of −3 and 3, respectively, the total score of IRPs were calculated, sorted, and transformed to some ability values. If these ability values are assumed to approximately follow a standard normal distribution, the number of examinees possessing every attribute pattern can be obtained. These true attribute patterns were later used to calculate the CPCR and AAMR of the GDD-P method.

Next, the rule illustrated in the Appendix is used to generate the data (i.e., ORPs) for polytomous response. The main principle of the rule is that small difference between an alternative score and an ideal score will lead to a big likelihood of observing this alternative score. Then, a random number sampled from a uniform distribution U(0, 1) was generated to simulate the examinees’ ORPs. Under each attribute hierarchy, slippage probability was designed to be 10%, 20%, 30%, or 40%. For each of the 16 combinations from four slippage probabilities and four attribute structures, 30 data sets are simulated and analyzed. The GDD-P method was used by fitting the response data with the polytomous IRT model, GRM, and used EM algorithm and EAP (expected a posterior) to estimate the item and ability parameters, respectively.

Results

The classification accuracy results were reported in Table 6. As Table 6 shows, the CPCR values of the GDD-P method under each hierarchy for the 10% slippage probability were all above 0.99; for the 20% slippage probability, the values were all higher than 0.97; for the 30% slippage probability, the values decreased to about 0.89; and for the 40% slippage probability, the lowest CPCR value was higher than 0.68. The AAMR values of the GDD-P method for the slippage probability of 10%, 20%, 30%, and 40% under four hierarchies were all above 0.90.

Table 6.

Mean CPCR and AAMR for the GDD-P method in Study 1.

		Classification accuracy
Attribute hierarchy	Slippage probability (%)	Mean CPCR	Mean AAMR
Linear	10	0.9986	0.9998
	20	0.9921	0.9987
	30	0.9751	0.9956
	40	0.9288	0.9868
Convergent	10	0.9916	0.9987
	20	0.9739	0.9951
	30	0.9188	0.9821
	40	0.7724	0.9449
Divergent	10	0.9956	0.9993
	20	0.9797	0.9965
	30	0.9319	0.9878
	40	0.8230	0.9657
Independent	10	0.9989	0.9998
	20	0.9816	0.9955
	30	0.8943	0.9708
	40	0.6886	0.9062

Note: CPCR = correct pattern classification rate; AAMR = average attribute match rate; GDD-P=GDD method for polytomous response.

In comparing the CPCR results under different attribute hierarchies given in Table 6, it can be found that under each slippage probability, the values for linear hierarchy were almost always higher than those for the other three structures. In addition, the CPCR values under the 10% and 20% slippage probabilities were slightly higher for independent structures than those for divergent and convergent hierarchy; for the 30% and 40% slippage probabilities, the CPCR values were highest for the divergent structure, followed by the convergent and then the independent structures. The AAMR values given in Table 6 showed similar trends as the CPCR values.

Study 2

Condition 2: Data From Polytomous Q-Matrix and Four Attribute Structures

This study differed from Study 1 in that the Q-matrix had polytomous elements. Although involving only K = 3 attributes, the four attribute structures in Figure 2 were designed as same as those in Study 1. The element of the Q-matrix was either 0 or 1 for A1; 0, 1, or 2 for A2; and 0, 1, 2, 3, or 4 for A3. For each attribute structure, the Q-matrix had J = 35 items and contained the matrix R given in Table 7 as a submatrix. The remaining columns were randomly sampled from all possible attribute patterns except the all-zero pattern. The remaining aspects of Study 2 (i.e., simulation method of ORPs, slippage probabilities, sample size, and number of replications) were identical to those in Study 1.

Table 7.

Matrix R for Study 2.

	Item
Attribute	1	2	3	4	5	6	7	8	9	10	11
Linear
A1	1	1	1	1	1	1	1	1	1	1	1
A2	0	1	2	1	1	1	1	2	2	2	2
A3	0	0	0	1	2	3	4	1	2	3	4
Independent
A1	1	0	0	0	0	0	0
A2	0	1	2	0	0	0	0
A3	0	0	0	1	2	3	4
Convergent
A1	1	0	0	1	1	1	1	1	1	1	1
A2	0	1	2	1	1	1	1	2	2	2	2
A3	0	0	0	1	2	3	4	1	2	3	4
Divergent
A1	1	1	1	1	1	1	1
A2	0	1	2	0	0	0	0
A3	0	0	0	1	2	3	4

Figure 2.

Attribute structures for Study 2.

Results

The classification accuracy results of Study 2 are reported in Table 8. In this study, the CPCR values of the GDD-P method under each hierarchy for the 10% slippage probability were all above 0.98; for the 20% slippage probability, the values were all higher than 0.94; for the 30% slippage probability, the values decreased to no lower than 0.82; for the 40% slippage probability, the lowest CPCR value was around 0.60. The AAMR values of the GDD-P method for the 10%, 20%, 30%, and 40% slippage probabilities under the four hierarchies were all above 0.87. Due to the more stringent definition of CPCR, the CPCR values showed in Table 8 were consistently lower than the corresponding AAMR values.

Table 8.

Mean CPCR and AAMR for the GDD-P Method in Study 2.

		Classification accuracy
Attribute hierarchy	Slippage probability (%)	Mean CPCR	Mean AAMR
Linear	10	0.9918	0.9988
	20	0.9696	0.9952
	30	0.9163	0.9853
	40	0.8360	0.9687
Convergent	10	0.9922	0.9984
	20	0.9672	0.9926
	30	0.8913	0.9729
	40	0.7753	0.9393
Divergent	10	0.9856	0.9978
	20	0.9501	0.9920
	30	0.8688	0.9758
	40	0.7336	0.9428
Independent	10	0.9886	0.9982
	20	0.9437	0.9887
	30	0.8222	0.9545
	40	0.6083	0.8760

Note: CPCR = correct pattern classification rate; AAMR = average attribute match rate; GDD-P=GDD method for polytomous response.

In comparing the CPCR results under different attribute hierarchies, it was found that for each slippage probability, the values for linear and convergent hierarchies were all higher than those for divergent and independent structures. This trend became clearer as the slippage probability increased from 10% to 40%. The AAMR values had a similar trend as the CPCR values. Note that other conditions that represented different numbers of K and J have been considered in simulation but the results were generally similar to the results included in the preceding studies.

By summarizing the results for the two studies, it should be noted that the polytomous Q-matrix design is helpful but not necessary to any polytomous-score test. The choice between dichotomous Q-matrix and polytomous Q-matrix could closely depend on the number of attributes in a test. When the polytomous Q-matrix was designed to analyze the polytomous scores, the attribute number and the levels of the attributes should be designed to be relatively small. Otherwise, the R matrix for guaranteeing the identification of attributes may include too many items such that the resulting test length may not be practicable.

If the attribute patterns are only partially ordered, which might render the sum score not a good statistic to model the relationship between the ability and an attribute pattern, the distribution of the attribute patterns in the first two studies may be not practical. With this limitation in mind, the third study was designed.

Study 3

Condition 3: Data From Dichotomous Q-Matrix and Higher Order Latent Trait Models

Here, the stringent assumption about the attribute structures in the first two studies was relaxed so that the attribute pattern number could be 2^K− 1 even if some attributes are correlated. In addition, different from the attribute pattern generation method in the first two studies, the idea of the higher order latent trait models for cognitive diagnosis (de la Torre & Douglas, 2004) was implemented to generate the true attribute patterns of N = 5,000 examinees. In this study, the attribute number was fixed to K = 5, item number J = 31, and the dichotomous Q-matrix composed of all the attribute patterns except for the all-zero pattern was used. The unidimensional trait θ was assumed to follow the standard normal distribution. The probability of an attribute pattern α given θ was $P (α | θ) = \prod_{k = 1}^{K} P (α_{k} | θ)$ , where $P (α_{k} = 1 | θ) = {(1 + \exp (- β_{k} θ - γ_{k}))}^{- 1}$ , and the item parameters β_k and γ_k (k = 1, . . . , 5) were assumed to follow normal distributions N(0,1) and log-normal (0,0.5), respectively. Similar to the first two studies, the observed response data of the examinees were generated following the rule described in the Appendix. The slippage probabilities were kept at 10%, 20%, 30%, and 40%. For each slippage probability, 30 data sets were simulated and analyzed.

Results

As shown in Table 9, the CPCR values of the GDD-P method for the 10% and 20% slippage probabilities were all above 0.99. The CPCR values of the method for the 30% and 40% slippage probabilities were higher than 0.96 and equal to 0.84, respectively. AAMR values of the method for the 10%, 20%, 30%, and 40% slippage probabilities were all higher than 0.95.

Table 9.

Mean CPCR and AAMR for the GDD-P Method in Study 3.

	Classification accuracy
Slippage probability (%)	Mean CPCR	Mean AAMR
10	0.9994	0.9998
20	0.9952	0.9988
30	0.9669	0.9919
40	0.8400	0.9579

Note: CPCR = correct pattern classification rate; AAMR = average attribute match rate; GDD-P=GDD method for polytomous response.

A Real Data Analysis

Data Description

In this section, real data were chosen as an example to illustrate the use of the GDD-P method for diagnosing the attribute patterns of the examinees. The test was a science booklet within PISA (The Program for International Student Assessment) administered in 2006 and contained 22 items. The data were composed of the responses of 1,779 students from mainland China, Macao, Hong Kong, and Taipei. According to PISA 2006 science assessment framework, three competencies (i.e., attributes), namely, identifying scientific issues, explaining phenomena scientifically, and using scientific evidence, were probed in the test. Although the focus of PISA is not in individual but aggregated scores, cognitive diagnosis analysis of a low-stakes test was used here as an example of how additional individual-level information can be obtained when some items in the test represent questions from the same stems.

According to the design of PISA 2006 Science, one or more questions were asked in every item, and each question probed only one attribute. The polytomous and dichotomous Q-matrices for the test were identified based on this design (see Table 10). To do so, the frequency of every attribute that was probed in every item was identified and used as the corresponding element of the polytomous Q-matrix. For instance, there was only one question asked for Item 7 and it probed A3, so it produces the corresponding vector (0, 0, 1) for the polytomous Q-matrix. For Item 4, three questions were asked: One probed A1 and the other two probed A2, so the corresponding vector was (1, 2, 0) for the polytomous Q-matrix. Following this design, the number of attribute patterns for the polytomous Q-matrix should be 4 × 4 × 3 = 48. Second, the dichotomous Q-matrix can be defined in the traditional way by reducing the positive values of the polytomous Q-matrix to 1. The number of attribute patterns for the dichotomous Q-matrix was 2³ = 8.

Table 10.

Two Types of Q-Matrix for the PISA Test.

	Polytomous Q-matrix			Dichotomous Q-matrix
Item	A1	A2	A3	A1	A2	A3
1	1	0	1	1	0	1
2	1	1	0	1	1	0
3	0	1	0	0	1	0
4	1	2	0	1	1	0
5	0	2	2	0	1	1
6	0	1	2	0	1	1
7	0	0	1	0	0	1
8	0	3	0	0	1	0
9	1	1	2	1	1	1
10	0	1	2	0	1	1
11	3	0	0	1	0	0
12	3	0	2	1	0	1
13	0	1	1	0	1	1
14	0	2	2	0	1	1
15	2	0	1	1	0	1
16	0	3	0	0	1	0
17	0	2	1	0	1	1
18	0	3	0	0	1	0
19	2	0	2	1	0	1
20	0	1	2	0	1	1
21	1	1	2	1	1	1
22	0	0	2	0	0	1

Note: PISA = The Program for International Student Assessment; A1 = identifying scientific issues; A2 = explaining phenomena scientifically; A3 = using scientific evidence.

Analysis and Results

For the polytomous Q-matrix, the mapping from the attribute patterns to the IRPs follows the item scoring rule of Equation 4. As discussed in the “Q-Matrix Design and Scoring Method” section, a submatrix which contains the patterns corresponding to Items 2, 3, 7, 11, 15, 16, 17, and 22 can be found in the Q-matrix to make sure the mapping is one to one. Due to space limitation, the corresponding relationship was not shown. Similarly, the mapping between the attribute patterns and the IRPs for the dichotomous Q-matrix was also one to one. This is because the reachability matrix was also contained in the Q-matrix. The IRPs for the dichotomous Q-matrix are shown in Table 11.

Table 11.

Ideal Response Patterns for the Dichotomous Q-Matrix of the PISA Test.

	Attribute pattern (A1, A2, A3)
Item	(0,0,0)	(1,0,0)	(0,1,0)	(0,0,1)	(1,1,0)	(1,0,1)	(0,1,1)	(1,1,1)
1	0	1	0	1	1	2	1	2
2	0	1	1	0	2	1	1	2
3	0	0	1	0	1	0	1	1
4	0	1	2	0	3	1	2	3
5	0	0	2	2	2	2	4	4
6	0	0	1	2	1	2	3	3
7	0	0	0	1	0	1	1	1
8	0	0	3	0	3	0	3	3
9	0	1	1	2	2	3	3	4
10	0	0	1	2	1	2	3	3
11	0	3	0	0	3	3	0	3
12	0	3	0	2	3	5	2	5
13	0	0	1	1	1	1	2	2
14	0	0	2	2	2	2	4	4
15	0	2	0	1	2	3	1	3
16	0	0	3	0	3	0	3	3
17	0	0	2	1	2	1	3	3
18	0	0	3	0	3	0	3	3
19	0	2	0	2	2	4	2	4
20	0	0	1	2	1	2	3	3
21	0	1	1	2	2	3	3	4
22	0	0	0	2	0	2	2	2

Note: PISA = The Program for International Student Assessment; A1 = identifying scientific issues; A2 = explaining phenomena scientifically; A3 = using scientific evidence.

The GDD-P method, in which the generalized distance index was constructed by the GRM, was used to analyze the data with the two types of Q-matrices given in Table 10, respectively. Based on the classification results, the means of the students’ attribute patterns (i.e., the extent to which each attribute is mastered) from four different geographic areas were calculated (see Table 12). For the polytomous Q-matrix in Table 10, the attribute pattern that corresponds to the highest level of attribute mastery is (3, 3, 2). The highest and lowest levels of attribute masteries were bolded and italicized, respectively, for each attribute. With the polytomous Q-matrix, students from Hong Kong had the highest mastery of A1, whereas students from Mainland China had the highest masteries of A2 and A3; students from Macao had the lowest mastery levels for the three attributes. However, when the dichotomous Q-matrix was used to identify the students’ attribute patterns, different trends emerge: Students from Hong Kong did not have the highest mastery of A1; students from Mainland China had the lowest mastery of A2; although students from Macao still had the lowest mastery of A1 and A3, it was surprising that they had the highest mastery of A2. This phenomenon could be an indication of the information loss in converting the polytomous Q-matrix to a dichotomous Q-matrix.

Table 12.

Mean of Students’ Mastery Number Across Three Attributes Representing Science Competency.

	Attributes in the polytomous Q-matrix			Attributes in the dichotomous Q-matrix
Area	A1	A2	A3	A1	A2	A3
Taipei	2.062	2.401	0.957	0.701	0.248	0.771
Hong Kong	2.199	2.409	0.927	0.678	0.275	0.756
Macao	1.909	2.263	0.608	0.573	0.298	0.680
Mainland China	1.965	2.470	1.046	0.626	0.204	0.809

Note: A1 = identifying scientific issues; A2 = explaining phenomena scientifically; A3 = using scientific evidence.

Summary and Discussion

In this article, the authors develop an IRT-based method, the GDD-P, to carry out cognitive diagnosis for polytomous items. The core idea of the method is to identify IRPs by the generalized distance index for polytomous response, which can not only work with the scoring method used in the simulation studies but also with any other item scoring method for polytomous response. As an extension of Tatsuoka’s dichotomous Q-matrix, the Q-matrix with polytomous elements was also introduced in this article to accommodate attributes that have more than two levels. In addition, the use of the R or G_R matrix was proposed to ensure the one-to-one mapping between the attribute and the IRPs, which is a prerequisite for the identification of the attribute patterns for the proposed polytomous Q-matrix and the scoring method. The simulation studies indicate the proposed GDD-P method can recover the correct attribute patterns with a relatively high level of accuracy under different Q-matrix designs and attribute dependencies. The real data analysis showed how the GDD-P can be applied to a test with polytomous items using either the polytomous or the dichotomous Q-matrix.

Although the proposed procedure is promising, the practicality of applying the method to real testing situations requires a thoughtful consideration of a few important issues. First, in practical implementations of the GDD-P method, the number of attribute patterns should be carefully considered because too many attribute patterns will either affect the classification accuracy or require longer tests. As simulation studies indicate, if the number of attributes is not small and almost none of the attributes have extreme prerequisite relationships, there will be a large number of attribute patterns. If the attributes are probed at different levels and some elements of polytomous Q-matrix are not very small, the total number of attribute patterns may be too large to be practicable. Therefore, the attribute number, prerequisite relationship of attributes, and the polytomous Q-matrix design, all of which are related with the total number of attribute patterns and the correct classification rate, need to be considered before using the GDD-P method.

Second, the simulation studies and real data analysis showed that the GDD-P method can be implemented under both the dichotomous and polytomous Q-matrices. The choice between the two types of Q-matrices should be made according to purpose of the test and the constraints under which the test will be administered. Trade-off exists in choosing between the two Q-matrices. Compared with the dichotomous Q-matrix, the polytomous Q-matrix is capable of probing an attribute at different levels, which makes the diagnostic results more informative. However, the design of the polytomous Q-matrix would result in a larger number of attribute patterns, which can affect the precision of the classification of the attribute patterns for a fixed test length. A basic factor that can be considered to decide the more appropriate Q-matrix is the attribute number. If this number is small, the polytomous Q-matrix can be considered because a relatively simple Q-matrix design can be used to probe the levels of attributes. Otherwise, the dichotomous Q-matrix may be a better choice for analyzing the test with polytomous scores.

Third, it is important for the IRT-based methods such as the GDD-P to choose the proper item response functions to construct the discriminating indices. Although the IRT model used by the GDD-P method in this work was mainly chosen from the unidimensional models, one may also consider the appropriateness of multidimensional IRT models for constructing the discriminating index. Furthermore, to get more accurate cognitive diagnosis in practice, model selection procedure can be used to choose which IRT model has the best fit to the real data.

Finally, as shown by several researches on dichotomous Q-matrix (e.g., de la Torre, 2008; Rupp & Templin, 2008), Q-matrix misspecification can affect the accuracy of cognitive diagnostic methods for the examinee classification. Because similar problem can occur with the polytomous Q-matrix, domain experts should exert the utmost care to ensure that attribute levels of the polytomous Q-matrix are correctly specified. In addition, techniques for validating the specifications of the polytomous Q-matrix based on empirical data should also be given proper attention in the future.

Footnotes

Appendix

Acknowledgements

The authors would like to thank the editor and the anonymous reviewers for insightful comments and valuable suggestions. The first author also would like to thank China Scholarship Council for her visiting scholarship in USA.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by National Natural Science Foundation of China (11171029).

References

Chiu

Douglas

J. A.

(2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633-665.

de la Torre

(2008). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343-362.

de la Torre

(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179-199.

de la Torre

Douglas

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.

Ding

S. L.

Yang

S. Q.

Wang

W. Y.

(2010). The importance of reachability matrix in constructing cognitively diagnostic testing. Journal of Jiangxi Normal University (Natural Science), 34, 490-494.

Embretson

S. E.

Reise

S. P.

(2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.

J. B.

(2005). A polytomous extension of the fusion model and its Bayesian parameter estimation (Unpublished doctoral dissertation). University of Wisconsin, Madison.

J. B.

Y. M.

(2007, April). Cognitively diagnostic psychometric models: An integrative review. Paper presented at NCME, Chicago, IL.

10.

Haertel

E. H.

(1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.

11.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.

12.

Leighton

J. P.

Gierl

M. J.

Hunka

S. M.

(2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41, 205-237.

13.

Maris

(1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.

14.

Nichols

P. D.

Chipman

S. F.

Brennan

R. L.

(Eds.). (1995). Cognitively diagnostic assessment. Hillsdale, NJ: Erlbaum.

15.

Rupp

A. A.

Templin

J. L.

(2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78-96.

16.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores (Psychometrika Monograph No. 17). Richmond, VA: Psychometric Society.

17.

Sun

Zhang

Xin

Bao

(2011). A cognitive diagnosis method based on Q-matrix and generalized distance. Acta Psychologica Sinica, 43, 1095-1102.

18.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.

19.

Tatsuoka

K. K.

(1995). Architecture of knowledge structure and cognitive diagnosis: A statistical pattern recognition and classification approach. In Nichols

P. D.

Chipman

S. F.

Brennan

R. L.

(Eds.), Cognitively diagnostic assessment (pp. 327-359). Hillsdale, NJ: Erlbaum.

20.

Woodruff

D. J.

Hanson

B. A.

(1996). Estimation of item response models using the EM algorithm for finite mixture (ACT Research Rep. 96–6). Iowa City, IA: ACT.

Attribute	Attribute pattern
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
A1	0	1	0	0	0	0	0	1	1	1	1	1	0	0	0	0	0	0	1	1	1	1	1	1
A2	0	0	1	2	0	0	0	1	2	0	0	0	1	2	1	2	1	2	1	2	1	2	1	2
A3	0	0	0	0	1	2	3	0	0	1	2	3	1	1	2	2	3	3	1	1	2	2	3	3

Attribute	Attribute pattern
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
A1	0	1	0	0	0	0	0	1	1	1	1	1	0	0	0	0	0	0	1	1	1	1	1	1
A2	0	0	1	2	0	0	0	1	2	0	0	0	1	2	1	2	1	2	1	2	1	2	1	2
A3	0	0	0	0	1	2	3	0	0	1	2	3	1	1	2	2	3	3	1	1	2	2	3	3

A Polytomous Extension of the Generalized Distance Discriminating Method

Abstract

Keywords

GDD Method for Polytomous Response

GDD Method

Definition of Generalized Distance Index for Polytomous Response

Q-matrix Design and Scoring Method

Design of Q-Matrix

Item Scoring Method

Sufficiency of Polytomous Q-Matrix

Matrix R for Polytomous Q-Matrix

Matrix GR for Polytomous Q-Matrix

Simulation

Study 1

Condition 1: Data From Dichotomous Q-Matrix and Four Attribute Structures

Results

Study 2

Condition 2: Data From Polytomous Q-Matrix and Four Attribute Structures

Results

Study 3

Condition 3: Data From Dichotomous Q-Matrix and Higher Order Latent Trait Models

Results

A Real Data Analysis

Data Description

Analysis and Results

Summary and Discussion

Footnotes

Appendix

Acknowledgements

Declaration of Conflicting Interests

Funding

References

Matrix G_R for Polytomous Q-Matrix

Attribute	Attribute pattern
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
A1	0	1	0	0	0	0	0	1	1	1	1	1	0	0	0	0	0	0	1	1	1	1	1	1
A2	0	0	1	2	0	0	0	1	2	0	0	0	1	2	1	2	1	2	1	2	1	2	1	2
A3	0	0	0	0	1	2	3	0	0	1	2	3	1	1	2	2	3	3	1	1	2	2	3	3