Conjunctive and Disjunctive Extensions of the Least Squares Distance Model of Cognitive Diagnosis

Abstract

Many models of cognitive diagnosis, including the least squares distance model (LSDM), work under the conjunctive assumption that a correct item response occurs when all latent attributes required by the item are correctly performed. This article proposes a disjunctive version of the LSDM under which the correct item response occurs when at least one attribute is correctly applied. Also, under both the conjunctive and disjunctive versions of the LSDM, this article demonstrates an approach to estimating the conditional probability that (a) a specific pattern of p attributes, (b) exactly p attributes, and (c) at least p attributes will be correctly performed across locations on the logit scale in the item response theory under the one-, two-, or three-parameter logistic model. Such information can be useful for interpretations and decisions based on a person’s performance on attributes that govern the correct responses on binary items under unidimensional item response theory calibrations for assessment in education, psychology, and other fields.

Keywords

cognitive diagnosis item response theory psychometrics

Cognitive diagnosis deals with identification, validation, and analysis of latent attributes that underlie responses on items of assessment instruments in psychology, education, and other behavioral fields. Typically, the item responses are scored on a binary (1/0) scale. The attributes may refer to various latent characteristics such as cognitive processes, operations, and trait states. Cognitive diagnosis models (CDMs) integrate cognitive psychology and psychometrics (e.g., de la Torre, 2008, 2009; de la Torre & Douglas, 2004; DiBello, Stout, & Roussos, 1995; Dimitrov, 2007; Embretson, 1984, 1995; Embretson & Wetzel, 1987; Fischer, 1973; Gitomer & Rock, 1993; Henson & Douglas, 2005; Mislevy, 1996; Snow & Lohman, 1989; C. Tatsuoka, 2002; K. K. Tatsuoka, 1983, 2009; Templin & Henson, 2006). CDMs have been developed in a variety of frameworks such as item response theory (IRT), primarily Rasch modeling (e.g., Embretson, 1984, 1995; Fischer, 1973; Whitely & Schneider, 1981), combining IRT information with algebraic methods (e.g., Dimitrov, 2007; K. K. Tatsuoka, 1985), and nonparametric IRT, Bayesian modeling, and latent class modeling (e.g., de la Torre & Douglas, 2004; Henson & Douglas, 2005; Junker & Sijtsma, 2001; Mislevy, 1996; Templin & Henson, 2006; von Davier & Rost, 1997; von Davier & Yamamoto, 2004).

A key element in all models of cognitive diagnosis is the so-called Q-matrix. When a set of K attributes (e.g., cognitive operations and/or processes) is hypothesized to underlie the responses on J items, the Q-matrix is a J × K matrix with elements q_jk = 1 if item j requires attribute k, and q_jk = 0, otherwise (j = 1, . . . , J; k = 1, . . . , K). The identification of attributes that govern the responses on items of interest and correct specification of the respective Q-matrix are conditions of critical importance to the validity of results obtained under CDMs.

Most CDMs are conjunctive models as they are based on the assumption that a correct item response is produced when all attributes required by the item are mastered. Examples of conjunctive CDMs are the multicomponent latent trait model (Embretson, 1984), the LSDM (Dimitrov, 2007), the DINA (deterministic inputs; noisy “and” gate) model (de la Torre & Douglas, 2004; Haertel, 1989; Junker & Sijtsma, 2001; Macready & Dayton, 1977), and the NIDA (noisy inputs; deterministic “and” gate) model (Maris, 1999). For example, DINA is a stochastic conjunctive model in which the probability for individual i to correctly answer item j is conditioned on a conjunctive parameter, ξ_ij, as follows:

ξ_{i j} = \prod_{k = 1}^{K} α_{i k}^{q_{j k}},

(1)

where α _ik = 1 if individual i has mastered attribute A _k and α _ik = 0 otherwise, and q_jk is an element of the Q-matrix for item j and attribute A _k . With this, the probability for correct item response is specified as follows:

P (X_{i j} = 1 | ξ_{i j}) = (1 - s_{j})^{ξ_{i j}} (g_{j})^{1 - ξ_{i j}},

(2)

where s_j is the slip parameter (the probability that a master of all necessary attributes “slips” and answers the item incorrectly) and g_j is the guess parameter (the probability that a nonmaster of at least one attribute answers the item correctly).

There are situations, however, in which the item responses are not governed by the restrictive assumption that all attributes must be correctly applied. Such situations may occur in assessments of cognitive, behavioral, psychological, and personality traits. In assessment of pathological gambling, for example, Templin and Henson (2006) proposed a disjunctive version of the DINA model, referred to as DINO (deterministic input; noisy “or” gate) model. Although the parameter ξ_ij under DINA equals unity (ξ_ij = 1) only when person i has mastered all attributes required by item j, under DINO this parameter is redefined as a parameter ω_ij, which equals unity (ω_ij = 1) when at least one of these attributes has been mastered (Templin & Henson, 2006). That is,

ω_{i j} = 1 - \prod_{k = 1}^{K} {(1 - α_{i k})}^{q_{j k}} .

(3)

With this, the probability of correct item response under the DINO model is determined as

P (X_{i j} = 1 | ω_{i j}) = (1 - s_{j})^{ω_{i j}} (g_{j})^{1 - ω_{i j}} .

(4)

The Least Squares Distance Model

The LSDM (Dimitrov, 2007), which is of primary interest here, is a conjunctive CDM. However, unlike DINA (and other nonparametric models), the LSDM is based on information about item calibration under a unidimensional IRT model—the one-parameter (or Rasch), two-parameter, or three-parameter logistic models (1PL, 2PL, or 3PL). Unlike any other CDM, the LSDM does not require item score information, as long as IRT estimates of the item parameters are available. Specifically, using IRT estimates of the item parameters and the Q-matrix, the LSDM provides probability curves for individual attributes on the logit scale of IRT item calibration, as well as information about potential Q-misspecifications for individual test items. Under the LSDM, the probability of correct item response is presented as a product of the probabilities of correct processing of the attributes required by the item, that is,

P (X_{i j} = 1 | θ_{i}) = \prod_{k = 1}^{K} {[P (A_{k} = 1 | θ_{i})]}^{q_{j k}},

(5)

where X_ij is the binary (1/0) response of individual i on item j, θ _i is the trait score (in logits) of individual i, A _k is the kth attribute, and q_jk is the element of the Q-matrix for item j and attribute k (q_jk = 1 if item j requires attribute A _k and q_jk = 0, otherwise).

The LSDM is performed in steps as follows:

IRT estimates of the item parameters are obtained through IRT calibration or other sources (e.g., item banks or study reports).

A set of fixed θ values is selected to cover a targeted interval on the logit scale (say, from −4.0 to 4.0, with a step of 0.5).

The probability of correct item response, P(X = 1|θ), is computed for each θ value using the item parameter estimates under a specific IRT model of item calibration (1PL, 2PL, or 3PL).

Taking the natural logarithm on both sides of Equation (5) generates a norm of linear equations ||QX − L|| for each θ value, where Q is the Q-matrix, X is a vector of unknown elements ln P(A_k = 1|θ_i), and L is a vector of known elements P(X_ij = 1|θ_i)

The norm ||QX − L|| is minimized via least squares distances and the solution, X_k, is used to obtain the probability for correct processing of attribute A_k: P(A_k = 1|θ) = exp(X_k)

The attribute probabilities across θ values are tabulated and graphed to obtain attribute probability curves (APCs).

The item characteristic curve (ICC) of each item is recovered through the product of the probabilities of the attributes required by the item to evaluate the Q-matrix for possible misspecifications.

LSDM results can be particularly useful in (a) validation screening of attributes required by IRT-calibrated items (e.g., taken from an existing IRT item bank) prior to test administration and (b) providing information about probability curves of attributes on the logit scale, thus keeping the analysis of items and their underlying attributes within the familiar and widely used logit scale in IRT. The analytic simplicity of the LSDM allows for a straightforward development of a disjunctive version of the LSDM, as an IRT-based analog to the disjunctive model DINO (Templin & Henson, 2006), and other extensions described in the following.

Purpose of the Study

The purpose of this article is to extend the conjunctive LSDM (Dimitrov, 2007) at two levels. First, to provide a disjunctive version of the LSDM under which the target response on an item (X = 1) may occur when at least one of the attributes associated with the item is correctly applied. The notation LSDM-C is used here to denote the conjunctive LSDM (Dimitrov, 2007), whereas LSDM-D denotes the disjunctive LSDM. The second goal is to upgrade the LSDM by estimating the conditional probabilities that (a) specific patterns of p attributes, (b) exactly p attributes, and (c) at least p attributes will be correctly performed by individuals at a given location, θ, on the IRT logit scale (p = 1, . . . , K). The proposed approach works under LSDM-C and LSDM-D, thus providing flexibility and richness of psychometric information that can be useful for diagnostic decisions about individuals and groups based on attribute-related criteria. All estimation procedures are executable by a computer program named LSDM-CD (Least Squares Distance Model of Cognitive Diagnosis). LSDM-CD is developed in the framework of MATLAB (MathWorks, Inc., 2010) but is also adapted to operate as a stand-alone application for both Windows and Linux operating systems.¹

Method

To avoid confusion in terminology, some clarifications on terms and notations used in this section are necessary. First, target response (X = 1) can be the correct item response in an achievement test, endorsement of a statement in a personality test, manifestation of an observed behavior, and so on. Also, depending on the context, an attribute is performed, applied, or processed correctly. Furthermore, attributes associated with an item (in the Q-matrix) can also mean that they are required by (or related to) the item.

Disjunctive LSDM

Let A₁, A₂, . . . , A _K denote K attributes that are hypothesized to govern the target responses (X = 1) of individuals on a set of items that measure an underlying latent trait (or ability), θ. Under the conjunctive LSDM (LSDM-C; Dimitrov, 2007), the target response (X = 1) on an item is obtained if all attributes associated with the item in the Q-matrix are performed correctly. In this case, the probability of correct item response on item j by an individual i with a trait score θ _i on the logit scale, P(X_ij = 1|θ_i) is presented in Equation (5).

Under LSDM-D, the target response (X = 1) on an item is obtained if at least one of the attributes associated with the item in the Q-matrix is performed correctly. In this case, the probability of the target response (X = 1) to occur is estimated through the following transformations of Equation (5):

1 - P (X_{i j} = 1 | θ_{i}) = \prod_{k = 1}^{K} [1 - P (A_{k} = 1 | θ_{i})]^{q_{j k}},

(6)

which, after simple algebra, yields the following analytic form of the LSDM-D:

P (X_{i j} = 1 | θ_{i}) = 1 - \exp {\sum_{k = 1}^{K} q_{j k} ​ \ln [1 - P (A_{k} = 1 | θ_{i})]} .

(7)

Conditional Probabilities of Performing Attribute Patterns

Let $P (A_{p}^{v} | θ)$ denote the probability that an individual at the trait level θ will perform correctly a specific pattern, ν, of p attributes. With P(A_k = θ) denoting the probability for an individual at θ to correctly perform attribute A _k , and taking into account the IRT assumption of local independence, we have

P (A_{p}^{v} | θ) = \prod_{k = 1}^{K} P {(A_{k} | θ)}^{v (k)} [1 - P {(A_{k} | θ)}^{1 - v (k)}],

(8)

where ν(k) is the kth binary element in pattern ν——a combination of p (out of K) elements; ν(k) = 1 if attribute A _k participates in pattern ν and ν(k) = 0, otherwise.

Conditional Probabilities of Performing Exactly p Attributes

Let $P (A_{p}^{=} | θ)$ denote the probability that an individual at the trait level θ will perform correctly exactly p attributes (i.e., any pattern of p attributes). Also, let $C_{K}^{p}$ denote the set of all possible patterns of p (out of K) elements. For convenience, these patterns are arranged here in a matrix of $(\begin{matrix} K \\ p \end{matrix})$ rows and K columns, with entries of 0 and 1. Each row has p elements equal to 1 and the remaining (K − p) elements equal to 0. Under these notations, the probability for an individual at θ to correctly perform exactly p attributes (any pattern of p attributes) is

P (A_{p}^{=} | θ) = \sum_{v \in C_{K}^{p}} P (A_{p}^{v} | θ),

(9)

where $P (A_{p}^{v} | θ)$ is estimated via Equation (8) and the summation is performed by rows of the $C_{K}^{p}$ matrix (i.e., across all patterns of p attributes: $v \in C_{K}^{p}$ ).

Conditional Probabilities of Performing at Least p Attributes

Let $P (A_{p}^{\geq} | θ)$ denote the probability that an individual at the trait level θ will perform correctly at least p attributes. For a set of K attributes (A₁, A₂, . . . , A _K ), an individual will perform correctly at least p attributes if this individual performs correctly exactly p, or exactly p + 1, . . . , or exactly K attributes. In probabilistic parlance, these are independent events, so the probability of performing at least p attributes is the sum of the probabilities of performing exactly p, or exactly p + 1, . . . , or exactly K attributes. Thus, the probability for an individual at θ to perform at least p attributes is given by

P (A_{p}^{\geq} | θ) = \sum_{k = p}^{K} P (A_{k}^{=} | θ),

(10)

where $P (A_{k}^{=} | θ)$ is estimated via Equation (9).

To summarize, the conditional probabilities for correct performance of a specific pattern of p attributes, exactly p attributes, and at least p attributes are estimated through Equations (8), (9), and (10), respectively, where $P (A_{k} | θ)$ ——the conditional probability to correctly perform attribute A _k is estimated either under the LSDM-C or LSDM-D (see Equations 5 and 7, respectively).

Validity Issues

As both LSDM-C and LSDM-D use estimates of item parameters, the assumption is that these estimates are obtained under an adequate model–data fit of the respective IRT model——1P (or Rasch), 2P, or 3P model. Thus, testing for IRT model–data fit should be performed prior to using LSDM. When item parameter estimates are taken from, say, an IRT item bank or other sources, researchers should attend to evidence of IRT model–data fit provided with such sources. Assuming dependable estimates of the item parameters, the validity of LSDM-based results is concerned with the adequacy of the set of attributes and possible misspecification in the Q-matrix.

Regarding the conjunctive model (LSDM-C), Dimitrov (2007) outlined some empirical criteria related to validity of attributes and their association with individual test items. First, the APCs should exhibit an expected monotonic behavior, relative difficulty, and discrimination. For example, if LSDM-C analysis is conducted on a set of reading comprehension items, it is logical to expect that the APCs should increase with the increase of the underlying ability of reading comprehension. Second, possible misspecifications in the Q-matrix are signaled by the lack of acceptable LSDM-based recovery of the ICCs. Under LSDM-C, the ICC recovery for an item is obtained by multiplying the probabilities for correct processing of the attributed required by the item. The mean absolute difference (MAD) between the ICC of an item and its LSDM recovery represents an index of Q-matrix fit for this particular item (MAD = 0 indicates perfect ICC recovery). As suggested by Dimitrov (2007), MAD < .05 indicates an adequate ICC recovery, 05 ≤ MAD < .10 indicates a tolerable ICC recovery, and MAD ≥ .10 indicates a poor ICC recovery. Overall, this rule of thumb was supported by a series of simulated studies on LSDM (Dimitrov, Romero, & Ponsoda, 2008; Romero, 2010). More refined information in this regard, which is beyond the scope of this article, is provided by Romero (2010) under various simulation conditions for number of items, number of attributes, and relative difficulty of attributes.

The aforementioned rule of thumb for MAD values in checking for misspecifications in the Q-matrix applies also to the disjunctive model (LSDM-D). However, while the ICC recovery of an item under LSDM-C is based on Equation (5), the ICC recovery of an item under LSDM-D is based on Equation (7). It is important to emphasize also that the criterion of monotonic behavior of APCs, which is expected under LSDM-C, does not fully apply to LSDM-D. This is because under the LSDM-D, a positive item response (X = 1) is obtained if at least one of the attributes associated with the item in the Q-matrix is performed correctly. This implies, for example, that an increasing APC of a relatively easy attribute may decrease at high ability levels where patterns of attributes that are more likely to occur do not contain the easier attribute under consideration. Thus, given the element of randomness associated with the occurrence of “at least one” attribute in the manifestation of a target response (X = 1), the criterion for monotonic behavior of APCs is not strictly applicable under LSDM-D.

It is also important to note that the selection of LSDM-C versus LSDM-D is guided by the nature (conjunctive vs. disjunctive) of the attributes that are sufficient for obtaining the targeted response (X = 1) on individual items (i.e., all vs. at least one, respectively). In other words, the LSDM choice (LSDM-C or LSDM-D) is not a post hoc decision based on a criterion of “better data fit.” It seems fair to say that, when applicable, LSDM-C would be the proper choice in most education and psychological tests (e.g., ability or performance testing). The practical use of LSDM-D would be primarily in determining whether “at least” a given number of attributes are processed for classification decisions related to, say, level of depression, anxiety, and other psychological or behavioral traits (assuming unidimensionality of the trait). The LSDM-C and LSDM-D are illustrated next in Examples 1 and 2, respectively. The computer program LSDM-CD was used to render tabulated results, APCs, ICC recovery, and probability curves for specific pattern of p attributes, exactly p attributes, and at least p attributes (p = 1, . . . , K).

Example 1

This example illustrates an application of the conjunctive model (LSDM-C; Dimitrov, 2007) with the proposed extensions for estimating $P (A_{p}^{v} | θ)$ , $P (A_{p}^{=} | θ)$ , and $P (A_{p}^{\geq} | θ)$ across θ values on the logit scale. The input data, provided in Table 1, consist of a Q-matrix (13 items × 5 attributes) and estimates of the item difficulty parameter under the Rasch model (Rasch, 1960). Specifically, Rasch calibration was conducted on the binary scores (1 = correct, 0 = incorrect) of 799 ninth-grade students on 13 items in an algebra test of simple linear equations. The following five attributes were hypothesized to underlie the correct solutions of these equations: A₁ = collecting terms, A₂ = removing parentheses, A₃ = removing denominators, A₄ = solving for a variable with nonnumerical coefficients, and A₅ = processing nontrivial cases. For example, the correct solution of the equation 2(x − n) = 5 would require correct processing of A₂ and A₄ (this is Item 3 in Table 1). Nontrivial cases were defined as situations that are either not typical (e.g., an equation 17 − 2 = 4x − 4x + 15, which yields 0x = 0) or involve some level of complexity such as removing brackets or a set of parentheses. Of course, these five attributes can be further refined, but the purpose here is to illustrate LSDM-C, not to develop a comprehensive cognitive model of solving algebra equations.

Table 1.

Rasch Item Parameter Estimates and Q-Matrix for Algebra Equation Items and Five Attributes

		Attributes
Item	$\hat{b}$	A₁	A₂	A₃	A₄	A₅
1	−1.463	1	1	0	0	0
2	−1.841	1	0	0	0	0
3	0.873	0	1	0	1	0
4	−0.489	0	1	0	0	1
5	0.425	1	0	0	1	0
6	0.486	1	1	0	0	1
7	−1.652	1	0	0	0	0
8	−0.383	1	0	0	0	1
9	2.097	1	0	1	0	1
10	−0.818	1	1	0	0	0
11	0.673	1	1	0	0	1
12	1.921	1	1	0	1	1
13	0.171	1	1	0	0	1

Note. $\hat{b}$ = Rasch estimate of the item difficulty parameter.

Prior to conducting LSDM-C, it was found that the data fit the Rasch model under the conditional maximum-likelihood chi-square test for the entire set of items and the Wald z test for fit of individual items. Specifically, the chi-square value was not statistically significant, χ²(12) = 19.67, p > .05, and the Wald z values for individual items were within the acceptable range from −2.00 to 2.00. The item calibration was performed using the computer program for Rasch analysis LPCM-WIN (Fisher & Ponocny-Seliger, 1998).

Attribute Probability Curves

As described earlier, LSDM-C solutions of Equation (5) render estimates of probabilities for correct performance of attributes across θ levels on the logit scale (Dimitrov, 2007). For the data in Table 1, the APCs are shown in Figure 1. As expected under the LSDM-C, the APCs increase with the increase of the ability on the logit scale. Figure 1 also shows, for example, that A₃ (removing denominators) is the most difficult attribute, whereas A₁ (collecting terms) is, for the most part, the easiest attribute for the study population of examinees.

Figure 1.

Attribute probability curves under the conjunctive least squares distance model (LSDM-C) analysis of algebra equations and five underlying attributes (A₁ = collecting terms, A₂ = removing parentheses, A₃ = removing denominators, A₄ = solving for a variable with nonnumerical coefficients, and A₅ = processing nontrivial cases)

ICC Recovery

The ICC recovery of an item under LSDM-C is the product of the APCs for the attributes required by that item. In light of the criteria related to MAD between an ICC and its LSDM-C recovery, discussed earlier, the results for the 13 algebra items indicated an adequate recovery for nine items, with MAD < .05, and tolerable recovery for four items, with .05 < MAD < .09. For illustration, the ICC recovery of one item (Item 3, with MAD = .0256) is depicted in Figure 2. Overall, the expected monotonic behavior of the APCs and ICC recovery of items provide satisfactory evidence of LSDM-C validity for the purpose of this example.

Figure 2.

Recovery of the item characteristic curve for Item 3 under the conjunctive least squares distance model analysis of five attributes for the algebra test items

Probability for Correct Performance of Attribute Patterns

The estimation of conditional probabilities for correct performance of specific attribute patterns is illustrated here for all possible patterns of three and four (out of five) attributes at two locations on the logit scale (θ = −1 and θ = 1). Table 2 shows LSDM-C estimates of conditional probabilities for correct performance of each attribute (A₁, . . . , A₅) at θ = −1 and θ = 1. The estimates of conditional probabilities of correct performance of all possible patterns of three and four attributes are provided in Table 3. They are obtained using Equation (8) with the probability estimates for individual attributes provided in Table 2. For example, let us take the first pattern of three attributes in Table 3, ν = (1 1 1 0 0). The probability of correct performance of this pattern at θ = 1, $P (A_{3}^{v} | θ = 1) = . 0261,$ is obtained as follows:

\begin{array}{l} P (A_{3}^{v} | θ = 1) = \prod_{k = 1}^{5} P {(A_{k} | θ)}^{v (k)} {[1 - P (A_{k} | θ)]}^{1 - v (k)} = P (A_{1} | θ) P (A_{2} | θ) P (A_{3} | θ) \\ [1 - P (A_{4} | θ)] [1 - P (A_{5} | θ)] = (1.000) (. 9551) (. 1858) (1 - . 4728) (1 - . 7207) = . 0261 \end{array}

Table 2.

Conditional Probabilities for Correct Performance of Attributes at θ = −1.0 and θ = 1.0 for the Algebra Test Items

	P(A_k\|θ)Estimate
Conditional Probability P(A_k\|θ)	θ = −1.0	θ = 1.0
P(A₁\|θ)	.7684	1.0000
P(A₂\|θ)	.5649	.9551
P(A₃\|θ)	.0272	.1858
P(A₄\|θ)	.0784	.4728
P(A₅\|θ)	.2442	.7207

Table 3.

Conditional Probabilities for Correct Performance of All Possible Patterns of Three and Four (out of Five) Attributes at θ = −1.0 and θ = 1.0 for the Algebra Test

Pattern Matrix: $C_{5}^{3}$	$P (A_{3}^{v} \| θ = - 1)$	$P (A_{3}^{v} \| θ = 1)$	Pattern Matrix: $C_{5}^{4}$	$P (A_{4}^{v} \| θ = - 1)$	$P (A_{4}^{v} \| θ = 1)$
1 1 1 0 0	.0082	.0261	1 1 1 1 0	.0007	.0234
1 1 0 1 0	.0250	.1027	1 1 1 0 1	.0027	.0674
1 1 0 0 1	.0950	.2955	1 1 0 1 1	.0081	.2650
1 0 1 1 0	.0005	.0011	1 0 1 1 1	.0002	.0028
1 0 1 0 1	.0020	.0032	0 1 1 1 1	.0001	.0000
1 0 0 1 1	.0062	.0125
0 1 1 1 1	.0002	.0000
0 1 1 0 1	.0008	.0000
0 1 0 1 1	.0024	.0000
0 0 1 1 1	.0001	.0000

The LSDM-C estimates of probabilities for correct performance of the patterns of four attributes in Table 3 are depicted in Figure 3. As can be seen, pattern (1 1 0 1 1), which does not contain the most difficult attribute (A₃), is much easier than all other patterns across all ability levels on the logit scale, whereas pattern (0 1 1 1 1) almost never occurs.

Figure 3.

Probability curves for patterns of four (out of five) attributes under the conjunctive least squares distance model analysis of five attributes for the algebra test items (the attributes, from left to right, in each pattern are A₁, A₂, A₃, A₄, and A₅)

Probability of Performing Correctly Exactly p Attributes

Under LSDM-C, the probability of performing correctly exactly p attributes (i.e., any pattern of p attributes) is the sum of the conditional probabilities of correct performance of all possible patterns of p attributes (p = 1, . . . , K). For example, using the results in Table 3, the probability of performing correctly exactly four (out of five) attributes for persons located at θ = −1 on the logit scale is $P (A_{4}^{=} | θ = - 1)$ = .0007 + .0027 + .0081 + .0002 + .0001 = .0118. For persons located at θ = 1, the probability is $P (A_{4}^{=} | θ = 1)$ = .0234 + .0674 + .2650 + .0028 + .0000 = .3586. The probability curves for correct performance of exactly p attributes (p = 0, 1, 2, 3, 4, 5) are shown in Figure 4. The θ scores at which two adjacent probability curves intersect are thresholds for equal probabilities of performing correctly exactly p or exactly p + 1 attributes (p = 0, 1, 2, 3, 4). In this example, the LSDM-C output reported the following as thresholds: (a) θ = −2.370 for p = 0, (b) θ = −1.211 for p = 1, (c) θ = 0.056 for p = 2, (d) θ = 1.152 for p = 3, and (e) θ = 2.188 for p = 4.

Figure 4.

Probability curves for correct performance of exactly p attributes (p = 0, 1, . . . , 5) under the conjunctive least squares distance model analysis of five attributes for the algebra test items

Probability of Performing Correctly At Least p Attributes

Under LSDM-C, the estimation of the probability of performing at least p (out of K) attributes, $P (A_{p}^{\geq})$ is estimated by using Equation (10). In this example, the probability curves for $P (A_{p}^{\geq})$ are shown in Figure 5, where the horizontal line (at probability level .5) can be useful when diagnostic classifications are based on a probability of .5 or higher for performing at least p attributes. The “cutting scores” on the ability scale for correct performance of at least p attributes, with a probability of .5 or higher, were found to be the following: (a) θ = −2.635 for p = 1, (b) θ = −1.220 for p = 2, (c) θ = 0.030 for p = 3, (d) θ = 1.168 for p = 4, and (e) θ = 2.277 for p = 5.

Figure 5.

Probability curves for correct performance of at least p attributes (p = 1, . . . , 5) under the conjunctive least squares distance model analysis of five attributes for the algebra test items

Example 2

This example illustrates the disjunctive model (LSDM-D) for simulated data. The item parameters (discrimination, a, and difficulty, b) of 25 items were generated, with a randomly selected from a uniform distribution, U(0, 2), and b randomly selected from the standard normal distribution, N(0, 1). Probabilities of correct item responses were generated under the IRT two-parameter logistic model, P(X = 1|θ, a, b) for values of θ from −5.0 to 5.0, with a “step” of 0.5. A Q-matrix for six attributes (A₁, . . . , A₆) associated with the 25 items was also generated to reflect the disjunctive rule of correct item performance under the LSDM-D. The rationale behind using simulations in this example was to produce data based on the disjunctive rule of correct item performance for an appropriate illustration of the LSDM-D. The resulting item parameters and Q-matrix are provided in Table 4.

Table 4.

Simulated Item Parameters and Q-Matrix for LSDM-D Analysis

Item	b	a	A₁	A₂	A₃	A₄	A₅	A₆
1	−2.0437	1.2165	1	0	1	0	0	0
2	−1.0343	1.0246	0	1	0	0	0	1
3	−1.0116	1.0003	0	1	0	0	0	0
4	−0.4369	0.9521	0	0	1	0	0	0
5	1.8547	0.9502	0	0	0	0	0	1
6	−1.9838	1.0194	1	0	0	0	0	1
7	−1.0390	1.1021	0	1	0	0	1	0
8	−2.0648	1.1344	1	0	0	1	0	1
9	−1.0390	1.1021	0	1	0	0	1	0
10	−2.0648	1.1344	1	0	0	1	0	0
11	0.9184	1.0952	0	0	0	0	1	0
12	−1.9838	1.0194	1	0	0	0	0	0
13	−2.0437	1.2165	1	0	1	0	0	1
14	−2.1772	1.2523	1	1	0	0	0	0
15	−1.0390	1.1021	0	1	0	0	1	0
16	0.3950	1.0475	0	0	0	1	0	1
17	−2.0648	1.1344	1	0	0	1	0	1
18	−1.0390	1.1021	0	1	0	0	1	0
19	−1.4298	1.3185	0	1	1	1	0	1
20	−0.4640	0.9974	0	0	1	0	0	1
21	−1.9838	1.0194	1	0	0	0	0	0
22	−1.4165	1.3136	0	1	1	1	0	0
23	0.9184	1.0952	0	0	0	0	1	0
24	−0.5452	1.1733	0	0	1	0	1	1
25	0.9184	1.0952	0	0	0	0	1	0

Note. LSDM-D = disjunctive least squares distance model.

The logic of LSDM-C estimation of probabilities for specific patterns of p attributes, exactly p attributes, and at least p attributes (p = 1, . . . , K), illustrated in Example 1, carries over the estimation under LSDM-D. Keep in mind, however, that although the conditional probabilities of correct attribute performance, P(A _k |θ) are estimated via Equation (5) under LSDM-C, they are estimated via Equation (7) under LSDM-D. The results in this example were obtained using the LSDM-D option in the computer program LCDM-CD. As expected for the simulated data in this example, the MAD values for LSDM-D recovery of the ICCs for all 25 items were below the cutting value of .05 for adequate recovery (the MAD values ranged from .0014 to .0421).

For space consideration, shown here are only probability curves for correct performance of exactly p attributes (p = 0, 1, . . . , 6; Figure 6) and correct performance of at least p attributes (p = 1, . . . , 6; Figure 7). As noted earlier, the information depicted in Figure 7 would probably be of main interest for classification of individuals into trait-related categories (e.g., depression or some pathological behavior). For example, using the disjunctive DINO model, Templin and Henson (2006) classified persons as probable pathological gamblers when the probability that such persons meet at least 5 (out of 10) criteria was .5 or higher. In Figure 7, the “cutting scores” on the ability scale for correct performance of at least p attributes, with a probability of .5 or higher, were found to be the following: (a) θ = −2.228 for p = 1, (b) θ = −1.281 for p = 2, (c) θ = −0.575 for p = 3, (d) θ = 0.301 for p = 4, (e) θ = 1.215 for p = 5, and (f) θ = 2.448 for p = 6.

Figure 6.

Probability curves for correct performance of exactly p attributes (p = 0, 1, . . . , 6) under disjunctive least squares distance model analysis of simulated data on 25 binary items and six attributes

Figure 7.

Probability curves for correct performance of at least p attributes (p = 1, . . . , 6) under disjunctive least squares distance model analysis of simulated data on 25 binary items and six attributes

Discussion

The LSDM of cognitive diagnosis (Dimitrov, 2007) is based on a conjunctive model in which a correct item response assumes correct performance of all attributes required by the item (see Equation 5). Unlike other CDMs, the LSDM does not require item score information as long as IRT estimates of the item parameters are available (e.g., from an item bank or published sources). LSDM renders estimates of probabilities of correct attribute performance and validity information about recovery of ICCs from hypothesized attributes across trait scores on the IRT logit scale.

This article provides a straightforward extension of the conjunctive model (LSDM-C) to a disjunctive version (LSDM-D) under which a target item response (X = 1) is obtained when at least one of the attributes associated with the item is correctly performed. This is achieved by applying the LSDM algorithm using Equation (7) for the relationships between the probability of a correct item response and the probabilities of correct performance of attributes associated with the item.

Further extensions of both LSDM-C and LSDM-D render estimates of probabilities that (a) specific patterns of attributes, (b) an exact number of attributes, and (c) at least a specified number of attributes will be correctly performed by individuals at a given location on the IRT logit scale. The LSDM-C and LSDM-D allow researchers to conduct analysis and interpretations of both test items and attributes in a unified framework of IRT calibration for unidimensional binary items. The relative simplicity of the LSDM-based estimation procedures allows for their straightforward coding in the syntax of computer programs for mathematics or statistical data analysis such as MATLAB (MathWorks, Inc., 2010) and R (R Development Core Team, 2009). In this article, all results (probability estimates, ICC recovery, etc.) were obtained using the computer program LSDM-CD.

In conclusion, given the variety of contexts and measurement scenarios for cognitive diagnosis, the choice of a specific model (e.g., LSDM-C or DINA; LSDM-D or DINO) in any particular case would depend on the purpose of the study, nature of the data, and so on. It is important to emphasize that, unlike other models of cognitive diagnosis, LSDM-based analyses render psychometric information about binary items and their attributes on the same scale—namely, the familiar logit scale in IRT. The LSDM (Dimitrov, 2007) and its psychometric extensions proposed in this article can be particularly useful in analyses and interpretations of binary items and underlying (cognitive, behavioral, or psychological) attributes within a unified framework of unidimensional IRT calibrations in psychology, education, and other disciplines. Possibilities for multidimensional extensions of the LSDM-C and LSDM-D can be examined in future research.

Footnotes

Notes

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

de la Torre

(2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343-362.

de la Torre

(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.

de la Torre

Douglas

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.

DiBello

L. V.

Stout

W. F.

Roussos

L. A.

(1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In Nichol

Chipman

Brennan

(Eds.), Cognitive diagnostic assessment (pp. 361-389). Hillsdale, NJ: Lawrence Erlbaum.

Dimitrov

D. M.

(2007). Least squares distance method of cognitive validation and analysis for binary items using their item response theory parameters. Applied Psychological Measurement, 31, 367-387.

Dimitrov

D. M.

Romero

S. J. R.

Ponsoda

(2008, March). Sensitivity of the least-squares distance method of cognitive analysis to misspecification in the incidence Q-matrix: A simulation study. Paper presented at the meeting of the American Educational Research Association, New York.

Embretson

S. E.

(1984). A general latent trait model for response processes. Psychometrika, 49, 175-186.

Embretson

S. E.

(1995). A measurement model for linking individual learning to process and knowledge: Application to mathematical reasoning. Journal of Educational Measurement, 32, 277-294.

Embretson

S. E.

Wetzel

C. D.

(1987). Component latent trait models for paragraph comprehension tests. Applied Psychological Measurement, 11, 175-193.

10.

Fischer

G. H.

(1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37, 359-374.

11.

Fisher

G. H.

Ponocny-Seliger

(1998). Structural Rasch modeling. Handbook of the usage of LPCM-WIN 1.0. Groningen, Netherlands: ProGamma.

12.

Gitomer

D. H.

Rock

(1993). Addressing process variables in test analysis. In Frederiksen

Mislevy

R. J.

Bejar

(Eds.), Test theory for a new generation of tests (pp. 243-268). Hillsdale, NJ: Lawrence Erlbaum.

13.

Haertel

E. H.

(1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.

14.

Henson

Douglas

(2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29, 262-277.

15.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.

16.

Macready

G. B.

Dayton

C. M.

(1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99-120.

17.

Maris

(1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.

18.

MathWorks

Inc

. (2010). MATLAB. The language of technical computing (Version 7.10.0). Natick, MA: Author.

19.

Mislevy

R. J.

(1996). Test theory reconceived. Journal of Educational Measurement, 33, 379-416.

20.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research. (Expanded edition, 1980. Chicago, IL: University of Chicago Press)

21.

R Development Core Team. (2009). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

22.

Romero

S. J. R.

(2010). Propiedades y aplicaciones del método de las distancias mínimo-cuadráticas (LSDM) para la validaciÓn y análisis de atributos cognitivos [Properties and applications of the least squares distance method (LSDM) for validation and analysis of cognitive attributes] (Unpublished doctoral dissertation [in Spanish]). Universidad Autonoma de Madrid, Spain.

23.

Snow

R. E.

Lohman

D. F.

(1989). Implications of cognitive psychology for educational measurement. In Linn

(Ed.), Educational measurement (3rd ed., pp. 263-331). New York, NY: Macmillan.

24.

Tatsuoka

(2002). Data-analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society Series C (Applied Statistics), 51, 337-350.

25.

Tatsuoka

K. K.

(1983). Rule-space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.

26.

Tatsuoka

K. K.

(1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational Statistics, 10, 55-73.

27.

Tatsuoka

K. K.

(2009). Cognitive assessment: An introduction to the rule space method. New York, NY: Routledge.

28.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.

29.

von Davier

Rost

(1997). Self-monitoring—A class variable? In Rost

Langeheine

(Eds.), Applications of latent trait and latent class models in the social sciences (pp. 296-305). Münster, Germany: Waxmann.

30.

von Davier

Yamamoto

(2004). Partially observed mixtures of IRT models: An extension of the generalized partial-credit model. Applied Psychological Measurement, 28, 389-406.

31.

Whitely

S. E.

Schneider

L. M.

(1981). Information structure for geometric analogies: A test theory approach. Applied Psychological Measurement, 5, 383-397.