Quantifying Local,Response Dependence Between Two Polytomous Items Using the Rasch Model

Abstract

Models of modern test theory imply statistical independence among responses, generally referred to as local independence. One violation of local independence occurs when the response to one item governs the response to a subsequent item. Expanding on a formulation of this kind of violation as a process in the dichotomous Rasch model, this article generalizes the dependence process to the case of the unidimensional, polytomous Rasch model. It then shows how the magnitude of this violation can be estimated as a change in the location of thresholds separating adjacent categories in the second item caused by the response dependence on the first. As in the dichotomous model, it is suggested that this index is relatively more tangible in interpretation than other indices of dependence that are either a weight in the interaction term in a model or a correlation coefficient. One function of this method of assessing dependence is likely to be in the development of tests and assessment formats where evidence of the magnitude of dependence of one item on another in a pilot study can be used as part of the evidence in deciding which items will be retained in a final version of a test or which formats might need to be reconstructed. A second function might be to identify the magnitude of response dependence that may then need to be taken into account in some other way, perhaps by applying a model that takes account of the dependence.

Keywords

polytomous Rasch model local independence response dependence writing assessment performance assessment halo

Models of modern test theory generally assume statistical independence among responses of persons to multiple items of an assessment instrument. In modern test theory, this independence is generally referred to as local independence (Andrich, 1991; Lazarsfeld & Henry, 1966). Andrich and Kreiner (2010) noted that, although generally assumed from the perspective of statistical modeling, this assumption of independence reflects an ideal design of an instrument composed of multiple items. They make the case that the two main reasons that tests are composed of many locally independent items are that the greater the number of different statistically independent items, the greater the potential for both the precision and validity of measurement. Thus, if other relevant factors such as the number of items, the administration of the instrument, sample size, alignment of persons, and items are held constant, responses to locally independent items provide more information than do responses that are locally dependent.

Andrich and Kreiner’s (2010) concern was with unidimensional variables in which there is only one person parameter, and they review a range of related topics in handling violations of local independence, including the structural sources of dependence and the approaches to indentifying and accounting for it. Their focus was on responses that are dichotomous. Their review will not be repeated but only summarized in the present article. Regarding the sources of violations, they note the two kinds of generic violations of local independence identified in Marais and Andrich (2008): The first is where responses to subsets of items interact with subsets of persons so that more than one dimension plays a role in the responses; second is where the response to an item is governed by a response to another, generally previously administered, item. Marais and Andrich give examples of each kind of violation.

Andrich and Kreiner (2010) also noted that there are a range of procedures for identifying and accounting for dependence, including ones that arise from tests of violations of fit when the model assumes unidimensionality and response independence, and ones in which models quantify local dependence as interaction terms (e.g., Haberman, 2007; Kreiner & Christensen, 2004).

Andrich and Kreiner (2010) identified and quantified response dependence between two dichotomously scored items using an approach different from those previously appearing in the literature. They used a recent characterization of the violation of local independence in the dichotomous Rasch model as a response process in which the location of a dependent item was changed by its dependence on another, independent item. They then showed how the magnitude of this change in location could be estimated. The advantage of estimating the magnitude of dependence in this way, they argued, was that it makes tangible its effect in terms of the values of the key parameters of the items, their respective locations. Because such an analysis is not based on simply modeling response dependence, but in identifying and quantifying its magnitude, the main benefits are likely to be in the improvement in the construction of items and their response formats.

This article generalizes the results shown in Andrich and Kreiner (2010) for dichotomously scored items using the dichotomous Rasch model to the case of response dependence between two polytomously scored items with more than two, putatively ordered categories, using the polytomous Rasch model (PRM). In achievement testing with dichotomously scored items, response dependence can arise when the correct response to one item in some way affects the probability of a correct response to a subsequent item. Practical guides on test construction include advice on how to construct items that do not have such a property, for example, obtaining the correct answer to one test item should not be based on having correctly answered a prior item (Mehrens & Lehman, 1991). In the case of polytomous response formats, response dependence may occur when a particular rating on one item either implies logically, or creates empirically, the same rating on another item. Marais and Andrich (2008) suggested a halo effect as an example of the latter where an assessment of some performance by a single assessor on multiple criteria provides a greater relationship among the criteria than if different assessors assessed the different criteria independently. Such assessments are very common in psychology and education. A brief analysis of a real example of this kind is provided in this article.

This article focuses on the dependence between two polytomous items which have the same number of ordered categories. A single simulated example and an example with real data are used to illustrate the procedure. It is stressed, however, that the simulation is not a study of the properties of the estimates, which in any case are well known to be consistent (Andersen, 1973; Zwinderman, 1995) when conditional maximum likelihood estimation, as is the case in the examples, is used. In addition, in conditioning out the person parameter, the person distribution plays no role in the estimation and in the assessment of the magnitude of any dependence among these items. These prove to be important properties in the method described in this article for estimating the effect of dependence.

The rest of the article is structured as follows: The section titled “The PRM and the Formulation of Response Dependence” reviews the PRM, the formulation of response dependence in the dichotomous case, and its generalization to the polytomous case. The sections on illustrative simulation example and illustrative real example follow, and the last section is a summary.

The PRM and the Formulation of Response Dependence

There are a number of forms in which the PRM can be expressed. The one expression convenient for this article (Andrich, 2010) takes the form

\Pr {X_{n i} = x_{i}} = [\exp (ψ_{x i} + x_{i} β_{n}] / γ_{n i},

where (a) $x_{i} \in {0, 1, 2, \dots, m_{i}}$ is the integer response variable termed the score for person $n$ who responds to item $i$ and has a maximum value of $m_{i}$ (b) $B_{n k}$ is the proficiency of person n (c) $ψ_{xi}, x = 0, 1, 2, \dots, m_{i}$ are score coefficients for item $i$ defined according to $ψ_{x i} = - \sum_{k = 0}^{x_{i}} δ_{i k}, k = 0, 1, 2, \dots, m_{i}; δ_{i 0} \equiv 0,$ where $δ_{i k}$ is the threshold point at which a response in the two successive scores, $k - 1$ and $k$ , are equally likely and (d) $γ_{n i} = \sum_{x = 0}^{m_{i}} \exp (ψ_{x i} + x_{i} β_{n})$ is a normalizing factor that ensures $\sum_{x = 0}^{m_{i}} Pr {X_{ni} = x_{i}} = 1$ . Threshold $δ_{0 i} \equiv 0$ , which does not exist, is included for notational convenience in the above expressions. Figure 1 shows an example of the category characteristic curves of an item with three thresholds and therefore a maximum score of $m_{i} = 3$ .

Figure 1.

Response probability curves for item j with and without dependence of d = 0.25 on item i, where x_ni = 2 and x_nj = 2

A special case of the model in which all items have the same $m$ thresholds, $δ_{1}, δ_{2}, \dots, δ_{m}$ takes the form

Pr {X_{ni} = x_{i}} = [\exp (ψ_{x} + x_{i} β_{n}] / γ_{ni},

where $ψ_{x i} = ψ_{x} = - \sum_{k = 0}^{x_{i}} δ_{k}, k = 0, 1, 2, \dots, m$ in which all items have the same maximum score $m$ and the same thresholds $δ_{k}, k = 0, 1, 2, \dots, m; δ_{0} \equiv 0$ . The formulation in Equation 1 is often termed a partial credit parameterization of the Rasch model (Wright & Masters, 1982), and that of Equation 2, a rating scale parameterization (Andrich, 1978). The former is more general than the latter in that the thresholds can take on different values for different items; in the latter, the thresholds take on the same values for different items. In terms of a response process of a single person to a single item, there is no difference between Equations 1 and 2. The focus in this article is on Equation 1, the more general form, in which the values of the thresholds for different items may be different, even though the number of thresholds between any two items studied for local dependence in this case is the same. A more general form of the model is when the number of thresholds also varies among items, but in studying response dependence in this article, the authors are not concerned with this case.

It is observed from Figure 1, and it can be proved, that the range in which the probability of the response $x$ is the greatest of the response probabilities is bounded by the adjacent thresholds $δ_{x - 1}$ and $δ_{x}$ (Andrich, 2005). This observation is relevant in formulating response dependence between two polytomous items. The dichotomous model of Equation 3 for two ordered categories, $m_{i} = 1$ , is a further special case of Equation 2 and simplifies to

Pr {X_{ni} = x_{i}} = \exp (x_{i} (β_{n} - δ_{i})) / γ_{ni},

in which there is only the one threshold, $δ_{i}$ .

Local Response Dependence With Dichotomous Responses

The formulation of response dependence (Andrich & Kreiner, 2010; Marais & Andrich 2008) for the dichotomous Rasch model of Equation 3 is reviewed. Statistical independence for the response of person $n$ to items $i, j$ , implies

Pr {X_{nj} = x_{j} | X_{ni} = x_{i}} = Pr {X_{nj} = x_{j}} .

Marais and Andrich (2008) introduced a violation of this condition in the dichotomous Rasch model as follows:

\begin{matrix} Pr {X_{nj} = 1 | X_{ni} = 1} = \exp (β_{n} - (δ_{j} - d)) / [1 + \exp (β_{n} - (δ_{j} - d))], \\ Pr {X_{nj} = 1 | X_{ni} = 0} = \exp (β_{n} - (δ_{j} + d)) / [1 + \exp (β_{n} - (δ_{j} + d))], \end{matrix}

where $| d | \neq 0$ is the magnitude of the dependence. Clearly, Equation 5 does not satisfy Equation 4.

To interpret Equation 5 explicitly, Marais and Andrich (2008) supposed $d > 0$ and that the items assessed educational proficiency, scored 1 as correct and 0 as incorrect, and therefore that the location parameters of the items could be interpreted as their relative difficulties. Then a correct response $x_{i} = 1$ by person $n$ to item $i$ , which will be referred as the independent item, reduces the difficulty of item $j$ , which will be referred as the dependent item, to $δ_{j} - d$ , thus increasing the probability of the same correct response $x_{j} = 1$ to item $j$ ; similarly, the response $x_{i} = 0$ on item $i$ increases the difficulty of item $j$ to $δ_{j} + d$ , thus increasing the probability of the same incorrect response $x_{j} = 0$ to item $j$ . The value of $d$ can be either positive or negative: If it is negative, then a correct response on item $i$ implies that the probability of an incorrect response on item $j$ would be greater than if there were no response dependence.

Using the above formulation, Andrich and Kreiner (2010) showed that the dependence effect $d$ could be estimated and its deviation from a hypothesized value of 0 tested for statistical significance. The first purpose of the present article is to generalize the formulation of response dependence of Equation 5, as a violation of Equation 4, to the case of polytomous responses characterized by the PRM of Equation 1. The second is to generalize the estimation procedure in Andrich and Kreiner to estimate the magnitude of any response dependence defined earlier.

Local Response Dependence With Polytomous Responses

Although response dependence in Equation 5 can be interpreted as a change in relative difficulty, a second interpretation, more readily generalized to the case of polytomous responses, is to consider the effect that the change in the point of dichotomization has on the range of the continuum of the dependent item.

First, consider a response to item $j$ , which is independent of a response to item $i$ . Then, it is noted that the continuum is dichotomized at $δ_{j}$ . Thus, if $β > δ_{j}$ , then $Pr {X_{ni} = 1} > 0.5$ , and if $β < δ_{j}$ , then $Pr {X_{ni} = 1} < 0.5$ . Now consider the effect of dependence $d > 0$ in Equation 5, when the response to independent item $i$ is $X_{ni} = 1$ . In this case, $δ_{j} \to δ_{j} - d$ . Thus, the point of dichotomization of the continuum has been shifted from $δ_{j}$ to $δ_{j} - d$ . Moreover, this increases the relative range of the continuum in which $Pr {X_{ni} = 1} > 0.5$ . A complementary interpretation can be made if the response to independent item $i$ is $X_{ni} = 0$ .

This interpretation of response dependence as inducing a change in the range of the continuum in which item $j$ is more likely to have the same response as independent item $i$ is used to formulate response dependence in the case of polytomous items. Specifically, if the response to independent $i$ is $X_{ni} = x_{i}$ , then to increase the probability of a response $X_{nj} = x_{j}$ for dependent item $j$ , the range of the continuum in which the response $X_{j} = x_{j}$ is most likely increased relative to when there is no dependence.

Let the response to independent item $i$ be $X_{ni} = x_{i}$ according to Equation 1, and suppose that there is a $d > 0$ analogous to that in Equation 5, which increases the probability of the response $X_{nj} = x_{j} = x_{i}$ . To increase this probability, the distance between the thresholds that define the range of the continuum in which this probability is a maximum is increased. Specifically, let $δ_{kj}^{*} = δ_{kj} - d$ for $k = x_{i}$ and $δ_{kj}^{*} = δ_{kj} + d$ for $k = x_{i} + 1$ . However, in conjunction with these shifts in the location of the two adjacent thresholds, it is necessary to shift the thresholds on either side of $δ_{kj}, δ_{(k + 1) j}$ accordingly. Thus, for $0 < x_{i} < m_{i}$ , all thresholds $δ_{kj}, k < x_{i}$ are shifted to the left, those $δ_{kj}, k > x_{i} + 1$ are shifted to the right, and for the extreme scores $0$ and $m_{i}$ , all thresholds are shifted to the right or left, respectively. This shift ensures that the effect on the change of probabilities is confined to an increase in probability of the response $X_{nj} = x_{j}$ of dependent item $j$ as a result of the response of $X_{ni} = x_{i}$ on independent item $i$ . The complementary reduction in probabilities for the remaining scores, $X_{nj} = x_{j}, x_{j} \neq x_{i}$ , which arises from the constraint that the probabilities of the responses to item $j$ must still sum to 1, is then a direct consequence of the increase in probability of $X_{nj} = x_{j}$ .

In summary, let $δ_{kj}^{*} = δ_{kj} - d$ for all $k \leq x_{i}$ and let $δ_{kj}^{*} = δ_{kj} + d$ for all $k > x_{i}$ . Then the $Pr {X_{nj} = x_{j} | X_{ni} = x_{i}}$ can be written in the form of Equation 1 as

Pr {X_{nj} = x_{j} | X_{ni} = x_{i}} = [\exp (ψ_{xj}^{*} + x_{j} β_{n}] / γ_{nj},

where $ψ_{xj}^{*} = - \sum_{k = 0}^{x_{i}} δ_{kj}^{*}, k = 0, 1, 2, \dots, m_{i}; δ_{0 j}^{*} \equiv 0$ .

Figure 1 provides a graphical depiction of the process where (a) item $j$ has a maximum score of $m_{j} = 3$ and its three threshold values are, respectively, $- 1.00, - 0.25, 1.25$ ; (b) the response to item $j$ is dependent on the response to item $i$ with $d = 0.25$ ; and (c) the specific response to $i$ is $x_{ni} = 2$ according to Equation 1. From this formulation, for any value of the person location $β_{n}$ , $Pr {X_{nj} = 2_{j} | X_{ni} = 2_{i}} > Pr {X_{nj} = 2_{j}}$ .

Table 1 shows $Pr {x_{nj} = 2}$ without and with response dependence on item $i$ for a person location $β$ at three values on the continuum. The first and third values are at the two thresholds that define the range of the continuum in which $x_{nj} = 2$ has the greatest probability, before and after dependence is invoked, and the middle one is at the peak probability itself, which is at the same location of 0.5 in both cases. It is evident from Table 1 that in each case, the corresponding probabilities are greater with, than without, response dependence.

Table 1.

Probabilities of x_nj = 2 and x_nji for Selected Values of β

No dependence, β at	−0.250	0.500	1.250
Pr{x_nj = 2} (no dependence)	0.371	0.445	0.475
Dependence, β at	−0.500	0.500	1.500
Pr{x_nj = 2\|x_ni} (dependence)	0.384	0.466	0.540

Estimation of the Magnitude of Dependence d

To estimate the magnitude of dependence $d$ , the data is reconstructed in two steps. First, the responses $X_{nj} = x_{j}$ of item $j$ that depend on item $i$ are resolved into $m_{i} + 1$ distinct items (recall $m_{i} = m_{j}$ ), where each resolved item is composed of responses of those persons whose responses on item $i$ are $X_{ni} = x_{i}$ . A resolved item $j$ whose response corresponds to response $x_{i} = x_{j}$ is denoted as $j_{xi}$ . Thus, in the preceding example of Figure 1 and Table 1 in which $m_{i} + 1 = 4$ , there are four resolved items, $j_{0 i}, j_{1 i}, j_{2 i}, j_{3 i}$ .

Of course, the original item $j$ is deleted from the matrix. In the data matrix so reconstructed, responses to the four resolved items $j_{xi}; x = 0, 1, 2, 3$ never overlap in the sense that if a person has a response to $j_{i 0}$ , for example, then the person does not have a response to any of $j_{1 i}, j_{2 i}, j_{3 i}$ . The four items form structurally missing data relative to each other and are, therefore, structurally independent.

However, because their responses are all dependent on the responses to item $i$ , they are statistically dependent, indirectly, in a matrix of responses that includes responses to item $i$ . Therefore, to remove this dependence, in a second step, item $i$ is also deleted from the matrix. In the resultant matrix, there is no further statistical dependence that involves the resolved items $j_{xi}; x = 0, 1, 2, 3$ . This independence is a direct generalization of resolution of dichotomous items presented in Andrich and Kreiner (2010).

Clearly, having eliminated items $i$ and original item $j$ , to conduct any analyses, there must be other items in the data matrix. Suppose that the responses to the remaining items in the response matrix conform to the Rasch model satisfactorily and that there are sufficient items to permit the estimation of the thresholds of the resolved items $j_{xi}; x = 0, 1, 2, 3$ . Although there are $j_{xi}; x = 0, 1, 2, 3$ resolved items, there are only $x = 1, 2, 3$ thresholds for each item. Let ${\hat{δ}}_{j 1 i}, {\hat{δ}}_{j 2 i}, {\hat{δ}}_{j 3 i}$ be their estimates. Then from the definitions of Equation 6, Table 2 shows the respective estimates in terms of ${\hat{δ}}_{jxi}$ and $\hat{d}$ .

Table 2.

Estimates of the Thresholds for Each Resolved Item j_ix

x_i	Item	${\hat{δ}}_{j 1 i}$	${\hat{δ}}_{j 2 i}$	${\hat{δ}}_{j 3 i}$
0	$j_{i 0}$	${\hat{δ}}_{j 1} + d$	${\hat{δ}}_{j 2} + d$	${\hat{δ}}_{j 3} + d$
1	$j_{i 1}$	${\hat{δ}}_{j 1} - d$	${\hat{δ}}_{j 2} + d$	${\hat{δ}}_{j 3} + d$
2	$j_{i 2}$	${\hat{δ}}_{j 1} - d$	${\hat{δ}}_{j 2} - d$	${\hat{δ}}_{j 3} + d$
3	$j_{i 3}$	${\hat{δ}}_{j 1} - d$	${\hat{δ}}_{j 2} - d$	${\hat{δ}}_{j 3} - d$

In general, let the estimates of ${\hat{δ}}_{jki}, k = 1, 2, \dots, m_{j}$ for $x_{i}, x_{i} = 0, 1, \dots, m_{i}$ , $m_{i} = m_{j}$ , be denoted as ${\hat{δ}}_{jki (x_{i})}$ . Then a subset of these estimates can be used to estimate $d$ . Specifically, Table 3 shows this relevant subset of estimates ${\hat{δ}}_{j (k = x) i x_{i}}$ for the example of Table 2. These are specifically the pairs ${\hat{δ}}_{j (k = x) i (x_{i} - 1)}$ and ${\hat{δ}}_{j (k = x) i x_{i}}$ , for ${\hat{δ}}_{j (k = x) i (x_{i})}, k = 1, 2, 3,$ where $x_{i} = 0, 1, 2, 3$ .

Table 3.

Subset of Estimates of the Thresholds for Each Resolved Item $j_{ix}$ Relevant for Estimating $d$

$x_{i}$	${\hat{δ}}_{j 1 i (x_{i})}$	${\hat{δ}}_{j 2 i (x_{i})}$	${\hat{δ}}_{j 3 i (x_{i})}$
0	${\hat{δ}}_{j 1} + \hat{d}$
1	${\hat{δ}}_{j 1} - \hat{d}$	${\hat{δ}}_{j 2} + \hat{d}$
2		${\hat{δ}}_{j 2} - \hat{d}$	${\hat{δ}}_{j 3} + \hat{d}$
3			${\hat{δ}}_{j 3} - \hat{d}$
$[({\hat{δ}}_{ji (k = x) (x_{i} - 1)} - ({\hat{δ}}_{ji (k = x) (x_{i})}] / 2$	${\hat{d}}_{1}$	${\hat{d}}_{2}$	${\hat{d}}_{3}$

The mean of the successive differences of each of these pairs gives an estimate of $d$ , where for notational convenience, each of these means may be denoted ${\hat{d}}_{k}, k = 1, 2, \dots, m_{j}$ . These means are shown in the last row of Table 3. Thus, an estimate of $d$ is obtained from each of the $m_{j}$ thresholds, which is consistent with the formulation of the response dependence with respect to each of the $m_{j} + 1$ response categories and $m_{j}$ thresholds. The mean of these $m_{j}$ estimates ${\hat{d}}_{k}, k = 1, 2, \dots, m_{j}$ provides a single estimate $\hat{d}$ . In summary,

{\hat{d}}_{k} = \frac{\sum_{k = 1}^{m_{j}} [{\hat{δ}}_{ji (k = x) (x_{i} - 1)} - {\hat{δ}}_{ji (k = x) (x_{i}))}] / 2}{m_{j}} = \frac{\sum_{k = 1}^{m_{j}} {\hat{d}}_{jk}}{m_{j}} .

The estimate ${\hat{d}}_{k}$ may be negative or positive. If ${\hat{d}}_{k} < 0$ , it would imply that a response $x_{ni}$ in the independent item $i$ induces a smaller probability of a response $x_{nj}$ in the dependent item $j$ . This decrease may occur in cases where some kind of compensation appears in assessment on different criteria, a kind of effect that is the opposite of a halo effect.

Standard Errors of the Magnitude of the Estimate of d

In general, the hypothesis is that $d = 0$ . Because estimates of ${\hat{δ}}_{jik}$ have standard errors ${\hat{σ}}_{j (k = x) i (x_{i})}$ , which are provided routinely from maximum likelihood theory, it is possible to test this hypothesis. An estimate of the standard error ${\hat{σ}}_{d}$ of $\hat{d}$ is obtained by building it up from the standard error of each estimate ${\hat{d}}_{k}$ :

{\hat{d}}_{k} = [({\hat{δ}}_{ji (k = x) (x_{i} - 1)} - {\hat{δ}}_{ji (k = x) (x_{i})}] / 2,

$x_{i} = 1, 2, 3 \dots m_{i}$ , where, because of the elimination of responses of both the original items $i$ and $j$ , the estimates ${\hat{δ}}_{ji (k = x) (x_{i} - 1)}$ and ${\hat{δ}}_{ji (k = x) (x_{i})}$ are independent. Using the standard formulation of the variance of the mean difference between two independent variables, from Equation 8, the standard error ${\hat{σ}}_{k}$ of ${\hat{d}}_{k}$ is given by

{\hat{σ}}_{k} = \sqrt{({\hat{σ}}_{ji (k = x) (x_{i} - 1)}^{2} + {\hat{σ}}_{ji (k = x) (x_{i})}^{2}) / 4},

$x_{i} = 1, 2, 3, \dots, m_{i}$ . Table 4 shows all the variances of the errors of the estimates for the example in Table 3. Equation 9 is a generalization of the standard error of the estimate of $\hat{d}$ in the case of dichotomous items, which was discussed in substantial detail in Andrich and Kreiner (2010).

Table 4.

Estimated Variance Errors of Each $({\hat{δ}}_{jk} \pm \hat{d})$ and of Each $d_{k}, k = 1, 2, \dots, m_{j}$

$x_{i}$	${\hat{δ}}_{ji 1} \| x_{i}$	${\hat{δ}}_{ji 2} \| x_{i}$	${\hat{δ}}_{ji 3} \| x_{i}$
0	$({\hat{δ}}_{j 1} + \hat{d}) {\hat{σ}}_{ji (1) (0)}^{2}$
1	$({\hat{δ}}_{j 1} - \hat{d}) {\hat{σ}}_{ji (1) (1)}^{2}$	$({\hat{δ}}_{j 2} + \hat{d}) {\hat{σ}}_{ji (2) (1)}^{2}$
2		$({\hat{δ}}_{j 2} - \hat{d}) {\hat{σ}}_{ji (2) (2)}^{2}$	$({\hat{δ}}_{j 3} + \hat{d}) {\hat{σ}}_{ji (3) (2)}^{2}$
3			$({\hat{δ}}_{j 3} - \hat{d}) {\hat{σ}}_{ji (3) (3)}^{2}$
	${\hat{σ}}_{j 1}^{2} = \frac{{\hat{σ}}_{ji (1) (0)}^{2} + {\hat{σ}}_{ji (1) (1)}^{2}}{4}$	${\hat{σ}}_{j 2}^{2} = \frac{{\hat{σ}}_{ji (2) (1)}^{2} + {\hat{σ}}_{ji (2) (2)}^{2}}{4}$	${\hat{σ}}_{j 3}^{2} = \frac{{\hat{σ}}_{ji (3) (2)}^{2} + {\hat{σ}}_{ji (3) (3)}^{2}}{4}$
$Pooled {\hat{σ}}_{j}^{2} = \frac{({\hat{σ}}_{ji (1) (0)}^{2} + {\hat{σ}}_{ji (1) (1)}^{2}) + ({\hat{σ}}_{ji (2) (1)}^{2} + {\hat{σ}}_{ji (2) (2)}^{2}) + ({\hat{σ}}_{ji (3) (2)}^{2} + {\hat{σ}}_{ji (3) (3)}^{2})}{4 + 4 + 4}$

With $m_{j}$ estimates ${\hat{d}}_{k}$ , the mean of these estimates, $\hat{d} = \sum_{k = 1}^{m_{j}} \frac{{\hat{d}}_{jk}}{m_{j}}$ , is taken to obtain a single estimate $\hat{d}$ . To obtain a single estimate of the standard error ${\hat{σ}}_{d}$ of $\hat{d}$ , it is supposed that the error variances of the estimates ${\hat{d}}_{k}, k = 1, 2, \dots, m_{j}$ are homogeneous. In that case, to obtain a single estimate ${\hat{σ}}_{d}^{2},$ these variances can be pooled. A single pooled estimate ${\hat{σ}}_{d_{k}}^{2}$ of the variance of each ${\hat{d}}_{k}, k = 1, 2, \dots m_{j},$ in the case of Table 4, is shown in its last row. In general,

Pooled {\hat{σ}}_{d_{k}}^{2} = \frac{\sum_{k = 1}^{m_{j}} \sum_{x = k - 1}^{k} {\hat{σ}}_{ji (k) (x)}^{2}}{4 m_{j}} .

Then the variance of the mean of the $m_{j}$ estimates of ${\hat{d}}_{k}, k = 1, 2, \dots, m_{j}$ , which is the estimate of $d$ , is given by dividing Equation 10 by $m_{j}$ , giving

{\hat{σ}}_{d}^{2} = \frac{\sum_{k = 1}^{m_{j}} \sum_{x = k - 1}^{k} {\hat{σ}}_{ji (k) (x)}^{2}}{4 m_{j}^{2}}

and

{\hat{σ}}_{d} = \frac{\sqrt{\sum_{k = 1}^{m_{j}} \sum_{x = k - 1}^{k} {\hat{σ}}_{ji (k) (x)}^{2}}}{2 m_{j}} .

A Simulated Example

This section introduces a small simulated example that illustrates the procedure for estimating the magnitude of dependence based on the preceding formulation.

Simulation Design

The simulated example had the following features:

There were 30 items, all items had three thresholds, and therefore four categories and a maximum score of $m = 3$ ;

20 items had no dependence structure;

5 items had response dependence on the immediately preceding item, in particular, Item 22 was dependent on Item 21, 24 on 23, 26 on 25, 28 on 27, and 30 on 29;

the dependence value in the preceding formulation was $d = 0.25$ ;

the thresholds were randomly, uniformally generated, and then reordered to be in the natural order, in the range $- 4.0, 4.0$ within the set of 20 independent items and within the set of 10 items with a response-dependent structure; and

there were 2000 persons, normally distributed with mean 0 and standard deviation 2.0 $(β ~ N (0, 2^{2}))$ , making the persons range approximately between −4 and 4 logits.

Results

The item parameters were estimated using the computer program RUMM2030 (Andrich, Sheridan, & Luo, 2011), which uses the conditional, pairwise maximum likelihood algorithm (Andrich & Luo, 2003) to eliminate the person parameters and from which the estimates of the item parameters are consistent (Zwinderman, 1995). RUMM2030 also accounts for missing responses routinely, a necessary feature given that the resolved items constructed from each original item have structurally missing responses.

Using the structure of Table 3 to obtain an estimate of ${\hat{d}}_{k}, k = 1, 2, 3$ , Table 5 shows the estimates of these values for each of the five items which were dependent on the immediately previous item.

Table 5.

Estimates of $d$ From Resolved Item $j$ With Respect to Item $i$ for 5 Pairs of Items Where the Theoretical Value of $d = 0.25$

Items $(i, j)$	${\hat{d}}_{1}$	${\hat{d}}_{2}$	${\hat{d}}_{3}$	$\hat{d}$	MLE ${\hat{σ}}_{d}$	${\hat{σ}}_{\bar{d}}$	$z$
$(22, 21)$	0.293	0.203	0.232	0.242	0.023	0.026	10.335
$(23, 24)$	0.144	0.235	0.283	0.220	0.024	0.041	9.115
$(25, 26)$	0.264	0.289	0.280	0.278	0.025	0.007	11.202
$(27, 28)$	0.324	0.154	0.162	0.213	0.027	0.055	7.770
$(29, 30)$	0.199	0.265	0.059	0.174	0.029	0.061	5.958
Mean (pooled) values				0.226	0.026	0.038	8.705

It is evident from Table 5 that an excellent estimate ${\hat{d}}_{k}$ of $d$ is obtained from each item pair. The maximum likelihood (ML) based estimate ${\hat{σ}}_{d}$ from each item in Table 5 is obtained from Equation 12. An empirical estimate ${\hat{σ}}_{de}$ from the $m_{j} = 3$ estimates of item $j$ can be obtained as ${\hat{σ}}_{de} = \sqrt{\sum_{k = 1}^{3} {(d_{k} - \bar{d})}^{2} / m_{j} (m_{j} - 1)}$ , also shown in Table 5. With $m_{j} = 3$ , the number of replications is relatively small, and it is expected that the empirical estimate will be less stable than the theoretical estimate; however, it can be seen from Table 5 that the theoretical and empirical estimates of the standard error are of a compatible order of magnitude. In addition, the estimate of $d = 0.25$ , ranging from 0.174 to 0.278, is excellent. It is clear from Table 5 that, in each case, the null hypothesis, $d = 0$ , would be rejected using the standardized normal deviate $z = (\bar{\hat{d}} - 0) / {\hat{σ}}_{d}$ where ${\hat{σ}}_{d}$ is given by the MLE estimate.

Given five item pairs which are statistically independent, and each pair having the same dependence value of $d = 0.25$ , it is possible to pool them and their standard errors across the five item pairs. Thus, the mean of the 15 estimates, notated $\bar{d}$ is considered, three from each of the five item pairs. Because the number of thresholds is the same for each item, this approach will give the same estimate as in the last row of Table 5, the mean of the estimates from each item.

This mean estimate, $\bar{d} = 0.226$ , is shown in the last row of Table 5. To obtain the pooled standard error of $\bar{d}$ obtained from the 15 estimates, Equation 11 is used, and it is assumed that these estimates are homogeneous across items. As evident in Table 5, homogeneity is a reasonable assumption. The pooled estimate of the standard error of $\bar{d}$ across the 15 estimates is then given by

{\hat{σ}}_{\bar{d}}^{2} = \frac{\sum_{j = 1}^{5} \sum_{k = 1}^{m_{j}} \sum_{x = k - 1}^{k} {\hat{σ}}_{ji (k) (x)}^{2}}{5 (4 m_{j}^{2})} = \frac{0.121}{180} = 0.001,

and ${\hat{σ}}_{\bar{d}} = 0.026$ . This value is shown in the last row of Table 5, as is the standardized $z = 0.226 / 0.026 = 8.705$ , which is clearly significant at conventional levels of significance.

The estimated empirical standard deviation of these 15 estimates is 0.072, giving a standard error of the mean of $0.072 / \sqrt{15} = 0.019$ . This value is close to that obtained from the MLE estimates of 0.026. In summary, from this illustrative example, it is evident that the estimates and their standard errors are recovered more than adequately. This outcome of course is to be expected given that the estimates of the thresholds parameters are consistent and that the theoretical errors are derived from ML theory. As indicated earlier, the example is provided illustratively rather than as a study of the properties of the estimates.

Finally, to consolidate the illustration of estimating the value of $d$ , an identical simulation to the preceding one, except that the dependence value was set to $d = 0$ , was conducted. The estimate $\hat{d}$ obtained across five replicated item pairs was $\bar{d} = - 0.031$ with a theoretical standard error of ${\hat{σ}}_{\bar{d}} = 0.026$ (as in the example with $d = 0.25$ ). This gives $z = - 0.031 / 0.026 = - 1.192$ , which implies accepting the null hypothesis $\bar{d} = 0$ at conventional levels of significance. Thus, when $d = 0$ , the null hypothesis that $\bar{d} = 0$ was accepted.

A Real Example

To use the preceding approach to quantify response dependence in the analysis of real data, two preliminary conditions need to be satisfied. First, it is necessary to have a hypothesis as to which items might be dependent. Second, the remaining items of the test should fit the RM reasonably well. Of course, if the dependence between the hypothesized pairs of items is not sufficiently large to be detected at the level of power provided by the sample size, then the estimate of dependence $d$ will not be significantly different from 0.

One way of generating a hypothesis of whether two specific items show response dependence is to analyze the data with the PRM, form standardized residuals between the observed and expected responses from the parameter estimates and the model, and study the correlations among these residuals. A correlation of high magnitude between the residuals of two items would suggest some form of dependence between them that is not accounted for by the PRM. To ensure reasonable fit of the data to the model for the remaining items, items that show gross misfit to the model can be eliminated. Then the estimate of the magnitude of the dependence between two items is relative to that set of items which fit the model. In conjunction with such a quantitative estimate, it is necessary to consider the substantive features of the items and to understand the dependence in these terms.

This way of generating a hypothesis of response dependence between items is based on a post hoc analysis of data. However, these kinds of approaches involve general tests of fit, and if all items have some common feature, the misfit may not be detected (Smith, 2002). The power of detecting misfit with a specific hypothesis is greater than the power of a general test of fit. The test of a hypothesis of dependence is one such specific test which might show significance even though general tests of fit would indicate no or little violation of the model. Thus, an alternative, theoretically driven, method for generating a hypothesis of response dependence is from the structure and format of the items in relation to each other. The real data analyzed in the following are one such example where the hypothesis of response dependence was generated from the response structure and format and where response dependence is detected even though general tests of fit did not point to any response dependence.

The Gyagenda and Engelhard (2009) Data Set

The example used illustratively in the following involves a data set analyzed comprehensively by Gyagenda and Engelhard (2009), where the tests of fit they used did not indicate misfit to the Rasch model (Gyagenda & Engelhard, 2009). However, as shown in this article, a specific test of a hypothesis of response dependence showed that it was significantly different from zero.

The data set contains ratings of 366 eighth-grade students’ essays in the Georgia High School Writing Test. Each student was rated by 20 raters on four criteria: content/organization, style, conventions, and sentence formation. On each criterion, a rater assigned a rating of “inadequate,” scored 1; “minimal,” scored 2; “good,” scored 3; and “very good,” scored 4. Thus, in the notation of the PRM, $x_{ni} = 0, 1, 2, 3$ with $m_{i} = m_{j} = 3$ .

Gyagenda and Engelhard (2009) analyzed the data to explore rater, criterion, and student gender influences on writing ability. They used classical and modern test theory approaches in their analysis and compared the strengths and limitations of the two approaches.

Following the classical test theory approach, test reliability indicated through alpha coefficients with the criteria as items, was .87. Interrater reliability, with the raters as items, was .99. Total and criterion scores were analyzed using ANOVA with rater and gender as explanatory variables. Results indicated statistically significant mean differences among raters. In addition, overall, female students scored higher than did males, resulting in a statistically significant gender difference. There was also a significant rater by gender interaction effect.

Following the modern test theory approach, they used the multifacet PRM model with the proficiency of the person, the difficulty of each criterion, the severity of the rater, and student gender as facets (Linacre & Wright, 2002; Lunz, Wright, & Linacre, 1990). The person separation index of reliability, based on a single person estimate across all criteria, was 0.99. Results from this analysis also revealed effects of rater and gender on student writing. Despite intensive training, raters rated with different severity. This effect has often been reported and has led to the development of the multifacet Rasch model (e.g., Lunz et al., 1990), used in the analysis. Female students scored on average 1 logit higher than did males. This female superiority over males in writing performance is well documented (e.g., Hedges & Nowell, 1995).

General tests of fit indicated that all criteria and raters fitted the model. Gyagenda and Engelhard (2009) did not report any evidence of local dependence, in particular no suggestion of a possible halo effect. When individual raters display a halo effect but the majority do not, then those raters typically misfit the multifacet Rasch model (Myford & Wolfe, 2004). However, when there is a common, possibly a structural reason for the presence of halo, such tests of fit are unlikely to identify any halo effect (cf. Smith, 2002).

These data satisfy the two preliminary conditions for using the approach outlined in this article. First, because a halo effect has been well documented in cases where raters rate on multiple criteria (e.g., Myford & Wolfe, 2004), there is a basis for an a priori hypothesis of local dependence. Second, Gyagenda and Engelhard (2009) reported that the data fit the RM reasonably well, which allows for the estimation of relative dependence among items.

In the next section, the data are reanalyzed and the possibility of response dependence between two of the four criteria is considered. In a comprehensive analysis, for example, for purposes of improving ratings, dependence between all the criteria would be investigated. However, because in this article the analysis is only an illustration of the methodology, a comprehensive analysis of their data is left for a later date.

Estimation of the Magnitude of Dependence d

Data were reanalyzed using the PRM of Equation 1 with the proficiency of the person and the difficulty of each rater–criterion combination as items. That is, the data were “racked” (Wright, 2003) with each person having ratings on 80 items (20 raters by 4 criteria). In Gyagenda and Engelhard (2009), the first rater was numbered 2 and the last 21, a numbering retained in this article. Thus, for any one person, Items 1 to 4 contained the ratings of Rater 2, Items 5 to 8 contained the ratings of Rater 3, Items 9 to 12 contained the ratings of Rater 4, and so on.

The specific hypothesis tested was of response dependence between Criterion 1 (content/organization) and Criterion 2 (style) ratings for each rater; that is, the hypothesis that Items 2, 6, 10, and so on were, respectively, dependent on Items 1, 5, 9, and so on. Each of the hypothesized dependent items was resolved into four distinct items, one for each category of the independent item. The independent item was deleted, as was the complete dependent item. Using the structure of Table 3, an estimate of $d_{k}, k = 1, 2, 3$ for each rater is obtained. Table 6 shows, for each rater, the estimates of these values as well as the MLE-based estimate ${\hat{σ}}_{d}$ and the standardized normal deviate z.

Table 6.

Estimates of $d$ for All Raters

Rater	Items	${\hat{d}}_{1}$	${\hat{d}}_{2}$	${\hat{d}}_{3}$	$\hat{d}$	${\hat{σ}}_{d}$	$z$
2	(1,2)	1.048	0.776	0.595	0.806	0.064	12.594
3	(5,6)	2.027	1.413	0.817	1.419	0.065	21.831
4	(9,10)	0.395	−0.104	0.232	0.174	0.068	2.559
5	(13,14)	0.600	0.144	0.435	0.393	0.063	6.238
6	(17,18)	1.311	0.637	0.995	0.981	0.071	13.817
7	(21,22)	0.039	0.190	0.409	0.212	0.065	3.262
8	(25,26)	−0.244	0.206	0.618	0.193	0.065	2.969
9	(29,30)	1.084	1.750	0.847	1.227	0.070	17.529
10	(33,34)	0.785	0.481	0.883	0.716	0.072	9.944
11	(37,38)	0.666	0.274	0.480	0.473	0.063	7.508
12	(41,42)	0.397	0.361	0.452	0.403	0.066	6.106
13	(45,46)	0.601	0.632	0.686	0.640	0.063	10.159
14	(49,50)	0.829	0.693	1.009	0.843	0.072	11.708
15	(53,54)	1.552	1.562	2.316	1.810	0.070	25.857
16	(57,58)	1.193	0.908	0.910	1.004	0.067	14.985
17	(61,62)	0.864	0.719	0.845	0.809	0.062	13.048
18	(65,66)	1.291	0.315	0.431	0.679	0.064	10.609
19	(69,70)	0.905	0.383	0.269	0.519	0.060	8.650
20	(73,74)	0.633	0.461	1.179	0.757	0.089	8.506
21	(77,78)	2.226	0.936	1.468	1.543	0.079	19.532

Note: All $z$ s are significantly different from 0 at the 0.01 level.

It is clear from Table 6 that the estimates of d are all positive with the range of estimates being $0.174 < \hat{d} < 1.810$ . All 20 estimates are significantly different from 0, at a 0.01 level of significance. Thus, even though general tests of fit indicated that the data fitted the model (Gyagenda & Engelhard, 2009), the procedure for identifying response dependence described in this article diagnosed and quantified statistically significant dependence between Criteria 1 and 2 in all raters. This diagnosis suggests that, as reported in Humphry and Heldsinger (2009), to remove the dependence, a feature of the response format that may go beyond the training of raters needs to be considered.

Summary

Methods for identifying violations of local independence in unidimensional models of modern test theory have been considered in the literature. This article notes that there are two generic sources of violation of local independence; one results from multidimensionality, and the other from response dependence between pairs of items. In the presence of response dependence, which is the concern of this article, the response of one item governs the response of one or more items which are dependent on it.

The possibility of clarifying and understanding response dependence when using the Rasch model is helped by having a mathematical formulation of a process that creates such dependence, a formulation which has been specified in the literature only relatively recently (Marais & Andrich, 2008). Andrich and Kreiner (2010) took this formulation of response dependence and showed how, using the dichotomous Rasch model, an estimate of the magnitude of dependence between two dichotomous items could be summarized as a change in the difficulty of an item because of its dependence on another item.

This article generalized the procedure shown in Andrich and Kreiner (2010) to the case of response dependence between two polytomously scored items with the same maximum score. Specifically, response dependence was formulated as inducing a change in the range of the thresholds separating adjacent scores of the dependent item such that a given response in one item increases the probability of the same response in the dependent item. Second, the estimation procedure of the magnitude of dependence, which was also a generalization of Andrich and Kreiner (2010), was derived.

This procedure for estimating the magnitude of dependence was illustrated in two ways: first in a simulation study and, second, with real data. In the first study, data were simulated with and without dependence. A detailed illustrative analysis of the responses simulated to have dependence recovered the simulated value of dependence very accurately. In the case of no dependence, as expected, none of the estimates of dependence were statistically significantly different from 0.

The procedure was illustrated in a real data set that was analyzed comprehensively by Gyagenda and Engelhard (2009) using classical test theory and the PRM of modern test theory. The data consist of ratings of student’s writing on four criteria by 20 raters. General tests of fit, shown by Gyagenda and Engelhard, did not indicate general misfit to the PRM. For purposes of this article, an a priori hypothesis of response dependence of ratings of Criterion 2 on Criterion 1 for all 20 raters was tested. Statistically significant response dependence was diagnosed for all 20 raters. The significant magnitudes of response dependence ranged from 0.519 to 1.810 logits, a relatively large effect. This study provides an example where general tests of fit do not indicate misfit but where a specific test of a hypothesis of dependence shows significant misfit.

Although the discussion has focused on identifying and quantifying response dependence in terms of an increase in distance between thresholds defining scores, this article was not concerned with accounting for this dependence. There are a number of procedures in the literature for doing so, including forming a polytomous item by summing the dependent dichotomous items (Andrich, 1985; Kreiner, 2007; Kreiner & Christensen, 2007; Wang, Bradlow, & Wainer, 2002; Wilson & Adams, 1995). However, as noted in the introduction, the ideal is to have no response dependence. Therefore, identifying dependence is particularly important in studies when raters are being trained and where criteria and formats are clarified. The magnitude of the effect on the thresholds of an item because of its dependence on another item is very tangible evidence of the impact of dependence. The evidence in this article suggests that at least reexamining the intrinsic relationship between the first two criteria in the Gyagenda and Engelhard (2009) data set would be useful in understanding the dependence. In addition, response dependence among other criteria can be studied.

Footnotes

Tim Dunne and Josh McGrane read the paper and each made excellent suggestions. The data in the real example were provided by George Engelhard.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported in this article was supported in part by Pearson and in part by an Australian Research Council Linkage grants with the Curriculum Council of Western Australia, Pearson Research and Assessment, and the Australian National Ministerial Council on Employment, Education, Training and Youth Affairs’ Performance Measurement and Reporting Task Force as Industry Partners.

References

Andersen

E. B.

(1973). Conditional inference for multiple choice questionnaires. British Journal of Mathematical and Statistical Psychology, 26, 31-44.

Andrich

(1978). Relationships between the Thurstone and Rasch approaches to item scaling. Applied Psychological Measurement, 2, 449-460.

Andrich

(1985). A latent trait model for items with response dependencies: Implications for test construction and analysis. In Embretson

S. E.

(Ed.), Test design (pp. 245-275). New York, NY: Academic.

Andrich

(1991). Essay review of Rolf Langeheine and Jurgen Rost, Latent Trait and Latent ClassAnalysis, New York, 1988. Plenum Press. Psychometrika, 56, 155-168.

Andrich

(2005). The Rasch model explained. In Alagumalai

Curtis

D. D.

Hungi

(Eds.), Applied Rasch measurement: A book of exemplars (pp. 308-328). Dordrecht, The Netherlands. Springer-Kluwer.

Andrich

(2010). Sufficiency and conditional estimation of person parameters in the polytomous Rasch model. Psychometrika, 75, 292-308.

Andrich

Kreiner

(2010). Quantifying response dependence between two dichotomous items using the Rasch model. Applied Psychological Measurement, 34, 181-192.

Andrich

Luo

(2003). Conditional estimation in the Rasch model for ordered response categories using principal components. Journal of Applied Measurement, 4, 205-221.

Andrich

Sheridan

B. E.

Luo

(2011). RUMM2030: Rasch unidimensional models for measurement. Perth, Australia: RUMM Laboratory.

10.

Gyagenda

I. S.

Engelhard

(2009). Using classical and modern measurement theories to explore rater, domain, and gender influences on student writing ability. Journal of Applied Measurement, 10, 225-246.

11.

Haberman

S. J.

(2007). The interaction model. In von Davier

Carstensen

C. H.

(Eds.), Multivariate mixture distribution Rasch models: Extensions and applications. (pp. 201-216). New York, NY: Springer.

12.

Hedges

L. V.

Nowell

(1995). Sex differences in mental test scores, variability and number of high-scoring individuals. Science, 269, 41-45.

13.

Humphry

Heldsinger

(2009). Do rubrics help to inform and direct teaching practice? In O’Keefe

Webb

Hoad

K. A.

(Eds.), ACER research conference series (pp. 57-62). Sydney, Australia: ACER.

14.

Kreiner

(2007). Validity and objectivity: Reflections on the role and nature of Rasch Models. Nordic Psychology, 59, 268-298.

15.

Kreiner

Christensen

K. B.

(2004). Analysis of local dependence and multidimensionality in graphical loglinear Rasch models. Communications in Statistics, Theory and Methods, 33, 1239-1276.

16.

Kreiner

Christensen

K. B.

(2007). Validity and objectivity in health related scales: Analysis by graphical loglinear Rasch models. In von Davier

Carstensen

C. H.

(Eds.), Multivariate and mixture distribution Rasch models: Extensions and applications (pp. 329-346). New York, NY: Springer.

17.

Lazarsfeld

P. F.

Henry

N. W.

(1966). Readings in mathematical social science. Chicago, IL: Science Research.

18.

Linacre

J.M.

Wright

B.D.

(2002) Journal of Applied Measurement, 3, 484-509

19.

Lunz

M. E.

Wright

B. D.

Linacre

J. M.

(1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3, 331-345.

20.

Marais

Andrich

(2008). Formalising dimension and response violations of local independence in the unidimensional Rasch model. Journal of Applied Measurement, 9, 200-215.

21.

Mehrens

W. A.

Lehman

I. J.

(1991). Measurement and evaluation in education and psychology (4th ed.). New York, NY: Harcourt Brace.

22.

Myford

C. M.

Wolfe

E. W.

(2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. In Smith

E. V.

Jr. Smith

R. M.

(Eds.), Introduction to Rasch measurement (pp. 518-574). Maple Grove, MN: JamPress.

23.

Smith

(2002). Detecting and evaluating the impact of multidimensionality using item fit statistics and principal component analysis of residuals. Journal of Applied Measurement, 3, 205-231.

24.

Wang

Bradlow

E. T.

Wainer

(2002). A general Bayesian model for testlets: Theory and applications. Applied Psychological Measurement, 26, 109-128.

25.

Wilson

Adams

R. J.

(1995). Rasch models for item bundles. Psychometrika, 60, 181-198.

26.

Wright

B.D.

Masters

G.N.

(1982). Rating Scale Analysis: Rasch Measurement. Chicago: MESA Press.

27.

Wright

B.D.

(2003) Rack and Stack: Time 1 vs Time 2. Rasch Measurement Transactions. 17, 905 906.

28.

Zwinderman

A. H.

(1995). Pairwise parameter estimation in Rasch models. Applied Psychological Measurement, 19, 369-375.