Observed Score and True Score Equating Procedures for Multidimensional Item Response Theory

Abstract

The purpose of this research was to develop observed score and true score equating procedures to be used in conjunction with the multidimensional item response theory (MIRT) framework. Three equating procedures—two observed score procedures and one true score procedure—were created and described in detail. One observed score procedure was presented as a direct extension of unidimensional IRT (UIRT) observed score equating and is referred to as the “Full MIRT Observed Score Equating Procedure.” The true score procedure and the second observed score procedure incorporated unidimensional approximation procedures to equate exams using UIRT equating principles. These procedures are referred to as the “Unidimensional Approximation of MIRT True Score Equating Procedure” and the “Unidimensional Approximation of MIRT Observed Score Equating Procedure,” respectively. Three exams were used to conduct UIRT observed score and true score equating, MIRT observed score and true score equating, and equipercentile equating. The equipercentile equating procedure was conducted for the purpose of comparison because this procedure does not explicitly violate the IRT assumption of unidimensionality. Results indicated that the MIRT equating procedures performed more similarly to the equipercentile equating procedure than the UIRT equating procedures, presumably due to the violation of the unidimensionality assumption under the UIRT equating procedures.

Keywords

item response theory multidimensional item response theory test equating

Large-scale testing programs often create and administer parallel test forms because of item exposure and test security issues (Kolen & Brennan, 2004). Although parallel forms are constructed to be as similar as possible in terms of content and statistical specifications, often the exams differ in difficulty. As a result, an adjustment must be made to correct for differences in difficulty for examinee scores to be comparable across test forms. This adjustment process is known as “equating.”

The equating process should be designed to yield as unbiased of an equating relationship as possible. If unbiased relationships are derived from the equating process, examinees can be compared on parallel forms because differences in form difficulty have been taken into account. If the equating procedure yields biased results, scores on different forms cannot be compared: There is no common metric on which scores can be evaluated (Kolen & Brennan, 2004).

Equating is often conducted within an item response theory (IRT) framework (Kolen & Brennan, 2004). The success of an IRT equating is dependent, in part, on the statistical assumptions being met. Unidimensionality, or the concept that an exam measures only one latent trait, is a foundational assumption in unidimensional IRT (UIRT). If the test measures more than one latent trait but the responses are analyzed using a UIRT model, the resulting ability estimates and item parameter estimates may be highly biased depending on the other variables being measured (Ansley & Forsyth, 1985; Reckase, 1985). Consequently, using UIRT models and methods to equate tests that measure more than one latent trait may result in very biased equating relationships.

An extension of UIRT, known as multidimensional IRT (MIRT), has been developed for use with multidimensional data (Ackerman, 1996; Ackerman, Gierl, & Walker, 2003; Reckase, 1985, 2009). Although there is a growing body of research in the theory and applications of MIRT, little research has been conducted on linking and equating in the MIRT framework. In fact, whereas MIRT scale linking procedures have been developed to place item and ability parameter estimates on the same scale (Hirsch, 1989; Li & Lissitz, 2000; Min, 2003; Oshima, Davey, & Lee, 2000; Thompson, Nering, & Davey, 1997), no procedures have yet been developed to equate number-correct scores within the MIRT framework. The purpose of this research is to (a) develop theoretical foundations for conducting observed score and true score equating within the MIRT framework, (b) demonstrate application of these procedures to real test data, and (c) compare MIRT equating results to results produced by UIRT equating and traditional equipercentile equating using the same data set.

UIRT Scale Linking

Before UIRT observed score or true score equating can be performed, item parameters and examinee abilities must first be estimated. Due to the scale indeterminacy property of UIRT, any linear transformation of the ability scale will yield the same probabilistic relationships, assuming that the ability scale and item parameters are transformed. To resolve this indeterminacy issue, UIRT calibrations are typically specified to yield an ability distribution with mean of 0 and standard deviation of 1 (Kolen & Brennan, 2004).

When parallel forms are to be equated, parameters on the different forms can either be estimated at the same time (concurrent calibration) or in separate calibrations. If concurrent calibration is used, no scale linking methods need to be employed to ensure that parameter estimates are on the same scale; the parameter estimates are already on the same scale. If separate calibrations are conducted under the equivalent groups design, groups that were administered different forms are assumed to be equivalent on the measured trait. That is, groups are assumed to have equal means and standard deviations on the ability scale. If both calibrations are specified to yield an ability distribution with mean of 0 and standard deviation of 1, no scale linking method is necessary because means and standard deviations are specified to be equal across groups (Kolen & Brennan, 2004).

However, when item parameters and abilities are calibrated separately for groups of examinees who are different with respect to the measured trait (i.e., nonequivalent groups), the estimates will be placed on different, yet linearly related, ability scales. Therefore, a scale linking procedure must be conducted to place item parameter estimates and ability estimates from the new scale to that of the base scale before equating can be conducted. When separate calibrations are performed under the common-item nonequivalent groups design, the scale of the new form and the scale of the base form may differ in two aspects: The scales may differ in (a) origin (i.e., mean) and (b) unit of measurement (i.e., standard deviation). As the scales are linearly related, two coefficients ( $K$ and $L$ ) must be determined to transform new scale ( $A$ ) parameters to the base scale ( $B$ ) as follows:

θ_{Bi} = K θ_{Ai} + L,

a_{Bj} = \frac{a_{Aj}}{K},

b_{Bj} = K b_{Aj} + L,

and

c_{Bj} = c_{Aj} .

In this series of transformations, $a$ is item discrimination, $b$ is item difficulty, and $c$ is the lower asymptote parameter. The subscripts $A$ and $B$ are scale indicators and the subscripts $i$ and $j$ are person and item indicators, respectively. The parameters $K$ and $L$ are two linking coefficients that account for differences in scale origin and unit of measurement. Several methods exist for estimating these parameters—namely, the mean/mean, mean/sigma, Haebara, and Stocking–Lord procedures (Kolen & Brennan, 2004).

MIRT Scale Linking

Several procedures have been developed to link scales within the compensatory MIRT framework (Davey, Oshima, & Lee, 1996; Hirsch, 1989; Li & Lissitz, 2000; Min, 2003; Oshima et al., 2000; Thompson et al., 1997; Yon, 2006). A generalized δ-dimensional compensatory MIRT model is defined as having form

p_{ij} (θ_{i}) = c_{j} + (1 - c_{j}) H_{j} (a_{j}^{T} θ_{i} + d_{j}) \equiv c_{j} + (1 - c_{j}) H_{j} (\sum_{k = 1}^{δ} a_{jk} θ_{k} + d_{j}),

where “ $a_{j}^{T} = (a_{j 1}, a_{j 2}, \dots, a_{j δ})$ , $a_{j 1}, a_{j 2}, \dots, a_{j δ}$ are nonnegative and not all zero, and $H_{j} (x)$ is any non-decreasing [link] function with $H_{j}' (x) \geq 0$ for all x and $H_{j}' (x) H_{j *}' (x *)$ not being zero identically as $(x, x *)$ varies for $j, j * = 1, 2, \dots, n$ ” (the use of the asterisk $(*)$ implies a different function; Zhang &Stout, 1999a, p. 133). $H_{j} (\cdot)$ represents a link function for this family of models (Zhang, 1996; Zhang & Stout, 1999a; Zhang & Wang, 1998).

Two commonly used generalized δ-dimensional compensatory models include the multidimensional compensatory three-parameter logistic model (M3PL) and the multidimensional compensatory normal ogive model. The M3PL can be written as

p_{ij} (θ_{i}) = c_{j} + (1 - c_{j}) (\frac{exp [1.7 (a_{j}^{T} θ_{i} + d_{j})]}{1 + exp [1.7 (a_{j}^{T} θ_{i} + d_{j})]}),

with corresponding link function

H_{j} (θ_{i}) = c_{j} + (1 - c_{j}) (\frac{exp [1.7 θ_{i}]}{1 + exp [1.7 θ_{i}]}) .

The multidimensional compensatory normal ogive model can be written as

p_{ij} (θ_{i}) = c_{j} + (1 - c_{j}) Φ (a_{j}^{T} θ_{i} + d_{j}),

with corresponding link function,

H_{j} (θ_{i}) = c_{j} + (1 - c_{j}) Φ (\cdot),

where $Φ (\cdot)$ is the standard normal distribution function.

Whereas scale linking procedures in the UIRT framework estimate two coefficients ( $K$ and $L$ ) to adjust for differences in (a) origin and (b) unit of measurement, MIRT scale linking procedures typically estimate scalar coefficients and matrices to adjust for differences in (a) translation (similar to “origin” in UIRT), (b) dilation (similar to “unit of measurement” in UIRT), (c) rotation, and (d) correlation. That is, separate calibrations within the MIRT framework may place ability estimates and item parameter estimates on different scales due to rotational indeterminacy, correlation indeterminacy, and indeterminacy in origin and unit of measurement.

Depending on the linking procedure and the assumptions associated with each procedure, several different mathematical expressions exist for transforming item parameter estimates and ability estimates from the new scale ( $A$ ) to the base scale ( $B$ ). However, a generic expression is as follows (Davey et al., 1996; Li & Lissitz, 2000; Min, 2003; Oshima et al., 2000; Thompson et al., 1997):

θ_{Bi} = T^{- 1} θ_{Ai} + β,

a_{Bj}^{T} = a_{Aj}^{T} T,

d_{Bj} = d_{Aj} - a_{Aj}^{T} T β,

and

c_{Bj} = c_{Aj} .

In this series of equations, $θ_{i}$ is a vector of multidimensional ability estimates, $a_{j}^{T}$ is a discrimination parameter vector, $d_{j}$ is an index related to the difficulty parameter, $T$ represents an oblique or orthogonal matrix to account for rotational indeterminacy and dilation indeterminacy, and $β$ represents a matrix to account for translation indeterminacy (Davey et al., 1996; Li & Lissitz, 2000; Min, 2003; Oshima et al., 2000). Depending on the linking design, $β$ might be equal to $0$ (Thompson et al., 1997), $T$ might be the product of a rotation matrix and a single dilation parameter (Li & Lissitz, 2000), $T$ might be the product of a rotation matrix and a diagonal dilation matrix that accounts for separate dilation parameters on each dimension (Min, 2003), or $T$ might simply be a rotation matrix (i.e., with a diagonal dilation matrix equal to identity). Note that the equations used to perform scale linking in the MIRT framework are direct extensions of their unidimensional counterparts, with matrices replacing scalar coefficients.

UIRT Equating

If examinee scores are to be reported on the θ scale or on a linear transformation of the θ scale, then only an appropriate scale linking method is required. However, often it is desired to use the number-correct scale or a transformation of this scale for score reporting. In this situation, a procedure must be conducted to equate observed scores or true scores across parallel forms of an exam. Two methods that have been created for this purpose are UIRT observed score equating and UIRT true score equating.

UIRT Observed Score Equating

To conduct UIRT observed score equating, conditional observed score distributions, $f (x | θ_{i})$ , are first determined at each ability level ( $θ_{i}$ ) using a recursive formula such as the Lord–Wingersky method (Lord & Wingersky, 1984). Once the conditional observed score distribution is determined at each specified ability level, this distribution is then multiplied by the ability density ( $ψ (θ)$ ) and summed¹ over all ability levels to determine a marginal observed score distribution for each form. That is,

f (x) = \sum_{θ} f (x | θ) ψ (θ) .

Once marginal distributions ( $f (x)$ and $f (y)$ ) are determined for each form, the two forms can be equated using traditional equipercentile methods.

UIRT True Score Equating

UIRT true score equating relates true scores on the two forms using the UIRT definition of true score. First, a true score on the new form ( $τ_{A}$ ) is selected. Usually, this value is between the sum of the lower asymptote parameters and the total number of items on the new form; that is, $\sum_{j : A} c_{j} < τ_{A} < N_{A}$ , where $N_{A}$ represents the total number of items on the new form. Recall that true score at a particular ability level ( $θ_{i}$ ) is defined as the sum of the probabilities of obtaining a correct response for each item; that is, $τ_{A} (θ_{i}) = \sum_{j : A} p_{ij} (θ_{i}; a_{j}, b_{j}, c_{j})$ . To determine the $θ_{i}$ associated with the particular true score on the new form, an iterative procedure—such as the Newton–Raphson method—is typically used. Once the particular $θ_{i}$ is computed, this value is then substituted into the definition of true score on the base form; that is, $τ_{B} (θ_{i}) = \sum_{j : B} p_{ij} (θ_{i}; a_{j}, b_{j}, c_{j})$ . Typically, corresponding true scores on the base form are derived for each new form integer score.

Multidimensional IRT Equating

No procedures currently exist for conducting observed score or true score equating in the MIRT framework. As the purpose of the present research is to create such procedures, three new equating procedures—two observed score procedures and one true score procedure—are described below. First, an observed score equating procedure is presented as a direct extension of UIRT observed score equating. This procedure is referred to as the “Full MIRT Observed Score Equating Procedure.” The true score equating procedure and the second observed score equating procedure use the unidimensional approximation methods detailed by Zhang and colleagues (Zhang, 1996; Zhang & Stout, 1999a; Zhang & Wang, 1998), and are referred to as the “Unidimensional Approximation of MIRT True Score Equating Procedure” and the “Unidimensional Approximation of MIRT Observed Score Equating Procedure,” respectively.

Full MIRT Observed Score Equating

Relatively straightforward extensions can be implemented to conduct observed score equating in the MIRT framework. In the UIRT framework, conditional observed score distributions (i.e., $f (x | θ_{i})$ ) are first determined at each ability level ( $θ_{i}$ ). The MIRT analog is to derive conditional distributions for each combination of ability levels (i.e., $f (x | θ_{i})$ ), where $θ_{i}$ denotes the ability vector. Similar to the UIRT framework, these can be computed using the Lord–Wingersky algorithm. To implement this algorithm, however, a vector of ability levels is used in place of a single ability level as follows:

f_{r} (x | θ_{i}) = f_{r - 1} (x | θ_{i}) (1 - p_{ir}) x = 0

= f_{r - 1} (x | θ_{i}) (1 - p_{ir}) + f_{r - 1} (x - 1 | θ_{i}) p_{ir} 0 < x < r

= f_{r - 1} (x - 1 | θ_{i}) p_{ir} x = r

In this series of equations, $f_{r} (x | θ_{i})$ is the conditional observed score distribution over the first $r$ items, $f_{r - 1} (x | θ_{i})$ is the conditional observed score distribution over the first $r - 1$ items, and $p_{ir}$ is the probability of correctly answering item $r$ for an examinee of ability vector $θ_{i}$ . In practice, this algorithm begins as

f_{r} (x | θ_{i}) = (1 - p_{ir}) x = 0

= p_{ir} x = 1

After this initial step, $r - 1$ iterations are required to compute the conditional observed score distribution ( $f_{r} (x | θ_{i})$ ) for $r$ items.

In the UIRT framework, these conditional distributions are then multiplied by the ability density ( $ψ (θ)$ ) and summed over all ability levels to obtain a marginal observed score distribution for each form. The MIRT analog is to multiply the conditional distributions by the multivariate ability density ( $ψ (θ)$ ) and to sum over all ability spaces using a modified version of Equation 8, in which ability vectors replace scalar ability levels. In this research, the multivariate standard normal distribution with uncorrelated axes (i.e., $θ ~ MVN (0, I)$ ) was used to model the multivariate ability density. Similar to the unidimensional case, traditional equipercentile methods can then be used to equate the two forms.

Unidimensional Approximation

Both the unidimensional approximation of the MIRT true score equating procedure and the unidimensional approximation of the MIRT observed score equating procedure are conducted by estimating unidimensional item parameters and unidimensional ability distributions from the multidimensional data. Motivation for using unidimensional estimates, along with the unidimensional estimation procedures, appears in the following.

Several problems arise when trying to extend UIRT true score equating to the MIRT framework. In the UIRT framework, true score equating is conducted by relating the forms to be equated through the respective test characteristic curves (TCCs). Recall that the TCC relates the UIRT definition of true score, $τ (θ_{i}) = \sum_{j} p_{ij} (θ_{i}; a_{j}, b_{j}, c_{j})$ , to ability level $θ$ . In MIRT, true scores are related to ability levels through the test characteristic surface (TCS), which is the multidimensional equivalent of the TCC. For each combination of ability levels (corresponding to each dimension), the probabilities of obtaining correct responses to each item are summed to form true scores (i.e., $τ (θ_{i}) = \sum p (θ_{i})$ ). As such, an infinite number of combinations of ability levels is associated with a particular true score. The problem arises in that, when the TCS is computed for the base form, different combinations of ability levels corresponding to the new form true score may map to different true scores on the base form. That is, there is no unique solution for mapping true scores on new form to true scores on the base form.

The problem discussed earlier can be bypassed using the results presented in Zhang (1996), Zhang and Stout (1999a), and Zhang and Wang (1998). These authors demonstrated that any set of item responses that can be adequately modeled by a multidimensional compensatory model can be closely approximated by a UIRT model with estimated unidimensional ability and item parameters. Zhang (1996), Zhang and Stout (1999a), and Zhang and Wang (1998) defined the estimated unidimensional ability ( $θ_{α}$ , where $α$ denotes a unidimensional approximation) as a standardized linear composite of the latent variables ( $θ_{1}, θ_{2}, \dots, θ_{δ}$ ). The authors estimate this composite as

{\hat{θ}}_{α} = {\hat{α}}^{T} \hat{θ} = \sum_{k = 1}^{δ} {\hat{α}}_{k} {\hat{θ}}_{k},

where $\sum_{k = 1}^{δ} {\hat{α}}_{k}^{2} = 1$ (note the use of the $\land$ symbol to represent an estimate as opposed to a known value). The coefficients for the linear composite are estimated as

{\hat{α}}_{k} = ω \sum_{j = 1}^{N} w_{j} E {H_{j}' ({\hat{a}}_{j}^{T} f \hat{Θ}) {[\sum_{j = 1}^{N} w_{j}^{2} H_{j} ({\hat{a}}_{j}^{T} f \hat{Θ}) [1 - H_{j} ({\hat{a}}_{j}^{T} f \hat{Θ})]]}^{- \frac{1}{2}}} {\hat{a}}_{jk}

= ω \sum_{j = 1}^{N} w_{j} E [\frac{H_{j}' ({\hat{a}}_{j}^{T} f \hat{Θ})}{\sqrt{Var (Y | f \hat{Θ})}}] {\hat{a}}_{jk}, k = 1, 2, \dots, δ,

where $f \hat{θ}$ is the complete latent trait vector (i.e., the latent trait vectors across all examinees), $w_{j}$ is the score weight, $ω$ is a positive constant such that $\sum_{k = 1}^{δ} {\hat{α}}_{k}^{2} = 1$ , and $E$ is the expectation operator. Under the assumption that all terms

w_{j} E [\frac{H_{j}' ({\hat{a}}_{j}^{T} f \hat{Θ})}{\sqrt{Var (Y | f \hat{Θ})}}], j = 1, 2, \dots, N

are equal, this becomes

{\hat{α}}_{k} = \frac{\sum_{j = 1}^{N} {\hat{a}}_{jk}}{\sqrt{\sum_{k = 1}^{δ} {(\sum_{j = 1}^{N} {\hat{a}}_{jk})}^{2}}},

where $N$ is the total number of items on the test. The terms $w_{j} E [H_{j}' ({\hat{a}}_{j}^{T} f \hat{θ}) / \sqrt{Var (Y | f \hat{θ})}]$

may be considered to be “compound weights” for items contributing to the test direction; each term is completely determined by the score weight $w_{j}$ and the item model (theoretical) weight $E [\frac{H_{j}' ({\hat{a}}_{j}^{T} f \hat{θ})}{\sqrt{Var (Y | f \hat{Θ})}}]$ which is mainly determined by the derivative of the link function and the item discrimination parameter vector. (Zhang, 1996, p. 25)

When using the multidimensional normal ogive model, corresponding unidimensional item parameter estimates are derived as a function of the linear composite coefficients ( $α$ ), the multidimensional item discrimination vector ( $a_{j}^{T}$ ), the index related to multidimensional item difficulty ( $d_{j}$ ), and the multidimensional ability covariance matrix ( $Σ$ ) in the population of interest as follows:

P (U_{j} = 1 | {\hat{θ}}_{α}) = E (U_{j} | {\hat{θ}}_{α}) = Φ ({\hat{a}}_{α j} {\hat{θ}}_{α} + {\hat{d}}_{α j}),

where $Φ$ is the standard normal cumulative distribution function (CDF) and

{\hat{a}}_{α j} = {(1 + {\hat{σ}}_{α j}^{2})}^{- \frac{1}{2}} {\hat{a}}_{j}^{T} \hat{Σ} \hat{α},

{\hat{d}}_{α j} = {(1 + {\hat{σ}}_{α j}^{2})}^{- \frac{1}{2}} {\hat{d}}_{j},

{\hat{b}}_{α j} = \frac{- {\hat{d}}_{α j}}{{\hat{a}}_{α j}},

{\hat{c}}_{α j} = {\hat{c}}_{j},

and

{\hat{σ}}_{α j}^{2} = {\hat{a}}_{j}^{T} \hat{Σ} {\hat{a}}_{j} - {({\hat{a}}_{j}^{T} \hat{Σ} \hat{α})}^{2} .

The authors continue by noting that the UIRT true score ( $T_{α}$ ) associated with the linear composite ( $θ_{α}$ )—which is the sum of the probabilities of obtaining correct responses over all items at each composite ability level—is expressed as

T_{α} = ξ (θ_{α}) \equiv ξ (\sum_{k = 1}^{δ} α_{k} θ_{k}) .

This expression preserves the property of UIRT true scores in that the function $ξ (\cdot)$ is strictly increasing.

The previous results imply that (a) a unidimensional composite ability ( $θ_{α}$ ) that approximates the ability which the test most closely measures can be constructed from any generalized δ-dimensional compensatory model and that (b) unidimensional item parameters corresponding to the unidimensional composite ability can be estimated. Consequently, a unidimensional TCC can be constructed that relates unidimensional composite ability level ( $θ_{α}$ ) to composite true score ( $T_{α}$ ).

Unidimensional Approximation of MIRT True Score Equating

Using the unidimensional approximations provided earlier, UIRT true score equating procedures can be conducted to relate composite true scores ( $T_{α}$ ) on the multidimensional test forms. First, a true score on new form, $τ_{α A}$ , can be selected for which the corresponding composite true score on the base form is desired. Using an iterative procedure (specifically, the Newton–Raphson method), the corresponding unidimensional composite ability level $θ_{α}$ associated with this new form true score can be obtained by minimizing the difference,

func (θ_{α i}) = τ_{α A} - \sum_{j : A} p_{ij} (θ_{α i}; a_{α j}, b_{α j}, c_{j}) .

Finally, using the UIRT definition of true score, the composite true score on the base form associated with the new form composite true score can be computed as

τ_{α B} (θ_{α i}) = \sum_{j : B} p_{ij} (θ_{α i}; a_{α j}, b_{α j}, c_{j}) .

Unidimensional Approximation of MIRT Observed Score Equating

To conduct unidimensional approximation of MIRT observed score equating, unidimensional item parameters and abilities can first be estimated using the methodology described earlier. Conditional distributions $f (x | θ_{α})$ can then be determined at each composite ability level ( $θ_{α}$ ) using the Lord–Wingersky recursive formula. The conditional distributions can then be multiplied by the estimated unidimensional ability distribution in the population of examinees and summed across the estimated unidimensional ability space using Equation 8. Once marginal distributions are computed for each form, the forms can be equated using traditional equipercentile procedures.

Data and Procedures

The data used in this study were collected under the equivalent groups equating design and came from two forms of the Iowa Tests of Educational Development (ITED; Forsyth, Ansley, Feldt, & Alnot, 2001), Level 17/18 battery. Specifically, each of the following tests within these batteries was equated: (a) Mathematics: Concepts and Problem Solving, (b) Analysis of Science Materials, and (c) Analysis of Social Studies Materials. The sample size for each form was 2,500.

These three sets of exams were selected to conduct the analyses on exams that differed in dimensionality. Specifically, the mathematics and social studies exams each measured four distinct dimensions according to the ITED content specifications, whereas the science exams each measured two distinct dimensions. Both mathematics exams contained 40 items, with 19, 10, 7, and 4 items per domain, respectively. Both social studies exams contained 50 items, with 16, 16, 10, and 8 items per domain, respectively. Finally, both science exams contained 48 items, with 27 and 21 items per domain, respectively. DETECT (Zhang & Stout, 1999b) was used for the purpose of dimensionality assessment. Results revealed that each of the test forms used in this analysis was “weak to moderately” multidimensional based on the DETECT value (all DETECT values ranged between 0.22 and 0.30). Therefore, the results in this study are based on exams that display a moderate multidimensional structure.

Each of the following procedures was conducted to equate the new form and base form for each exam: (a) UIRT observed score equating, (b) UIRT true score equating, (c) full MIRT observed score equating, (d) unidimensional approximation of MIRT observed score equating, (e) unidimensional approximation of MIRT true score equating, and (f) equipercentile equating. The equipercentile equating procedure was conducted for the purpose of comparison with the other procedures.

TESTFACT (Bock et al., 2003) was used to estimate unidimensional and multidimensional item parameters. The TESTFACT program essentially conducts a factor analysis procedure on interitem tetrachoric correlations using the marginal maximum likelihood procedure (Bock & Aitkin, 1981). The resulting parameterization is in accordance with either the unidimensional or multidimensional normal ogive model. These parameters were then substituted into the two-parameter logistic model by noting the fact that these models yield nearly identical probabilities. That is,

p_{α ij} (θ_{α i}) = Φ ({\hat{a}}_{α j} (θ_{α i} - {\hat{b}}_{α j})) \approx \frac{ex p^{1.7 {\hat{a}}_{α j} (θ_{α i} - {\hat{b}}_{α j})}}{1 + ex p^{1.7 {\hat{a}}_{α j} (θ_{α i} - {\hat{b}}_{α j})}},

where the first probability is parameterized according to the normal ogive model and the second probability is parameterized according to the logistic model. For the unidimensional procedures, only one dimension was specified for the TESTFACT calibrations. For the multidimensional procedures, each TESTFACT calibration was specified to yield an orthogonal solution, with the number of dimensions equal to the number of content categories identified on the ITED form.

Given that the DETECT results indicated “weak to moderate” multidimensionality in each form, it was decided to specify the number of dimensions equal to the number of content categories for two reasons. First, the number of specified dimensions would mirror the ITED framework. Second, and most important, systematic bias in estimation is only introduced when too few dimensions are specified in the multidimensional calibration; no systematic bias is introduced if the number of dimensions is overspecified (Reckase, 2009). Therefore, it was more important to overspecify than to underspecify the number of dimensions; given that the dimensionality assessment results indicated “weak to moderate” multidimensionality, specifying four dimensions for math and social studies and two dimensions for science seemed sufficient.

As the data used in this study were collected under the equivalent groups design, no scale linking procedures were required for the unidimensional procedures: The item parameter estimates were assumed to be on the same scale. After item parameters were estimated, the parameters were then used as input for the computer program PIE (a computer Program for UIRT Equating; Hanson & Zeng, 1995) to conduct unidimensional observed score and true score equating procedures. For the multidimensional procedures, however, item parameter estimates on the new form were first rotated to the scale of the base form to account for rotational indeterminacy (Thompson et al., 1997) before the equating procedures were conducted. Given that the equivalent groups design was used, only rotational indeterminacy had to be accounted for.²

To conduct the unidimensional approximation of the MIRT true score equating procedure and the unidimensional approximation of the MIRT observed score equating procedure, the unidimensional item parameters were first estimated employing the unidimensional approximation methodology (Equations 13a-13e) using the computer program R version 2.15.2 (R Development Core Team, 2013). These values were incorporated in the computer program PIE to conduct the observed score and the true score equating procedures.

To conduct the full MIRT observed score equating procedure, conditional observed score distributions were first determined for each combination of $θ$ values (using multivariate normal quadrature points) using a modified version of the Lord–Wingersky algorithm for the MIRT framework. The multivariate normal quadrature weights ( $ψ (θ)$ ) were then multiplied by the conditional observed score distributions ( $f (x | θ)$ ) and summed over the ability space to determine an estimated marginal observed score distribution. After this procedure was conducted for both forms, traditional equipercentile equating was conducted to equate the forms. The entire full MIRT observed score equating procedure was conducted using the computer programs, R version 2.15.2 (R Development Core Team, 2013) and RAGE-RGEQUATE (Zeng, Kolen, Hanson, Cui, & Chien, 2004). Specifically, R code (available from author upon request) was created to determine the conditional observed score distributions and the marginal observed score distribution for each form, and RAGE-RGEQUATE was used to conduct the equipercentile equating.

Scale Linking and Equating Assumptions

Several assumptions were made to conduct the MIRT equating procedures previously described. First, to compute unidimensional approximation item parameter estimates, the multidimensional ability distribution in the population of examinees was assumed to follow a multivariate normal distribution (i.e., $θ ~ MVN (0, Σ)$ ). Furthermore, to estimate the unidimensional approximation ability parameters, an assumption was made concerning the equivalence of item weights (see Equation 11c). The robustness of this assumption has not yet been empirically investigated, though other authors have permitted this assumption for other calculations (Zhang, 1996; Zhang & Stout, 1999a; Zhang & Wang, 1998).

For the full MIRT observed score equating procedure, the multivariate quadrature distribution was specified to follow the multivariate standard normal distribution with uncorrelated axes (i.e., $θ ~ MVN (0, I)$ ). This model specification is consistent with the orthogonal TESTFACT calibrations. Although the multidimensional procedures were conducted under an orthogonal, complex structure solution (i.e., items were allowed to measure more than one trait), the multidimensional procedures could have been conducted under a simple structure solution. In this situation, each item would be assumed to measure only one trait, and a correlation coefficient would be estimated for each pair of traits. However, the results from the dimensionality assessment revealed that this assumption was not reasonable for this data set, as the matrix of item responses did not adequately conform to simple structure.

Last, it was discovered that the MIRT scale linking procedures under the random groups design are not required to conduct the MIRT observed score and true score equating procedures. That is, the same equating relationships would result regardless of whether the orthogonal rotation was first incorporated to account for rotational indeterminacy. This will only hold when item parameters and ability estimates are calibrated with respect to orthogonal reference axes (i.e., the solution must be orthogonal as opposed oblique), and when the specified variance–covariance matrix for the ability estimates is the identity matrix (i.e., each measured trait must be specified as uncorrelated with other measured traits and of unit length). If either of these conditions does not hold, then the scale linking procedures under the random groups design must first be conducted.

The reason is that under an orthogonal rotation, the MIRT difficulty parameter ( $d$ ) will remain the same, though the discrimination parameters will change. However, the overall discrimination power for each item—which is related to the geometric length of the item as represented in multidimensional space—will not change because each item is only rotated orthogonally as opposed to being dilated as well. Furthermore, under the orthogonal rotation, the direction of best measurement for each item will be rotated the exact same amount. When conducting the Full MIRT observed score procedure, given that (a) the MIRT difficulty parameters will not change under an orthogonal rotation and (b) the geometric length of each item remains the same in multidimensional space, the conditional distribution ( $f (x | θ)$ ) at any set of quadrature points prior to rotation will be identical to the conditional distribution at the rotated set of quadrature points after the rotation is applied. Furthermore, because the ability density ( $ψ (θ)$ ) is specified to follow a multivariate standard normal distribution with zero correlation between dimensions, all vectors of quadrature points that maintain the same Euclidean distance from the origin of the reference axes also have the same probability density. Therefore, after the orthogonal rotation is applied, the ability density associated with the set of quadrature points prior to rotation will be identical to the ability density associated with the rotated set of quadrature points after the rotation is applied. Given that each conditional distribution is the same prior to rotation and after the rotation is applied (though each conditional distribution now appears at the rotated vector of quadrature points), and given that the ability density corresponding to each conditional distribution is the same prior to rotation and after the rotation is applied, the term $f (x | θ) ψ (θ)$ will be identical for the set of quadrature points before the rotation is applied and for the rotated set of quadrature points. Therefore, the marginal distribution will be identical for each form.

Concerning the unidimensional approximation procedures, recall that unidimensional item parameters are first estimated given the multidimensional parameter estimates. To compute unidimensional item parameters, the direction of best measurement is first determined at the test level and then unidimensional discrimination and difficulty parameters are computed. Under an orthogonal rotation, the direction of best measurement for each item will be rotated by exactly the same angle. Furthermore, the test-level direction of best measurement will be rotated by the same angle as the item-level directions of best measurement. The unidimensional item parameter estimates are primarily governed by the item discrimination vector, the test-level direction of best measurement, and the covariance matrix of ability estimates. As the test-level direction of best measurement and the item discrimination vector are rotated by the same angle, and the covariance matrix of ability estimates is the identity matrix in this situation, the resulting unidimensional item parameters will be the same prior to an orthogonal rotation and after an orthogonal rotation.

Evaluation Criteria

Several problems arose when trying to determine an appropriate criterion by which the performance of each equating procedure could be evaluated. First, the “true” equating relationship between each pair of forms was unknown; this would only be known if the true relationship was specified in a simulation study. Even if the true equating relationship was specified, however, this relationship would still differ between observed score procedures and true score procedures. Specifically, UIRT observed score procedures and UIRT true score procedures are not necessarily expected to yield the same results, given that they are defined differently. True score equating relates true scores on both forms to be equated; although no theoretical justification exists for applying the true score equating results to observed scores, often this is conducted in practice. Observed score equating, on the contrary, provides a statistical adjustment such that the observed score distributions on each form are as similar as possible.

As no perfect criterion existed for evaluating the performance of each equating procedure, the equipercentile equating procedure was used as a benchmark for comparison for the UIRT and MIRT equating procedures. As the assumptions associated with the equipercentile procedure are not expected to be violated in this study (as the UIRT procedures are expected to violate the unidimensionality assumption), this procedure might provide a better indication of how well each of the other equating procedures performs.

Differences and absolute differences between equated scores for the equipercentile procedure and each of the other five procedures were calculated to estimate how well each procedure performed. These values were also averaged across all score points to provide a single summary statistic for each procedure. Each difference was evaluated against the Difference That Matters (DTM) criterion (Dorans, Holland, Thayer, & Tateneni, 2003). Although Dorans et al. (2003) defined the DTM in terms of test linking procedures and in terms of subgroups of examinees, the DTM concept was extended in this study to serve as a criterion for strict parallel form equating across different methods of equating. Specifically, Dorans et al. define the DTM as a 0.5 number-correct score difference between linking results. In this study, the absolute differences were compared with the 0.5 criterion.

Results

Differences between each of the UIRT and MIRT equating procedures and the equipercentile procedure appear in Tables 1 to 3 for math, science, and social studies, respectively. The mean of the differences and the mean of the absolute value of each difference appear at the bottom of each table. Given that the equipercentile equating was considered as the benchmark for comparison in this study, these tables provide an indication of how well each procedure performed. Furthermore, these differences are plotted in Figures 1 to 3 for math, science, and social studies, respectively. These figures provide a visual representation of the numerical data presented in Tables 1 to 3. Each plot contains the difference between equated scores on the vertical axis and the new form number-correct scores on the horizontal axis. Dashed horizontal lines appear at values of −0.5 and 0.5 to indicate differences that exceeded the DTM criterion.

Table 1.

Differences With Equipercentile Equating Procedure for Math Exams.

	Observed score		True score
Number-correct score	UIRT	Approximation	Full MIRT	UIRT	Approximation
0	−0.033	−0.040	−0.025	0.014	0.014
1	−0.086	−0.086	−0.081	−0.035	0.003
2	−0.182	−0.160	−0.183	−0.117	−0.093
3	−0.318	−0.251	−0.301	−0.250	−0.195
4	−0.390	−0.260	−0.334	−0.328	−0.209
5	−0.440	−0.237	−0.324	−0.392	−0.193
6	−0.497	−0.216	−0.302	−0.460	−0.179
7	−0.560	−0.201	−0.275	−0.532	−0.171
8	−0.624	−0.192	−0.246	−0.599	−0.166
9	−0.672	−0.182	−0.216	−0.655	−0.160
10	−0.705	−0.168	−0.183	−0.697	−0.150
11	−0.723	−0.149	−0.143	−0.720	−0.132
12	−0.727	−0.123	−0.100	−0.728	−0.109
13	−0.720	−0.094	−0.057	−0.724	−0.085
14	−0.706	−0.066	−0.020	−0.713	−0.063
15	−0.690	−0.045	0.012	−0.698	−0.045
16	−0.673	−0.031	0.037	−0.683	−0.034
17	−0.658	−0.024	0.053	−0.669	−0.030
18	−0.649	−0.025	0.054	−0.658	−0.032
19	−0.646	−0.034	0.042	−0.654	−0.043
20	−0.652	−0.051	0.021	−0.658	−0.061
21	−0.665	−0.075	−0.008	−0.669	−0.086
22	−0.688	−0.106	−0.047	−0.689	−0.117
23	−0.716	−0.142	−0.089	−0.715	−0.153
24	−0.750	−0.183	−0.141	−0.746	−0.193
25	−0.785	−0.225	−0.197	−0.777	−0.233
26	−0.817	−0.266	−0.250	−0.804	−0.272
27	−0.842	−0.305	−0.300	−0.820	−0.307
28	−0.856	−0.338	−0.343	−0.820	−0.335
29	−0.855	−0.363	−0.376	−0.798	−0.354
30	−0.839	−0.380	−0.401	−0.750	−0.363
31	−0.796	−0.391	−0.414	−0.676	−0.362
32	−0.736	−0.396	−0.421	−0.579	−0.351
33	−0.670	−0.400	−0.420	−0.467	−0.333
34	−0.615	−0.407	−0.420	−0.353	−0.308
35	−0.583	−0.411	−0.417	−0.251	−0.278
36	−0.557	−0.401	−0.403	−0.177	−0.240
37	−0.491	−0.365	−0.360	−0.120	−0.169
38	−0.486	−0.365	−0.359	−0.140	−0.128
39	−0.548	−0.427	−0.418	−0.205	−0.134
40	−0.604	−0.510	−0.500	0.111	0.111
Mean	−0.616	−0.222	−0.216	−0.522	−0.164
Abs. Mean	0.616	0.222	0.227	0.528	0.171
Wt. Mean	−0.678	−0.159	−0.146	−0.653	−0.144
Wt. Abs.	0.678	0.159	0.162	0.653	0.144

Note: UIRT = unidimensional item response theory; MIRT = multidimensional IRT; Mean = unweighted mean difference across all score points; Abs. Mean = unweighted mean absolute difference across all score points; Wt. Mean = weighted mean difference across all score points; Wt. Abs. = weighted mean absolute difference across all score points.

Table 2.

Differences With Equipercentile Equating Procedure for Science Exams.

	Observed score			True score
Number-correct score	UIRT	Approximation	Full MIRT	UIRT	Approximation
0	0.244	0.333	0.342	−0.004	−0.004
1	0.373	0.464	0.504	0.325	0.364
2	0.457	0.541	0.602	0.437	0.519
3	0.518	0.599	0.675	0.510	0.601
4	0.567	0.640	0.731	0.562	0.648
5	0.607	0.666	0.780	0.600	0.675
6	0.613	0.653	0.781	0.604	0.661
7	0.577	0.594	0.714	0.567	0.600
8	0.537	0.524	0.611	0.525	0.529
9	0.492	0.448	0.481	0.483	0.453
10	0.446	0.370	0.332	0.441	0.375
11	0.401	0.293	0.185	0.401	0.297
12	0.355	0.218	0.075	0.361	0.224
13	0.308	0.148	0.045	0.321	0.155
14	0.261	0.083	0.070	0.279	0.091
15	0.210	0.023	0.088	0.234	0.033
16	0.158	−0.032	0.068	0.185	−0.019
17	0.102	−0.081	0.020	0.131	−0.067
18	0.043	−0.125	−0.050	0.073	−0.110
19	−0.017	−0.163	−0.127	0.012	−0.147
20	−0.077	−0.194	−0.190	−0.051	−0.179
21	−0.137	−0.220	−0.240	−0.115	−0.205
22	−0.197	−0.241	−0.272	−0.179	−0.227
23	−0.254	−0.256	−0.284	−0.243	−0.243
24	−0.311	−0.267	−0.284	−0.306	−0.256
25	−0.363	−0.271	−0.280	−0.366	−0.264
26	−0.414	−0.273	−0.284	−0.423	−0.268
27	−0.459	−0.270	−0.285	−0.477	−0.269
28	−0.497	−0.262	−0.276	−0.525	−0.266
29	−0.530	−0.250	−0.271	−0.567	−0.259
30	−0.557	−0.234	−0.238	−0.603	−0.248
31	−0.576	−0.213	−0.182	−0.631	−0.234
32	−0.587	−0.188	−0.109	−0.649	−0.215
33	−0.589	−0.157	−0.032	−0.657	−0.191
34	−0.576	−0.121	0.029	−0.653	−0.162
35	−0.549	−0.079	0.054	−0.635	−0.126
36	−0.509	−0.029	0.038	−0.602	−0.082
37	−0.452	0.029	0.023	−0.549	−0.029
38	−0.377	0.097	0.054	−0.476	0.037
39	−0.281	0.178	0.124	−0.377	0.117
40	−0.160	0.276	0.228	−0.249	0.216
41	−0.012	0.394	0.354	−0.084	0.340
42	0.164	0.538	0.490	0.124	0.496
43	0.370	0.714	0.651	0.386	0.695
44	0.602	0.930	0.857	0.714	0.950
45	0.830	1.161	1.092	1.105	1.258
46	0.839	1.086	1.044	1.293	1.341
47	0.381	0.528	0.538	1.360	1.174
48	−0.182	−0.140	−0.150	0.421	0.421
Mean	0.037	0.173	0.186	0.062	0.188
Abs. Mean	0.390	0.339	0.331	0.446	0.354
Wt. Mean	0.081	0.076	0.075	0.085	0.080
Wt. Abs.	0.323	0.227	0.199	0.347	0.230

Table 3.

Differences With Equipercentile Equating Procedure for Social Studies Exams.

	Observed score			True score
Number-correct score	UIRT	Approximation	Full MIRT	UIRT	Approximation
0	−0.166	0.035	0.030	0.026	0.026
1	−0.317	0.091	0.101	−0.169	0.113
2	−0.511	0.124	0.149	−0.333	0.216
3	−0.702	0.125	0.152	−0.534	0.233
4	−0.896	0.095	0.114	−0.751	0.203
5	−1.097	0.041	0.048	−0.973	0.142
6	−1.296	−0.030	−0.031	−1.191	0.063
7	−1.451	−0.078	−0.084	−1.364	0.008
8	−1.527	−0.062	−0.070	−1.454	0.017
9	−1.590	−0.047	−0.055	−1.528	0.022
10	−1.639	−0.036	−0.042	−1.588	0.025
11	−1.672	−0.026	−0.024	−1.632	0.028
12	−1.691	−0.016	−0.003	−1.660	0.033
13	−1.695	−0.004	0.023	−1.670	0.040
14	−1.683	0.014	0.054	−1.664	0.050
15	−1.656	0.035	0.091	−1.642	0.065
16	−1.611	0.061	0.135	−1.604	0.085
17	−1.554	0.090	0.182	−1.552	0.109
18	−1.486	0.120	0.229	−1.487	0.135
19	−1.410	0.151	0.273	−1.415	0.162
20	−1.329	0.181	0.309	−1.337	0.187
21	−1.248	0.207	0.340	−1.258	0.208
22	−1.168	0.225	0.365	−1.178	0.223
23	−1.090	0.237	0.381	−1.102	0.231
24	−1.019	0.240	0.385	−1.030	0.231
25	−0.955	0.233	0.381	−0.965	0.222
26	−0.899	0.217	0.366	−0.908	0.204
27	−0.855	0.190	0.339	−0.862	0.176
28	−0.822	0.153	0.303	−0.827	0.137
29	−0.803	0.103	0.256	−0.804	0.087
30	−0.796	0.044	0.198	−0.794	0.027
31	−0.804	−0.026	0.123	−0.798	−0.043
32	−0.824	−0.104	0.036	−0.815	−0.122
33	−0.857	−0.191	−0.064	−0.845	−0.208
34	−0.902	−0.284	−0.172	−0.885	−0.300
35	−0.955	−0.379	−0.287	−0.935	−0.394
36	−1.016	−0.476	−0.409	−0.993	−0.490
37	−1.082	−0.573	−0.536	−1.056	−0.585
38	−1.152	−0.668	−0.664	−1.122	−0.677
39	−1.221	−0.757	−0.788	−1.188	−0.763
40	−1.286	−0.838	−0.906	−1.247	−0.838
41	−1.342	−0.907	−1.013	−1.294	−0.899
42	−1.385	−0.959	−1.106	−1.321	−0.939
43	−1.409	−0.984	−1.172	−1.317	−0.953
44	−1.411	−0.981	−1.197	−1.264	−0.929
45	−1.348	−0.943	−1.175	−1.134	−0.850
46	−1.225	−0.866	−1.095	−0.881	−0.688
47	−1.096	−0.739	−0.977	−0.444	−0.426
48	−1.044	−0.737	−0.892	0.186	−0.141
49	−1.005	−0.777	−0.842	1.086	0.483
50	−0.877	−0.645	−0.576	0.224	0.224
Mean	−1.154	−0.199	−0.173	−1.006	−0.114
Abs. Mean	1.154	0.317	0.383	1.065	0.287
Wt. Mean	−1.299	−0.052	0.017	−1.273	−0.033
Wt. Abs.	1.299	0.211	0.282	1.277	0.212

Figure 1.

Differences with equipercentile equating procedure for mathematics.

Figure 2.

Differences with equipercentile equating procedure for science.

Figure 3.

Differences with equipercentile equating procedure for social studies.

After equating each of the three sets of exams and investigating the performance of each equating procedure, three themes were identified and will be described below in detail: (a) both unidimensional procedures performed similarly, and all three multidimensional procedures performed similarly; (b) unidimensional procedures and multidimensional procedures performed differently, though the pattern of equated scores was similar for both types of procedures; and (c) multidimensional procedures tended to perform more similar to the equipercentile equating procedure than the unidimensional procedures.

Similarities Within Psychometric Framework

The most significant differences between the five equating procedures were not necessarily by procedure type (i.e., observed score vs. true score) but rather by psychometric framework (i.e., unidimensional vs. multidimensional). Both of the unidimensional procedures performed similarly, and each of the three multidimensional procedures performed similarly. These results were not unexpected, as the UIRT procedures violated the unidimensionality assumption and therefore were expected to contain systematic bias. These trends are apparent in Figures 1 to 3, which depict the difference between the equipercentile equating results and each of the other respective equating results.

The fact that all three MIRT procedures performed very similarly—despite the fact that one of these procedures was based on a full MIRT model and the other two procedures incorporate multidimensional parameter estimates to approximate a UIRT model—is especially noteworthy. From this initial investigation, it might appear as if the unidimensional approximation methods derived by Zhang and colleagues (Zhang, 1996; Zhang & Stout, 1999a; Zhang & Wang, 1998) performed very well at approximating a MIRT model using unidimensional item parameters. However, this conclusion may be premature. That is, it is currently unknown as to how well these procedures approximate a full MIRT model, or why the full MIRT procedure and the unidimensional approximation procedures performed so similarly.

Similar Trends Across Psychometric Frameworks

A second noticeable theme in this study was that although the unidimensional equating procedures and the multidimensional equating procedures performed differently, equating trends were similar for both types of procedures. For example, at points along the score scale where the unidimensional procedures revealed greater differences between new form and base form exams, the multidimensional procedures also tended to reveal greater differences between the two forms.

Before examining this occurrence in detail and offering explanations as to why this occurred, the general notion of equating scores along a single number-correct continuum should be addressed. Given that all of the equating procedures conducted in this research attempted to equate scores along a single, number-correct continuum, the UIRT and the MIRT procedures might be expected to perform somewhat similarly to begin with. That is, although UIRT and MIRT models were used to facilitate the equating process—and although multiple dimensions were taken into account—the scores were still equated along a single, number-correct continuum. As a result, similar trends might be expected across the score scale regardless of the psychometric framework.

To investigate the similarities in trends across the score scale, however, unidimensional item parameter estimates were compared with the unidimensional approximation item parameter estimates. Note that in this study, three sets of item parameters were obtained for each item: unidimensional item parameter estimates, multidimensional item parameter estimates, and unidimensional approximation item parameter estimates. Direct comparisons between unidimensional item parameter estimates and multidimensional item parameter estimates are difficult to evaluate, given that the item parameters are used in conjunction with different psychometric models. However, the unidimensional item parameter estimates and the unidimensional approximation item parameter estimates can be directly compared given that both sets of parameters are used in conjunction with the unidimensional logistic model.

Table 4 provides summary statistics for the UIRT and unidimensional approximation discrimination and difficulty parameter estimates for each form used in this study. Table 5 provides the correlation between each set of corresponding parameter estimates. Overall, both sets of item parameter estimates (unidimensional item parameter estimates and unidimensional approximation item parameter estimates) appear to be very similar. The correlations between unidimensional discrimination parameters and unidimensional approximation discrimination parameters ranged from .964 to .990, and the correlations between unidimensional difficulty and unidimensional approximation difficulty parameter estimates ranged from .997 to .999. Although there was a strong linear relationship between both sets of item parameter estimates (as evidenced by the correlations), the magnitude of the parameter estimates was slightly different. Specifically, unidimensional approximation difficulty and discrimination parameter estimates tended to be slightly lower than their UIRT counterparts. Differences in these parameter estimates were smaller for the science exams than for the math and the social studies exams; it should be noted that the unidimensional procedures and the unidimensional approximation procedures performed more similarly for the science exams than for the math and the social studies exams.

Table 4.

Summary Statistics for Unidimensional and UA Item Parameter Estimates.

	Discrimination			Difficulty
	Median	M	SD	Median	M	SD
UIRT math new	0.989	1.004	0.412	0.570	0.804	1.156
UA math new	0.999	1.009	0.380	0.505	0.709	1.062
UIRT math base	0.967	0.978	0.405	0.877	1.145	1.115
UA math base	0.954	0.966	0.365	0.845	1.096	1.126
UIRT science new	1.015	0.986	0.369	0.449	0.977	1.410
UA science new	1.003	0.957	0.335	0.434	0.953	1.334
UIRT science base	0.935	0.913	0.402	0.535	0.867	1.112
UA science base	0.924	0.886	0.374	0.521	0.850	1.093
UIRT social studies new	0.929	0.975	0.361	0.357	0.538	1.004
UA social studies new	0.972	0.984	0.331	0.290	0.426	1.141
UIRT social studies base	0.837	0.896	0.344	0.508	0.809	1.089
UA social studies base	0.849	0.865	0.314	0.482	0.773	1.051

Note: UIRT = unidimensional item response theory; UA = unidimensional approximation; new = new form; base = base form.

Table 5.

Correlations Between Unidimensional and Unidimensional Approximation Item Parameter Estimates.

	Discrimination	Difficulty
Math new	.965	.999
Math base	.971	.999
Science new	.990	.999
Science base	.985	.999
Social studies new	.964	.997
Social studies base	.981	.999

Note: new = new form; base = base form.

The similarity between both sets of item parameter estimates may help to explain why the unidimensional procedures and the multidimensional procedures yielded similar trends across the score scale. However, the fact that the unidimensional procedures and the unidimensional approximation procedures performed similarly at some locations along the score scale—whereas the two sets of procedures performed differently at other locations along the score scale—may be explained via a generic discussion of how the dimensional structure of each exam may affect interpretations at various points along the scale.

When more than one trait is measured by an exam, differences between scores at one part of the scale may have an entirely different meaning than differences between scores at another part of the scale. For example, on the science exams, differences between scores at the lower end of the scale may be primarily due to differences between examinees on the biological science/life science trait, whereas differences between scores at the upper end of the scale may be primarily due to differences between examinees on the physical sciences/earth and environmental science trait. Centroid plots (Reckase, 2009) are often used to determine which trait(s) contribute the most toward differences in scores at various points along the scale.

In this study, there may be points along the scale that are more “unidimensional” than other points along the scale. That is, at some points along the score scale, differences between scores may be the result of differences on only one trait, whereas at other points along the score scale, differences between scores may be the result of differences on more than one trait. Therefore, the unidimensional and multidimensional equating procedures may perform more similarly where differences in scores are primarily due to differences on only one trait. Conversely, the unidimensional and multidimensional equating procedures may perform less similarly where differences in scores are primarily due to differences on more than one trait.

MIRT and UIRT Versus Equipercentile

A third noticeable theme in this study was that the multidimensional procedures performed more similarly to the equipercentile equating procedure than to the unidimensional procedures, which is apparent in Figures 1 to 3. These results were expected, given that the unidimensional procedures were expected to contain more systematic error than the multidimensional procedures due to the violation of the unidimensionality assumption. These results imply that the multidimensional equating procedures developed in this research may be preferable over unidimensional equating procedures when the data are not strictly unidimensional. The performance of the multidimensional procedures should be verified via a series of simulation studies before these procedures are fully implemented in practice, however.

Summary and Conclusion

The purposes of this research were to create equating procedures that can be used in conjunction with the MIRT framework and to demonstrate how these procedures were conducted using data from the ITED (Forsyth et al., 2001). Six equating procedures were conducted and evaluated in this study: (a) UIRT observed score equating, (b) UIRT true score equating, (c) full MIRT observed score equating, (d) unidimensional approximation of MIRT observed score equating, (e) unidimensional approximation of MIRT true score equating, and (f) equipercentile equating.

Both unidimensional equating procedures performed similarly and all three multidimensional equating procedures performed similarly. This is especially noteworthy provided that two of the multidimensional procedures were formed by approximating a UIRT model given multidimensional parameter estimates, whereas the other procedure was based on a “full” MIRT model. The unidimensional procedures and the multidimensional procedures tended to perform differently, though both sets of procedures did yield similar trends. Also, the multidimensional procedures performed more similarly to the equipercentile procedure than did the unidimensional procedures.

In conclusion, it appears as if the multidimensional equating procedures presented in this research may provide an adequate alternative to UIRT equating when the data are not strictly unidimensional. Provided that the dimensional structure of each form is appropriately taken into account by the multidimensional equating procedures, the results produced by these procedures may contain less systematic error than do the results produced by the unidimensional equating procedures when the data are not strictly unidimensional. However, the performance of these procedures should be empirically verified via a simulation study before these procedures are fully implemented in practice.

Footnotes

Acknowledgements

This article is an extension of the senior author’s doctoral thesis at the University of Iowa. The authors would like to thank Timothy Ansley, Robert Brennan, and Michael Kolen for their invaluable contribution, insight, and support toward this research.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Ackerman

T. A.

(1996). Graphical representation of multidimensional item response theory analyses. Applied Psychological Measurement, 20, 311-329.

Ackerman

T. A.

Gierl

M. J.

Walker

C. M.

(2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22, 37-53.

Ansley

T. N.

Forsyth

R. A.

(1985). An examination of the characteristics of unidimensional IRT parameter estimates derived from two-dimensional data. Applied Psychological Measurement, 9, 37-48.

Bock

R. D.

Aitkin

(1981). Marginal maximum likelihood estimation of item parameters: Applications of an EM algorithm. Psychometrika, 46, 443-459.

Bock

R. D.

Gibbons

Schilling

S. G.

Muraki

Wilson

D. T.

Wood

(2003). TESTFACT 4.0 [Computer software and manual]. Lincolnwood, IL: Scientific Software International.

Davey

Oshima

Lee

(1996). Linking multidimensional item calibrations. Applied Psychological Measurement, 20, 405-416.

Dorans

N. J.

Holland

P. W.

Thayer

D. T.

Tateneni

(2003). Invariance of score linking across gender groups for three advanced placement program exams. In Dorans

N. J.

(Ed.), Population invariance of score linking: Theory and applications to advanced placement program examinations (Research Report 03-27, pp. 79-118). Princeton, NJ: Educational Testing Service.

Forsyth

R. A.

Ansley

T. A.

Feldt

L. S.

Alnot

S. D.

(2001). The Iowa Tests of Educational Development, Forms A and B, Levels 15-17/18. Itasca, IL: Riverside Publishing.

Hanson

Zeng

(1995). PIE: A computer program for IRT equating (Windows Console Version, Revised by Z. Cui, May 20, 2004) [Manual]. Unpublished manuscript, College of Education. University of Iowa, Iowa City.

10.

Hirsch

(1989). Multidimensional equating. Journal of Educational Measurement, 26, 337-349.

11.

Kolen

Brennan

R. L.

(2004). Test equating, scaling, and linking: Methods and practices (2nd ed.). New York, NY: Springer.

12.

Lissitz

(2000). An evaluation of the accuracy of multidimensional IRT linking. Applied Psychological Measurement, 24, 115-138.

13.

Lord

F. M.

Wingersky

M. S.

(1984). Comparison of IRT true-score and equipercentile observed-score “equatings.” Applied Psychological Measurement, 8, 452-461.

14.

Min

(2003). The impact of scale dilation on the quality of the linking of multidimensional item response theory calibrations (PhD thesis). Michigan State University, East Lansing.

15.

Oshima

Davey

Lee

(2000). Multidimensional linking: Four practical approaches. Journal of Educational Measurement, 37, 357-373.

16.

R Development Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org

17.

Reckase

M. D.

(1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9, 401-412.

18.

Reckase

M. D.

(2009). Multidimensional item response theory. New York, NY: Springer.

19.

Thompson

Nering

Davey

(1997, June). Multidimensional IRT scale linking. Paper presented at the annual meeting of the Psychometric Society, Gatlinburg, TN.

20.

Yon

(2006). Multidimensional item response theory (MIRT) approaches to vertical scaling (PhD thesis). Michigan State University, East Lansing.

21.

Zeng

Kolen

M. J.

Hanson

B. A.

Cui

Chien

(2004). RAGE-RGEQUATE [Computer software]. Iowa City: University of Iowa.

22.

Zhang

(1996). Some fundamental issues in item response theory with applications (Unpublished doctoral dissertation). Department of Statistics, University of Illinois at Urbana–Champaign.

23.

Zhang

Stout

W. F.

(1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129-152.

24.

Zhang

Stout

W. F.

(1999b). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.

25.

Zhang

Wang

(1998, April). Relating reported scores to latent traits in a multidimensional test. Paper presented at the annual meeting of American Educational Research Association, San Diego, CA.