Characterizing Sampling Variability for Item Response Theory Scale Scores in a Fixed-Parameter Calibrated Projection Design

Abstract

A common practice of linking uses estimated item parameters to calculate projected scores. This procedure fails to account for the carry-over sampling variability. Neglecting sampling variability could consequently lead to understated uncertainty for Item Response Theory (IRT) scale scores. To address the issue, we apply a Multiple Imputation (MI) approach to adjust the Posterior Standard Deviations of IRT scale scores. The MI procedure involves drawing multiple sets of plausible values from an approximate sampling distribution of the estimated item parameters. When two scales to be linked were previously calibrated, item parameters can be fixed at their original published scales, and the latent variable means and covariances of the two scales can then be estimated conditional on the fixed item parameters. The conditional estimation procedure is a special case of Restricted Recalibration (RR), in which the asymptotic sampling distribution of estimated parameters follows from the general theory of pseudo Maximum Likelihood (ML) estimation. We evaluate the combination of RR and MI by a simulation study to examine the impact of carry-over sampling variability under various simulation conditions. We also illustrate how to apply the proposed method to real data by revisiting Thissen et al. (2015).

Keywords

item response theory linking calibrated projection restricted recalibration multiple imputation

Introduction

Linking refers to methods that use results obtained from one test to infer what the results would have been if another test had been used (Linn et al., 2009). Linking is widely used in both small and large scale assessments (e.g., LaFlair et al., 2017; Livingston & Kim, 2010; OECD, 2016; Phillips et al., 2014). Among multiple types of linking, this paper focuses on calibrated projection.

If two scales measure distinct but possibly highly correlated constructs, predicting scores from one scale to the other can be achieved by regression or projection. Regression considers a point-to-point prediction while projection predicts the score distribution. Projection is a relatively weak form of linking—it does not require the two tests to be linked measure exactly the same construct. Calibrated projection, as a special case of projection, combines both calibration and projection steps (Thissen et al., 2011, 2015). In calibrated projection, item responses are assumed to follow a two-dimensional simple structure Item Response Theory (IRT) model: Items from the two tests load separately on two correlated latent factors. Projected IRT scores for one scale can be obtained given only responses to the other scale due to the presence of the inter-factor correlation. Linking can be in either direction, but the predicted scores from opposite directions are mostly asymmetric. Applications of calibrated projection can be found in Thissen et al. (2011) and Thissen et al. (2015). Depending on whether the scales to be linked have been calibrated previously, model parameters in calibrated projection can be estimated in either one or two stages. In Thissen et al. (2011), item and latent variable (LV) density parameters are simultaneously estimated using one data set. In contrast, Thissen et al. (2015) attempted to link previously calibrated scales and estimated the LV density parameters conditional on the published item parameter estimates.

The original calibrated projection procedure used in Thissen et al. (2015) ignores the sampling variability in calibration, which may result in understated uncertainty for IRT scale scores and consequently impact the statistical inferences of linking. Thissen et al. (2015) showed the summed score conversion tables when projecting from one scale to another. Due to sampling variability, the estimated Posterior Standard Deviations (PSDs) in the conversion tables may need further adjustment. Several methods can be used to account for the impact of sampling variability in the further adjustment, such as the multiple imputation approach (MI; e.g., Yang et al., 2012) and the fully Bayesian method (e.g., Patz & Junker, 1999). In the context of Maximum Likelihood (ML) scoring, Cheng and Yuan (2010) also proposed an analytical approach to upward adjust the standard errors (SEs) of scores. It is noted that ML scores are computed without any LV distribution, whereas calibrated projection is not possible without a bivariate LV distribution. Therefore, Cheng and Yuan (2010)’s method is not further considered in the present work.

The MI approach has been used in previous studies to address the parameter estimation uncertainty in test equating designs. In a common-item linking design, Li and Lissitz (2004) used MI to obtain SEs of IRT equating coefficients, assuming no covariance among parameter estimates for different items. Zhang and Zhao (2019) adopted a similar MI approach, but different from Li and Lissitz (2004), they used the covariance matrix among item parameter estimates. Our study differs from Li and Lissitz (2004) and Zhang and Zhao (2019) in three aspects: (1) we considered a calibrated projection design, whereas they used a common-item nonequivalent group equating design; (2) their studies focused on the performance of MI in estimating the SEs of IRT parameter transformation coefficients, while we used MI to obtain the adjusted IRT scores; and (3) we used a two-stage estimation similar to Thissen et al. (2015) in our study.

Item Response Theory scoring in calibrated projection may benefit from using published item parameters. Kolen and Brennan (2014) indicated that there are potential benefits if the published item parameters from a previously calibrated item pool are re-used. For instance, across studies using independent samples, a common ability scale can be established by fixing the item parameter of common items at the same values and adjusting the LV scales accordingly. As another example, PROMIS studies collected samples in a principled way and published their calibrated item parameters based on a representative sample from the reference population. In this way, researchers always use the same items with the same set of parameters so that scores from separate studies are comparable.

A number of calibration studies of PROMIS scales have been published (e.g., Huang et al., 2016; Irwin et al., 2010; Pilkonis et al., 2011; Tucker et al., 2014), and these calibrated PROMIS scales have been used in many subsequent studies. It is possible to perform item recalibration by collecting a new sample for the purpose of linking. However, a complete recalibration may not be ideal when the scores from the previous established scales were already used. In addition, new item parameter estimates obtained by complete recalibration of a new sample could be computationally less stable because of small sample sizes and different distribution characteristics (Liu et al., 2019). As such, a complete recalibration is less preferred than using pre-existing item parameters.

The current work aims to address two issues in calibrated projection: (1) characterizing the sampling variability when estimation is done in two stages, especially when the calibration dataset is not accessible; and (2) accounting for sampling variability in the IRT model parameters when computing IRT scale scores and PSDs. The first issue is addressed by a two-stage calibrated projection method using a conditional estimation procedure, which is a special case of the Restricted Recalibration (RR). The second issue is addressed by the MI method. The rest of the paper is organized as follows. We first present in the context of a two-dimensional simple structure IRT model the methods we use to address the two issues mentioned above. Then, we report a simulation study with a goal to show the extent to which sampling variability is carried over to the scoring process. We also revisit a calibrated projection empirical example in Thissen et al. (2015) by applying the proposed methods.

Method

Calibrated Projection

When two scales measure somewhat distinct constructs, calibrated projection can be used to translate scores between the two scales. Under a two-dimensional simple structure IRT model (see Figure 1), scores on one scale and the corresponding precision estimates can be obtained given the responses on the other scale. Calibrated projection is based on multidimensional IRT models. When items are not calibrated in previous studies, calibrated projection can be done by fitting a two-dimensional model, with LV correlation and item parameters estimated simultaneously in one stage as in Thissen et al. (2011). When scales are previously calibrated, calibrated projection can be done by fixing the item parameters from the two scales at their published values, and estimating LV density parameters conditional on the fixed item parameters in two stages (Thissen et al., 2015). We focus on the latter case.

Figure 1.

Model parameters ( γ ) is composed of focal parameters ( η ) and nuisance parameters ( ξ ). Note. The nuisance parameters ( ξ ) are the published item parameters from the original calibrations. The focal parameters ( η ) are the LV density parameters, or specifically the means (μ₁, μ₂), variances $(σ_{1}^{2}, σ_{2}^{2})$ , and covariance (σ₁₂) that are estimated from the scoring sample conditional on the nuisance parameters ( ξ ). The test is composed of J₁ items in LV1 and J₂ items in LV2. J₁ + J₂ = J.

We focus on a two-dimensional Graded Response Model (De Ayala, 1994) in the present work; the techniques described in the sequel can be straightforwardly generalized to other MIRT models.

Suppose that the n × J data matrix X is composed of n independent and identically distributed (i.i.d) response vectors $X_{i} = {(X_{i, 1}, \dots, X_{i, J})}^{⊤}, i = 1, \dots, n$ , to a J-item test. Each item is assumed to have K ordered response categories, X_i,j ∈ {0, …, K − 1}, j = 1, …, J. Let γ denote the model parameters, and for the two LVs, let $θ_{i} = {(θ_{i, 1}, θ_{i, 2})}^{⊤}$ . The trace line, which describes the probability of endorsing category k as a function of the LV θ and model parameter γ for item j, can be expressed as

f_{j} (k | θ, γ_{j}) = \frac{1}{1 + \exp [- (a_{j}^{⊤} θ + d_{j, k})]} - \frac{1}{1 + \exp [- (a_{j}^{⊤} θ + d_{j, k + 1})]},

(1)

in which

a_{j} = {(a_{j, 1}, a_{j, 2})}^{⊤}

is a vector of item slope parameters, and d_j,k is the item intercept for kth category. The intercept parameters are in a strictly descending order and d_j,0 = ∞ and d_j,K = −∞. Under the two-dimensional simple structure model, θ₁ has estimated slopes a_j,1 on the first-scale items, and slopes a_j,2 for these items are fixed at 0. On the contrary, θ₂ has estimated slopes a_j,2 on the second-scale items, and fixed values of 0 for the first-scale items. For simple-structure models, only one a_j is non-zero for each j. As shown in Figure 1, we assume J₁ items 1, …, J₁ load on the factor θ₁, and J₂ items J₁ + 1, …, J load on the factor θ₂, J₁ + J₂ = J.

Suppose LV follows a multivariate normal distribution with the mean vector μ and covariance matrix Σ. That is to say, η is composed of μ and Σ (see Figure 1). More specifically, η is expressed as

η = {(μ_{1}, μ_{2}, σ_{1}^{2}, σ_{2}^{2}, σ_{12})}^{⊤},

(2)

in which μ₁ and μ₂ are the means and

σ_{1}^{2}, σ_{2}^{2}, σ_{12}

are the variances and covariance of θ₁ and θ₂. ξ denotes item parameters. The model parameters γ is composed of both sets of parameters,

γ = {(ξ^{⊤}, η^{⊤})}^{⊤}

. Based on the assumption of local independence, the joint likelihood of the fixed response pattern

x_{i} = {(x_{i, 1}, \dots, x_{i, J})}^{⊤}

and the LV θ conditional on γ can be expressed as

f (x_{i}, θ | γ) = ϕ (θ; μ, Σ) \prod_{j = 1}^{J} f_{j} (x_{i, j} | θ, ξ),

(3)

in which ϕ( θ ; μ , Σ) denotes the normal density of θ . The marginal likelihood f(x_i| γ ) can be calculated by integrating out θ in equation (3)

f (x_{i} | γ) = \int ϕ (θ; μ, Σ) \prod_{j = 1}^{J} f_{j} (x_{i, j} | θ, ξ) d θ .

(4)

We adjust the LV mean vector and covariance matrix such that the reference LV in the current study is on the same scale as the LV in the original calibration sample. Similar to what Thissen et al. (2015) did in their calibrated projection study, suppose the projection is from Scale 1 to Scale 2, to compute the projected scores on Scale 2 in a hypothetical calibration population that is the same as the reference population for Scale 1, the LV mean vector and covariance matrix are rescaled as follows. μ and Σ are rescaled such that the first LV has a mean of 0 and a variance of 1. The LV mean vector of θ₁ and θ₂ can be transformed to

μ_{r e f e r e n c e (1)} = [\begin{matrix} 0 \\ μ_{2}^{'} \end{matrix}] = [\begin{matrix} 0 \\ μ_{2} - \frac{σ_{12}}{σ_{1}^{2}} \times μ_{1} \end{matrix}],

(5)

in which

μ_{2}^{'}

is the transformed mean of θ₂. The transformed covariance matrix is

Σ_{r e f e r e n c e (1)} = [\begin{matrix} 1 & - \\ \frac{σ_{12}}{σ_{1}^{2}} & \frac{σ_{2}^{2}}{σ_{1}^{2}} \end{matrix}] .

(6)

Calculation steps for Equations (5) and (6) can be found in Equations (1.3)–(1.9) in Thissen et al. (2015). The T-score unit with a mean of 50 and a standard deviation of 10 (Pilkonis et al., 2011) is used throughout this paper for the summed-score expected a posteriori (SS-EAP) in both scales.

Item Response Theory Scoring

We focus on SS-EAP in this paper, but the proposed method can be adapted to response-pattern EAP scores. The joint likelihood of summed score s and θ is the sum of the joint response-pattern likelihoods over all patterns with summed score s and can be expressed as

f (s, θ | γ) = \sum_{\sum_{j} x_{i, j} = s} f (x_{i}, θ | γ),

(7)

and the posterior density can be obtained by Bayes’ Theorem

f (θ | s, γ) = \frac{f (s, θ | γ)}{\int f (s, θ | γ) d θ},

(8)

where θ is integrated out in the marginal likelihood of the summed score s in the denominator. Conditional on γ , the SS-EAP estimator is then computed by

{\hat{θ}}_{γ} = \int θ f (θ | s, γ) d θ,

(9)

and the PSD is calculated as the square root of the diagonal of the posterior variance-covariance matrix

V ({\hat{θ}}_{γ}) = \int (θ - {\hat{θ}}_{γ}) {(θ - {\hat{θ}}_{γ})}^{⊤} f (θ | s, γ) d θ .

(10)

If the estimates of γ are plugged into Equations (9) and (10), the output values are referred to as the plug-in (i.e., unadjusted) SS-EAPs and posterior variance.

Multiple Imputation

We applied an MI-based approach to adjust SS-EAPs and their corresponding PSDs to reflect sampling variability. The procedure is in line with the predictive inference framework in Liu and Yang (2018), which considers predicting the plausible values of θ that follow the true posterior distribution with density f( θ |s, γ ) in equation (8). In the MI-based approach, the imputed values of γ are generated from a multivariate normal distribution, which approximates the sampling distribution of the pseudo ML estimator (Gong & Samaniego, 1981; Parke, 1986).

The general method to obtain the MI-based SS-EAPs and PSDs is as described. Let $\hat{γ}$ be the estimated model parameters, which depend on potentially both the calibration sample Y and the scoring sample X, and Γ be the asymptotic covariance matrix (ACM) for $\hat{γ}$ . The density of the predictive distribution

f (θ | X, Y, s) = \int f (θ | s, γ) f (γ | X, Y) d γ .

(11)

As can be found in the RR section, by the asymptotic theory of pseudo ML estimator, the resulting two-stage estimator of model parameters $\hat{γ}$ is asymptotically normal (Liu et al., 2019). The posterior distribution f( γ |X, Y) can be approximated by the multivariate normal density $ϕ (γ; \hat{γ}, Γ)$ (Yang et al., 2012). Integrating out γ , the adjusted SS-EAP estimator of θ can be calculated by

\hat{θ} = \int {\hat{θ}}_{γ} ϕ (γ; \hat{γ}, Γ) d γ,

(12)

and the corresponding posterior variance can be computed by

V (\hat{θ}) = \int V ({\hat{θ}}_{γ}) ϕ (γ; \hat{γ}, Γ) d γ + \int (θ - {\hat{θ}}_{γ}) {(θ - {\hat{θ}}_{γ})}^{⊤} ϕ (γ; \hat{γ}, Γ) d γ .

(13)

In practice, the MI-based SS-EAPs and PSDs can be obtained by Monte Carlo approximation. Suppose M sets of parameters are drawn from the multivariate normal distribution $ϕ (γ; \hat{γ}, Γ)$ . Denote each set of imputed parameters as γ (m) for m = 1, …, M, then the posterior ${\hat{θ}}_{γ_{(m)}}$ can be calculated by plugging each γ (m) into equation (9). The empirical average can be used as an MI SS-EAP approximation to $\hat{θ}$ in equation (12) and computed as

\bar{θ} ≃ \frac{1}{M} \sum_{m = 1}^{M} {\hat{θ}}_{γ_{(m)}},

(14)

and the MI variance in equation (13) approximated by

V (\bar{θ}) ≃ \bar{V} + (1 + \frac{1}{M}) B .

(15)

The values computed by equation (14) and the square root of equation (15) are referred to as MI (i.e., adjusted) SS-EAPs and PSDs. $\bar{V}$ and $(1 + \frac{1}{M}) B$ , respectively, estimate the two components on the right-hand side of equation (13). The first component is the expectation of a conditional variance (i.e., within-imputation variance), and is computed by $\bar{V} = M^{- 1} \sum_{m = 1}^{M} V ({\hat{θ}}_{γ_{(m)}})$ ; the second component is the variance of a conditional expectation (i.e., between-imputation variance), which is denoted B and can be computed as $B = {(M - 1)}^{- 1} \sum_{m = 1}^{M} ({\hat{θ}}_{γ_{(m)}} - \bar{θ}) {({\hat{θ}}_{γ_{(m)}} - \bar{θ})}^{⊤}$ . The within-imputation variance reflects measurement error, and the between-imputation variance reflects sampling error. Besides the MI-based approach, there are other ways to approximate these integrals, such as drawing plausible values from predictive distributions as in Liu and Yang (2018).

We compute the ratio

r = \frac{(1 + M^{- 1}) B}{\bar{V}}

(16)

to quantify the extent to which the MI adjustment on PSDs is substantial. It reflects the impact of sampling variability of estimated model parameters on the precision of IRT scores—a large r implies a large additional uncertainty due to sampling error relative to the measurement error. For example, if r = 0.1, then uncertainty due to measurement error is 10 times as much as that due to sampling error. As indicated in equation (16), the uncertainty is gauged by variance.

Restricted Recalibration

One of the problems we encounter when using the MI approach to account for sampling variability is that model parameters are computed in two stages, which is a special case of RR. Restricted Recalibration concerns fitting a model parameterized by $γ = {(ξ^{⊤}, η^{⊤})}^{⊤}$ , in which the focal parameters η are estimated conditional on available estimates of nuisance parameters ξ (Liu et al., 2019). In the current calibrated projection design, η contains the LV density parameters, and ξ the item parameters. The full ACM for the two-stage estimates of γ can be obtained by the general theory of pseudo ML.

The full ACM from RR can be constructed by the following procedures. Let n′ denote the sample size of Y (i.e., calibration sample) and n denote the sample size of X (i.e., scoring sample). Under the Central Limit Theorem, when the n′ → ∞, estimated item parameters are i.i.d with finite mean of true item parameters ξ and the true covariance matrix of Ω_ξ( ξ ), then

\sqrt{n^{'}} (\hat{ξ} - ξ) \overset{d}{\to} N (0, Ω_{ξ} (ξ)) .

(17)

Under certain regularity conditions (Gong & Samaniego, 1981)

\sqrt{n} (\begin{matrix} \hat{ξ} - ξ \\ \hat{η} - η \end{matrix}) \overset{d}{\to} N (0, (\begin{matrix} c Ω_{ξ} & Ω_{ηξ}^{T} \\ Ω_{ηξ} & Ω_{η} \end{matrix})),

(18)

as the ratio of n′ to n converges to a constant c when both sample sizes tend to infinity (Liu et al., 2019). ξ and η denote the true values of nuisance and focal parameters, respectively.

\hat{ξ}

is estimated from original calibration samples, and

\hat{η}

is estimated by pseudo ML estimation conditional on

\hat{ξ}

using the scoring sample. Calibration samples and scoring samples are independent. The full ACM in the multivariate normal distribution in equation (18) is the per-observation covariance matrix for the true model parameters γ . Formulas for each block of full ACM can be found in Equations (7)–(10) in Liu et al. (2019).

Simulation

In this section, we report a simulation study to examine the impact of carry-over sampling error on IRT scores in both scales under various combinations of simulation conditions.

Design

The manipulated variables in the simulation study include: (1) LV correlations, (2) calibration sample size, and (3) scoring sample size. For simplicity, the projection in the simulation study is from Scale 2 to Scale 1.

In the simulation study, the published item parameters in Irwin et al. (2010) and Pilkonis et al. (2011) were used as true model parameters to generate the calibration and scoring sample data. All items have a 5-point Likert-type scale. Items were calibrated separately, and hence, LV mean and variance for each of the two scales were fixed to 0 and 1 in the calibration samples. Two LV correlation conditions were examined: 0.89 and 0.96. These two correlation values come from two linking studies in Thissen et al. (2015) and Thissen et al. (2011). In the scoring samples, the true means for LVs in Scale 1 and Scale 2 are 0.631 and 0.868, respectively, based on the estimates reported in Thissen et al. (2015). The true covariance matrix for LVs is $(\begin{matrix} 1.650 & 1.174 \\ 1.174 & 1.047 \end{matrix})$ when the correlation is 0.89, and $(\begin{matrix} 1.650 & 1.262 \\ 1.262 & 1.047 \end{matrix})$ when the correlation is 0.96. Sample size conditions were determined by two fully-crossed factors: 1) the calibration sample size n′ = 250, 500, and 1000, 2) the scoring sample size n = 250, 500, and 1000. Five hundred replications were conducted for each condition. Two independent samples were used for item calibration. The first calibration sample contains responses to eight Scale 1 items, and the second calibration sample contains responses to eight Scale 2 items. Items from the two scales were assumed to be calibrated in equal-size samples, for example, n^′ = 250 means there are 250 observations for item calibration in each scale. In addition, one scoring sample was used to (1) estimate LV density parameters conditional on item parameter estimates from the calibration samples, and (2) obtain both adjusted and unadjusted SS-EAPs and PSDs by calibrated projection from one scale to the other. Respondents in scoring samples were assumed to answer all 16 items in both scales.

All components were done in R (R Core Team, 2019). Data generation and model fitting were conducted using the mirt package (Chalmers, 2012). The expectation-maximization (EM) algorithm (Bock & Aitkin, 1981) was used to find the ML estimates of the two-dimensional simple structure item and LV density parameters. The default number of quadrature points in mirt were adopted—61 quadrature points for one-dimensional models and 31 quadrature points per dimension for two-dimensional models. The default settings of convergence tolerance .0001 and maximum number of iterations 500 were adopted. The expected Fisher information was approximated by Monte Carlo method proposed in Monroe (2019). The evaluation criterion is the r value. A larger r reflects a larger impact of sampling error on IRT scoring, and thus it indicates greater necessity to adjust the PSDs.

We considered two heuristic ACM constructions per the suggestion of a reviewer: the focal-only ACM approach and the block-diagonal ACM approach. Although only the full ACM approach is theoretically correct, we are interested in whether these two simplified ACM approaches yield comparable results. The focal-only ACM approach is identical to the naive method presented in Liu et al. (2019). In this approach, the estimates of the item parameters are considered known, and thus there is no carry-over impact of the sampling variability within the estimated item parameters. Only the per-observation ACM from equation (18) is used to impute the focal parameters. In the block-diagonal ACM approach, the focal parameters and nuisance parameters are treated as if they were independently estimated from separate samples, and thus the $Ω_{ηξ}^{T}$ and Ω_ηξ in equation (18) are set to 0. The same calculation procedures for MI SS-EAP, MI variance, and r are repeated for the focal-only and block-diagonal ACM approaches with the correspondingly imputed parameters.

Results

Figures 2 and 3 summarize the changes of r across summed scores when projecting from Scale 2 to Scale 1. Figure 2 shows the r values of Scale 1 and Scale 2 when the LV correlation is 0.89, and Figure 3 is when the LV correlation is 0.96. It can be found that r can be large in this projection. In Figure 2, under the condition when n^′ = 250 and n = 250, the maximum r in Scale 2 can reach 0.515. That means in this projection, the sampling error is 51.5% of the measurement error—it is indeed necessary to do the MI adjustment to the PSD in this case.

Figure 2.

r for Scale 1 and Scale 2 across summed scores when projecting from Scale 2 to Scale 1. LV correlation is 0.89. The calibration sample sizes are n’ = 250, 500, and 1000. The scoring sample sizes n = 250, 500, and 1000. Scale 1 and Scale 2 are depicted in different colors: gray for Scale 1, and black for Scale 2. ACMs are constructed by the full ACM (solid), focal-only ACM (dashed), and block-diagonal ACM (dotted) approaches. Note. ACM = asymptotic covariance matrix.

Figure 3.

r for Scale 1 and Scale 2 across summed scores when projecting from Scale 2 to Scale 1. LV correlation is 0.96. The calibration sample sizes are n’ = 250, 500, and 1000. The scoring sample sizes n = 250, 500, and 1000. Scale 1 and Scale 2 are depicted in different colors: gray for Scale 1, and black for Scale 2. ACMs are constructed by the full ACM (solid), focal-only ACM (dashed), and block-diagonal ACM (dotted) approaches. Note. ACM = asymptotic covariance matrix.

It is also noticeable in Figures 2 and 3 that regardless of the LV correlation, fixing one calibration sample size and one ACM approach, the r for the reference scale (e.g., Scale 2 when projecting from Scale 2 to Scale 1) is the same across different scoring sample size conditions. The reason is that the mean and variance of the reference scale are fixed to 0 and 1, respectively, and the only carry-over impact comes from the uncertainty in the item-calibration step. Therefore, when the calibration sample size is fixed, the r values are the same for the reference scale across different scoring sample size conditions. Moreover, Figures 2 and 3 show that when projecting from Scale 2 to Scale 1, the r values in Scale 1 (i.e., the projected scale) are larger when the LV correlation is larger. There is not much difference in the r values in Scale 2 between the two LV correlation conditions. This shows that a larger LV correlation could result in a smaller measurement error, and thus a larger r in Scale 1 when the sampling error does not differ much.

It can also be found that in both figures, the r for Scale 2 gradually increases until the summed score gets around 28 and drops after that. r increases when the sampling error increases and/or measurement error decreases, which means the magnitude of sampling error relative to measurement error becomes larger. It is found in Scale 2 that both the sampling error and the measurement error continue to increase after the summed score reaches 28, but the measurement error grows faster than the sampling error, and thus the r drops after that point. For Scale 1, r decreases a little bit and then increases gradually as summed score increases, but the adjustment is not always needed if we use $r > 0.1$ as an indicator of the necessity to adjust the PSDs.

Across the three ACM approaches in Figures 2 and 3, the r does not differ much when using the full ACM and the block-diagonal ACM approaches, but differs a lot from the r obtained using the focal-only ACM approach.

Summary

We used calibrated projection to link two scales when underlying constructs are correlated but not exactly the same. Restricted Recalibration was conducted to obtain the estimates of LV density parameters conditional on published item parameters of the two scales. The ACMs from different approaches have block structures. An MI-based procedure was used to account for the sampling variability that is carried over to the scoring process.

The results shown in Figures 2 and 3 can be used as a guideline for researchers to use the MI adjustment under different LV correlations, calibration sample size, and scoring sample size conditions. Our results indicate that the adjustment is larger for both scales when there is a smaller calibration sample. In the current projection setting, when projecting from Scale 2 to Scale 1, the r can be larger than 51% using the full ACM approach. This means that sampling error is over 51% of the measurement error, and indeed we need to perform PSD adjustment in such cases. When the new sample is small and the calibration sample is large, the r can still be about 15%. It is noted that the patterns found in the current study are consistent to the findings in Yang et al. (2012) that the impact of sampling variability on the uncertainty in SS-EAPs can be small. However, depending on how small the calibration sample size is relative to the number of estimated parameters and if the model is multidimensional, the impact of sampling variability on the PSDs can be large. In general, if r is larger than 0.1, we consider the adjustment is needed.

The r obtained using the full ACM approach is similar to the ones obtained using the block-diagonal ACM approach, but these r’s differ a lot to the ones obtained by the focal-only ACM approach. Among these three types of ACM approaches, only the full ACM approach is technically the correct way to construct the ACM. Moreover, a higher LV correlation results in a larger r in the projected scale.

Empirical Example

Data

We demonstrate how to account for the sampling variability of estimated model parameters that is carried over to projected scores in one of the scales being linked in Thissen et al. (2015). There are two test scales involved in the study—The PROMIS pediatric and adult anxiety test scales. Both of these scales were previously studied and their item parameters were published. Item parameters in these two scales are uncorrelated since the samples used to calibrate these two scales were collected separately. Researchers have access to the published item parameters in these two scales but not the original calibration data.

Items in PROMIS pediatric and adult anxiety tests were calibrated and evaluated by representative samples in previous studies. On the one hand, the PROMIS pediatric anxiety test contains 18 items in total. The items were randomly split into two test forms with equal number of items in each form. Each item was measured on the same five-point Likert-scale ranging from never (0) to always (4). 1529 participants aged 8–17 were recruited, and were randomly assigned to answer either test form 1 or test form 2. In total, 759 participants took test form 1 and 770 took test form 2. On the other hand, PROMIS adult anxiety test targets participants older than 18. The test contains 56 items for each of the 14 health-related measures including anxiety (Cella et al., 2010; Pilkonis et al., 2011). There are two test designs: full-bank and block testing designs. Participants in the full-bank testing group took all anxiety items. Other participants in the block testing group received content-balanced 98 items (i.e., 7 items each, for all the 14 domains). In total, 788 participants took full-bank anxiety items and 14,048 participants took items from the block testing design. The original article provided limited information about samples, so we approximated the sample size for the PROMIS adult anxiety scale by $788 + 14048 \times \frac{98}{56 \times 14} = 2544$ . With a rationale of random assignment and because of the content-balanced design, the expected sample size from the block design can be approximated by $14048 \times \frac{98}{56 \times 14} = 1756$ .

In the calibrated projection research, Thissen et al. (2015) collected new data that contain responses to both PROMIS pediatric and adult anxiety scales for the purpose of linking, which were also used as our scoring sample. In the new data, 874 teenagers aged 14–20 responded to eight short-form PROMIS pediatric-scale and eight short-form PROMIS adult-scale anxiety items (Thissen et al., 2015). Among the eight pediatric-scale items, three items are from pediatric test form 1, and five items are from test form 2. All of the eight adult-scale items are from a published short-form PROMIS adult anxiety test.

Constructing ACM

The per-observation ACM in equation (18) can be constructed using the aforementioned calibration and scoring samples. Model parameters include item parameters estimated from original calibration samples and LV density parameters estimated from the new scoring sample. The ACM can be divided into a three block-diagonal ACMs: ${\hat{Ω}}_{\hat{ξ}}$ , ${\hat{Ω}}_{\hat{η} \hat{ξ}}$ , and ${\hat{Ω}}_{\hat{η}}$ . Because we do not have access to the original calibration samples, the three blocks ${\hat{Ω}}_{\hat{ξ}}$ , ${\hat{Ω}}_{\hat{η} \hat{ξ}}$ , and ${\hat{Ω}}_{\hat{η}}$ are estimated by Monte Carlo simulation using the procedure described in Liu et al. (2019) and Monroe (2019).

The three components in full ACM can be further decomposed into sub-blocks. The block-structured ${\hat{Ω}}_{\hat{ξ}}$ can be divided into three components: ${\hat{Ω}}_{\hat{ξ} P (S F 1)}$ , ${\hat{Ω}}_{\hat{ξ} P (S F 2)}$ , and ${\hat{Ω}}_{\hat{ξ} A (S F)}$ , which are the ACMs of item parameters with respect to three items from pediatric short form 1, five items from pediatric short form 2, and 8-item adult short form, respectively. The respective subsample sizes for these three forms are 759, 770 and 2544, the calibration sample size is thus computed by n^′ = 759 + 770 + 2544 = 4073. The three full ACM blocks ${\hat{Ω}}_{\hat{ξ} P (S F 1)}$ , ${\hat{Ω}}_{\hat{ξ} P (S F 2)}$ , and ${\hat{Ω}}_{\hat{ξ} A (S F)}$ are divided by $\frac{759}{4073}, \frac{770}{4073}, \frac{2544}{4073}$ , respectively. c can be approximated by $\frac{n}{n^{'}} = \frac{874}{4073} = 0.215$ . The estimated mean vector $\hat{γ}$ of model parameters and the constructed full ACM in equation (18) are then plugged into the multivariate normal distribution $ϕ (γ; \hat{γ}, Γ)$ to conduct MI adjustment for the SS-EAPs and PSDs.

Results

Table 1 illustrates the unadjusted and adjusted SS-EAPs and PSDs when projecting from the pediatric to the adult scale. Table 2 shows these results for the other direction. It can be found that these tables are asymmetrical. Regardless of the projection direction, there is not much difference between unadjusted and adjusted SS-EAPs for both the pediatric scale and the adult scale in both tables. This means no additional SS-EAP adjustment is needed for both the pediatric and adult scales, given the current calibration and scoring sample size conditions. Figure 4 summarizes the distribution of raw summed scores in pediatric and adult scales in the real data for reference.

Table 1.

The Unadjusted and MI Adjusted SS-EAPs and PSDs for Summed Scores on the Pediatric Scale. The Projection Direction is from Pediatric to Adult Scale.

Pediatric Summed Scores	SS-EAP				PSD
Pediatric Summed Scores	Pediatric (Unadjusted)	Pediatric (Adjusted)	Adult (Unadjusted)	Adult (Adjusted)	Pediatric (Unadjusted)	Pediatric (Adjusted)	Adult (Unadjusted)	Adult (Adjusted)
0	32.32	32.31	41.63	41.59	5.76	5.78	5.44	5.50
1	36.65	36.63	44.71	44.66	4.88	4.92	4.98	5.03
2	39.15	39.12	46.49	46.44	4.65	4.69	4.87	4.92
3	41.43	41.40	48.10	48.06	4.35	4.38	4.73	4.77
4	43.34	43.31	49.46	49.42	4.18	4.21	4.65	4.69
5	45.08	45.06	50.70	50.67	4.02	4.05	4.58	4.61
6	46.67	46.66	51.83	51.81	3.91	3.94	4.53	4.56
7	48.16	48.15	52.89	52.87	3.83	3.86	4.50	4.53
8	49.58	49.57	53.90	53.88	3.78	3.80	4.47	4.50
9	50.94	50.94	54.87	54.85	3.74	3.76	4.46	4.48
10	52.25	52.26	55.80	55.79	3.71	3.74	4.44	4.47
11	53.54	53.54	56.72	56.71	3.69	3.72	4.44	4.46
12	54.80	54.80	57.61	57.61	3.68	3.71	4.43	4.46
13	56.04	56.04	58.50	58.49	3.68	3.70	4.43	4.46
14	57.27	57.27	59.37	59.36	3.68	3.70	4.43	4.46
15	58.48	58.49	60.23	60.23	3.68	3.70	4.43	4.46
16	59.69	59.70	61.09	61.09	3.68	3.70	4.43	4.46
17	60.90	60.90	61.95	61.95	3.67	3.70	4.43	4.46
18	62.11	62.11	62.81	62.81	3.67	3.70	4.43	4.46
19	63.32	63.33	63.67	63.68	3.67	3.70	4.43	4.46
20	64.54	64.55	64.54	64.55	3.67	3.70	4.43	4.46
21	65.78	65.79	65.42	65.43	3.66	3.70	4.42	4.46
22	67.02	67.04	66.31	66.32	3.66	3.70	4.42	4.46
23	68.29	68.31	67.21	67.22	3.66	3.70	4.42	4.47
24	69.57	69.60	68.12	68.14	3.66	3.71	4.42	4.47
25	70.89	70.93	69.06	69.09	3.68	3.73	4.43	4.48
26	72.26	72.30	70.03	70.06	3.70	3.77	4.44	4.50
27	73.68	73.73	71.04	71.08	3.75	3.81	4.46	4.52
28	75.19	75.25	72.11	72.16	3.81	3.89	4.49	4.55
29	76.81	76.88	73.26	73.32	3.90	3.98	4.53	4.60
30	78.62	78.70	74.56	74.62	4.05	4.14	4.59	4.67
31	80.64	80.72	75.99	76.06	4.21	4.30	4.66	4.75
32	83.55	83.62	78.06	78.12	4.67	4.74	4.88	4.98

Note. MI = Multiple Imputation; SS-EAP = summed-score expected a posteriori; PSD = Posterior Standard Deviation.

Table 2.

The Unadjusted and MI Adjusted SS-EAPs and PSDs for Summed Scores on the Adult Scale. The Projection Direction is from Adult to Pediatric Scale.

Adult Summed Scores	SS-EAP				PSD
Adult Summed Scores	Adult (Unadjusted)	Adult (Adjusted)	Pediatric (Unadjusted)	Pediatric (Adjusted)	Adult (Unadjusted)	Adult (Adjusted)	Pediatric (Unadjusted)	Pediatric (Adjusted)
0	37.12	37.11	32.11	32.13	5.51	5.51	8.37	8.41
1	43.20	43.19	38.93	38.95	3.33	3.35	6.77	6.79
2	45.84	45.83	41.89	41.91	2.82	2.83	6.47	6.48
3	47.86	47.85	44.16	44.18	2.54	2.55	6.32	6.33
4	49.43	49.43	45.92	45.95	2.24	2.25	6.18	6.18
5	50.76	50.75	47.41	47.43	2.21	2.22	6.17	6.16
6	52.14	52.13	48.96	48.98	2.21	2.22	6.17	6.16
7	53.33	53.32	50.29	50.31	1.94	1.95	6.05	6.05
8	54.25	54.25	51.33	51.36	1.85	1.87	6.01	6.01
9	55.29	55.28	52.49	52.52	2.08	2.09	6.11	6.10
10	56.50	56.48	53.85	53.86	2.09	2.11	6.11	6.11
11	57.51	57.50	54.99	55.00	1.84	1.86	6.01	6.01
12	58.33	58.33	55.91	55.94	1.81	1.83	6.00	6.00
13	59.30	59.30	56.99	57.02	2.06	2.07	6.10	6.09
14	60.48	60.46	58.31	58.32	2.10	2.12	6.12	6.11
15	61.51	61.50	59.48	59.49	1.87	1.90	6.02	6.03
16	62.38	62.38	60.45	60.47	1.86	1.88	6.02	6.02
17	63.42	63.41	61.61	61.63	2.10	2.11	6.12	6.11
18	64.63	64.61	62.98	62.97	2.08	2.11	6.11	6.12
19	65.65	65.63	64.12	64.13	1.85	1.89	6.01	6.03
20	66.54	66.54	65.11	65.14	1.90	1.93	6.03	6.05
21	67.63	67.62	66.35	66.35	2.13	2.14	6.13	6.14
22	68.84	68.81	67.70	67.69	2.04	2.07	6.09	6.11
23	69.81	69.80	68.79	68.80	1.82	1.87	6.00	6.04
24	70.73	70.74	69.82	69.85	1.94	1.98	6.05	6.08
25	71.90	71.89	71.14	71.14	2.14	2.16	6.14	6.16
26	73.13	73.10	72.52	72.50	1.97	2.03	6.06	6.11
27	74.13	74.14	73.64	73.66	1.83	1.91	6.01	6.07
28	75.24	75.27	74.89	74.93	2.09	2.13	6.11	6.16
29	76.71	76.71	76.54	76.54	2.21	2.25	6.16	6.22
30	78.19	78.21	78.20	78.23	2.17	2.25	6.15	6.24
31	80.00	80.05	80.22	80.29	2.59	2.64	6.35	6.43
32	83.20	83.24	83.81	83.88	3.52	3.56	6.89	6.99

Note. MI = Multiple Imputation; SS-EAP = summed-score expected a posteriori; PSD = Posterior Standard Deviation.

Figure 4.

The distribution of raw summed scores in pediatric and adult scales. The raw summed scores in both scales are grouped into bins with equal width of 4 ranging from 0 to 32.

PSD adjustment is not as much needed for both scales when the calibration sample size is large enough, no matter what the projection direction is. It can be found in both tables that the PSDs are on average larger for the projected scale, especially when projecting from adult to pediatric scale. Among both projection directions, the largest r is 0.054 in the adult scale when projecting from adult scale to pediatric scale. This means that the largest sampling error is only about 5% of the measurement error.

Discussion

Overall, the current research focuses on accounting for the carry-over sampling variability in a calibrated projection design, when the item parameters of the two scales to be linked are fixed at their published values. This study not only serves as an addition to the original calibrated projection study in Thissen et al. (2015) in that it provides a way to account for the impact of sampling variability but also in that it presents important findings and implications on IRT scoring by using information from previous studies.

The value of r gives the necessity to perform the PSD adjustment, and it reflects the relative uncertainty of the sampling error to the measurement error. When r values are large, there are big differences between the adjusted and unadjusted scoring values—we are understating the uncertainty in IRT scores if we do not do the adjustment. Thus, it is crucial to do the adjustment in this calibrated projection design if the r is large. It is recommended to always calculate the r in a calibrated projection design, since it helps to decide whether the adjustment needed is substantial.

There are three potentially fruitful future research directions to address current limitations. First, besides MI-based method, other methods such as a fully Bayesian method to account for the impact of sampling variability worth more explorations. Investigations are needed to compare the performance of other methods with the MI-based method in the framework of calibrated projection. Second, in the real data analysis, the construction of ACM is based on an approximation without knowing the exact design. Because detailed information for missing data is not available, it is not guaranteed the approximation accurately reflects the real scenario. Further work is needed to evaluate the performance of the approximation of ACM in real data. Third, the method in this paper can be extended to many other situations when the model is more complex. In addition to the one-to-one linking, calibrated projection procedures can be extended to many-to-one (Thissen et al., 2015) and many-to-many linking. Moreover, the performance of proposed methods in the current study can be further explored in more complicated statistical models such as multilevel IRT models (e.g., Fox, 2005) and two-tier models (Cai, 2010).

Supplemental Material

Supplemental Material—Characterizing Sampling Variability for Item Response Theory Scale Scores in a Fixed-Parameter Calibrated Projection Design

Supplemental Material for Characterizing Sampling Variability for Item Response Theory Scale Scores in a Fixed-Parameter Calibrated Projection Design by Shuangshuang Xu and Yang Liu in Applied Psychological Measurement

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Shuangshuang Xu

Supplemental Material

Supplemental material for this article is available online.

References

Bock

R. D.

Aitkin

(1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443–459. https://doi.org/10.1007/BF02293801

Cai

(2010). A two-tier full-information item factor Analysis model with applications. Psychometrika, 75(4), 581–612. https://doi.org/10.1007/s11336-010-9178-0

Cella

Riley

Stone

Rothrock

Reeve

Yount

Amtmann

Bode

Buysse

Choi

Cook

DeVellis

DeWalt

Fries

J. F.

Gershon

Hahn

E. A.

Lai

J. S.

Pilkonis

Revicki

Hays

(2010). The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194. https://doi.org/10.1016/j.jclinepi.2010.04.011

Chalmers

(2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, Articles, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

Cheng

Yuan

K.-H.

(2010). The impact of fallible item parameter estimates on latent trait recovery. Psychometrika, 75(2), 280–291. https://doi.org/10.1007/s11336-009-9144-x

Core Team

(2019). R: A language and environment for statistical computing. [Computer software manual]: https://www.R-project.org

De Ayala

(1994). The influence of multidimensionality on the graded response model. Applied Psychological Measurement, 18(2), 155–170. https://doi.org/10.1177/014662169401800205

Fox

J. P.

(2005). Multilevel IRT using dichotomous and polytomous response data. British Journal of Mathematical and Statistical Psychology, 58(1), 145–172. https://doi.org/10.1348/000711005X38951

Gong

Samaniego

F. J.

(1981). Pseudo maximum likelihood estimation: Theory and applications. The Annals of Statistics, 9(4), 861–869. https://www-jstor-org.web.bisu.edu.cn/stable/2240854

10.

Huang

Stucky

B. D.

Edelen

M. O.

Tucker

J. S.

Shadel

W. G.

Hansen

Cai

(2016). Calibration of the Spanish PROMIS smoking item banks. Nicotine & Tobacco Research, 18(7), 1635–1641. https://doi.org/10.1093/ntr/ntw005

11.

Irwin

Stucky

Langer

Thissen

DeWitt

E. M.

Lai

J.-S.

Varni

J. W.

Yeatts

DeWalt

(2010). An item response analysis of the pediatric PROMIS anxiety and depressive symptoms scales. Quality of Life Research, 19(4), 595–607. https://doi.org/10.1007/s11136-010-9619-3

12.

Kolen

M. J.

Brennan

R. L.

(2014). Test equating, scaling, and linking: Methods and practices. Springer Science & Business Media

13.

LaFlair

G. T.

Isbell

May

L. D. N.

Arvizu

M. N. G.

Jamieson

(2017). Equating in small-scale language testing programs. Language Testing, 34(1), 127–144. https://doi.org/10.1177/0265532215620825

14.

Y. H.

Lissitz

R. W.

(2004). Applications of the analytically derived asymptotic standard errors of item response theory item parameter estimates. Journal of Educational Measurement, 41(2), 85–117. https://doi.org/10.1111/j.1745-3984.2004.tb01109.x

15.

Linn

R. L.

McLaughlin

Thissen

(2009). Utility and validity of NAEP linking efforts. American Institutes for Research. https://eric.ed.gov/?id=ED506806

16.

Liu

Yang

J. S.

(2018). Interval estimation of latent variable scores in item response theory. Journal of Educational and Behavioral Statistics, 43(3), 259–285. https://doi.org/10.3102/1076998617732764

17.

Liu

Yang

J. S.

Maydeu-Olivares

(2019). Restricted recalibration of item response theory models. Psychometrika, 84(2), 529–553. https://doi.org/10.1007/s11336-019-09667-4

18.

Livingston

S. A.

Kim

(2010). Random-groups equating with samples of 50 to 400 test takers. Journal of Educational Measurement, 47(2), 175–185. https://www-jstor-org.web.bisu.edu.cn/stable/20778946

19.

Monroe

(2019). Estimation of expected Fisher information for IRT models. Journal of Educational and Behavioral Statistics, 44(4), 431–447. https://doi.org/10.3102/1076998619838240

20.

OECD (2016). PISA 2015 results (Volume I). https://doi.org/10.1787/9789264266490-en

21.

Parke

W. R.

(1986). Pseudo maximum likelihood estimation: The asymptotic distribution. The Annals of Statistics, 14(1), 355–357. https://www-jstor-org.web.bisu.edu.cn/stable/2241287

22.

Patz

R. J.

Junker

B. W.

(1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146–178. https://doi.org/10.3102/10769986024002146

23.

Phillips

Jia

Wise

L. L.

Wiley

Diaz

T. E.

Rahman

(2014). NAEP-TIMSS linking study: Technical report on the linking methodologies and their evaluations (NCES 2014-461) (p. 2011). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education

24.

Pilkonis

P. A.

Choi

S. W.

Reise

S. P.

Stover

A. M.

Riley

W. T.

Cella

Group

P. C.

(2011). Item banks for measuring emotional distress from the patient-reported outcomes measurement information system (PROMIS®): Depression, anxiety, and anger. Assessment, 18(3), 263–283. https://doi.org/10.1177/1073191111411667

25.

Thissen

Liu

Magnus

Quinn

(2015). Extending the use of multidimensional IRT calibration as projection: Many-to-One linking and linear computation of projected scores. In van der Ark

L. A.

Bolt

D. M.

Wang

W.-C.

Douglas

J. A.

Chow

S.-M.

(Eds.), Quantitative psychology research (pp. 1–16). Springer International Publishing.

26.

Thissen

Varni

J. W.

Stucky

B. D.

Liu

Irwin

D. E.

DeWalt

D. A.

(2011). Using the PedsQL^TM 3.0 asthma module to obtain scores comparable with those of the PROMIS pediatric asthma impact scale (PAIS). Quality of Life Research, 20(9), 1497–1505. https://doi.org/10.1007/s11136-011-9874-y

27.

Tucker

J. S.

Shadel

W. G.

Edelen

M. O.

Stucky

B. D.

Hansen

Cai

(2014). Development of the PROMIS® positive emotional and sensory expectancies of smoking item banks. Nicotine & Tobacco Research, 16(Suppl_3), S212–S222. https://doi.org/10.1093/ntr/ntt281

28.

Yang

J. S.

Hansen

Cai

(2012). Characterizing sources of uncertainty in item response theory scale scores. Educational and Psychological Measurement, 72(2), 264–290. https://doi.org/10.1177/0013164411410056

29.

Zhang

Zhao

(2019). Standard errors of IRT parameter scale transformation coefficients: Comparison of bootstrap method, delta method, and multiple imputation method. Journal of Educational Measurement, 56(2), 302–330. https://doi.org/10.1111/jedm.12210

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.27 MB