Asymptotic Standard Errors of Parameter Scale Transformation Coefficients in Test Equating Under the Nominal Response Model

Abstract

Researchers have developed a characteristic curve procedure to estimate the parameter scale transformation coefficients in test equating under the nominal response model. In the study, the delta method was applied to derive the standard error expressions for computing the standard errors for the estimates of the parameter scale transformation coefficients. This brief report presents the results of a simulation study that examined the accuracy of the derived formulas and compared the performance of this analytical method with that of the multiple imputation method. The results indicated that the standard errors produced by the delta method were very close to the criterion standard errors as well as those yielded by the multiple imputation method under all the simulation conditions.

Keywords

equating standard errors nominal response model delta method multiple imputation method

The nominal response model (NRM, Bock, 1972) is an item response theory (IRT, Baker & Kim, 2004) model for analyzing items that elicit nominal or categorical responses, which has been recently used to examine the functioning of categories for items with ordered categories (e.g., Preston et al., 2011; Preston et al., 2015). Under the NRM, the probability of a respondent with ability level $θ$ responding in a given category $t (t = 1, 2, \dots, m_{g})$ for an item $g (g = 1, 2, \dots, n)$ in Test $X$ denoted by $P_{Xgt}$ can be written as

P_{Xgt} = \frac{\exp (a_{Xgt} θ + c_{Xgt})}{\sum_{h = 1}^{m_{g}} \exp (a_{Xgh} θ + c_{Xgh})},

(1)

where $m_{g}$ is the number of categories for item $g$ , $n$ is the number of items, and $a_{Xgt}$ and $c_{Xgt}$ are the slope and intercept parameters for response category $t$ of item $g$ , respectively. For model identification, it is usually to impose constraints on item category slope and intercept parameters such that $\sum_{t = 1}^{m_{g}} a_{Xgt} = 0$ and $\sum_{t = 1}^{m_{g}} c_{Xgt} = 0$ .

One of the aspects of IRT equating is to place the parameter estimates for the items on the tests to be equated onto the same scale (Kolen & Brennan, 2014). This normally can be accomplished by conducting linear transformations on the estimates of the item parameters. Let us suppose that the parameters for the items in Test Y are transformed to the scale of the parameters for the items in Test X. In test equating under the NRM, the item category slope ( $a_{Ygt}$ ) and intercept ( $c_{Ygt}$ ) parameters for the items on Test Y can be transformed to the parameter scale of Test X via the following linear transformations (Baker, 1993; Kim & Hanson, 2002; Kim & Lee, 2006):

a_{Ygt}^{*} = \frac{a_{Ygt}}{A},

(2)

c_{Ygt}^{*} = c_{Ygt} - \frac{B}{A} a_{Ygt},

(3)

where A and B in Equations 2 and 3 are the two parameter scale transformation coefficients. A characteristic curve procedure for obtaining A and B in test equating under the NRM has been proposed by Baker (1993) and then revised by Kim and Hanson (2002).

The reporting of standard errors of equating has been advocated as a standard practice when conducting test equating (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014). Despite the availability of the characteristic curve procedure for finding A and B, the approaches to estimating the standard errors for the estimates of A and B have not been developed and examined. There are at least three approaches to obtaining the standard errors for the estimates of A and B in test equating under the IRT models: the bootstrap method (Kolen & Brennan, 2014), the delta method (Andersson, 2018; Ogasawara, 2001; Zhang, 2020a), and the multiple imputation method (Zhang & Zhao, 2019). The bootstrap method is a resampling approach. It requires an IRT calibration on each bootstrap sample, which makes it very computationally intensive and time-consuming (Kolen & Brennan, 2014). Therefore, it was not considered in the study. The multiple imputation method makes use of multiple sets of random imputations of the item parameter estimates to compute the standard errors of equating coefficients, which is significantly less computationally demanding than the bootstrap method (Zhang, 2020b; Zhang & Zhao, 2019). The delta method is an analytical approach, the application of which depends on the development of relatively complicated mathematical formulas for the standard error expressions (e.g., Andersson, 2018; Ogasawara, 2001; Zhang, 2020a). However, provided that the mathematical formulas are derived, the delta method is much more efficient than both the bootstrap method and the multiple imputation method. More details about the comparison of the three standard error estimation procedures can be seen elsewhere (e.g., Kolen & Brennan, 2014; Zhang, 2020b; Zhang & Zhao, 2019).

In the study, the delta method, which has been applied to obtain the standard errors for the estimates of the parameter scale transformation coefficients (i.e., A and B) in test equating under the dichotomous IRT models (e.g., Ogasawara, 2001) and some polytomous IRT models (e.g., Andersson, 2018; Zhang, 2020a), was used to derive the standard error expressions for computing the asymptotic standard errors for the estimates of A and B estimated using the characteristic curve method in test equating under the NRM (Baker, 1993; Kim & Hanson, 2002). The derivations of the formulas can be seen in the supplementary file. This report presents the results of the application of the formulated delta method with the stimulated data.

Simulation Study

Method

Similar to the research designs employed in the studies of Baker (1993) and Kim and Hanson (2002), for convenience but without loss of generality, it is assumed that a common test which is composed of 20 items (i.e., $g = 1, 2, \dots, 20$ ) was administered to two groups of respondents (Group 1 and Group 2). All the items in the test share the same number of categories (i.e., $k = 1, 2, \dots, m$ ). Both the item category slope and intercept parameters were sampled from a standard normal distribution in a manner that met the constraints imposed on the NRM model, that is, $\sum_{k = 1}^{m} a_{gk} = \sum_{k = 1}^{m} c_{gk} = 0$ . The item parameters estimated based on the Group 2 data were transformed to the scales developed based on the Group 1 data.

Three manipulated factors were considered in the simulation study: sample size, number of categories, and group difference in ability distributions. Three levels of sample size were simulated: $500$ , $1, 000$ , and $3, 000$ . Two levels of number of categories for the items were considered: four-category items (Baker, 1993) and six-category items (Kim & Hanson, 2002). The person parameters for the respondents in Group 1 were randomly drawn from a standard normal distribution (i.e., $θ_{1} ~ N (0, 1)$ ). The person parameters for the respondents in Group 2 were randomly generated from two distributions: $θ_{2} ~ N (0.1, 1 . 1^{2})$ versus $θ_{2} ~ N (0.5, 1 . 2^{2})$ . A combination of the different levels under the three manipulated factors resulted in a total of 12 simulated conditions ( $3 \times 2 \times 2$ ). Each of the 12 conditions was replicated 300 times.

The model calibrations were performed by using the R programming language with the package mirt (Chalmers, 2012) to fit the NRM to the simulated data. The sandwich estimator (Chalmers, 2018) was used to estimate the variance–covariance matrices for the item parameter estimates. The way how the variance–covariance matrices were transformed under different parameterizations can be seen in the supplementary file.

The R programming language was used to write code for estimating A and B and implementing the delta method to calculate the standard errors for the two coefficients. The R code can be made available upon request. For comparison, the multiple imputation method (Zhang & Zhao, 2019) was also applied to produce the standard errors for the two coefficients. The criterion standard errors were the empirical standard deviations of the estimated values of A and B calculated over 10,000 replications (Paek & Cai, 2014; Zhang & Zhao, 2019).

Results

Table 1 provides a summary of the criterion standard errors as well as the means and standard deviations of the estimated standard errors for the estimates of A and B calculated over the 300 replications. The results across the 12 simulated conditions consistently indicate that the standard errors produced by the delta method are nearly identical or extremely close to the criterion standard errors as well as those yielded by the multiple imputation method.

Table 1.

Descriptive Statistics of the Standard Errors for the Estimates of the Parameter Scale Transformation Coefficients Calculated Over the 300 Replications.

			A			B
Category	N	Ability	Empirical	Delta	MI	Empirical	Delta	MI
4	500	$N (. 1, 1 . 1^{2})$	.062	.062 (.004)	.063 (.005)	.070	.072 (.003)	.073 (.003)
		$N (. 5, 1 . 2^{2})$	.071	.071 (.005)	.072 (.005)	.079	.081 (.003)	.082 (.004)
	1,000	$N (. 1, 1 . 1^{2})$	.045	.043 (.002)	.043 (.002)	.050	.051 (.001)	.051 (.002)
		$N (. 5, 1 . 2^{2})$	.051	.048 (.002)	.049 (.002)	.056	.056 (.002)	.056 (.002)
	3,000	$N (. 1, 1 . 1^{2})$	.025	.025 (.001)	.025 (.001)	.029	.029 (.000)	.029 (.001)
		$N (. 5, 1 . 2^{2})$	.029	.028 (.001)	.028 (.001)	.032	.032 (.001)	.032 (.001)
6	500	$N (. 1, 1 . 1^{2})$	.059	.060 (.004)	.060 (.004)	.071	.072 (.002)	.073 (.003)
		$N (. 5, 1 . 2^{2})$	.067	.067 (.004)	.067 (.005)	.078	.079 (.003)	.080 (.003)
	1,000	$N (. 1, 1 . 1^{2})$	.042	.041 (.002)	.042 (.002)	.051	.050 (.001)	.051 (.002)
		$N (. 5, 1 . 2^{2})$	.047	.046 (.002)	.046 (.002)	.055	.055 (.002)	.055 (.002)
	3,000	$N (. 1, 1 . 1^{2})$	.024	.024 (.001)	.024 (.001)	.029	.029 (.000)	.029 (.001)
		$N (. 5, 1 . 2^{2})$	.027	.026 (.001)	.026 (.001)	.032	.032 (.001)	.032 (.001)

Note. Values within brackets are standard deviations. Empirical = criterion standard errors; MI = multiple imputation method;

Summary

The results of the simulation study suggest that the formulated delta approach presented in the current study can be utilized as a viable approach to determining the variability of the parameter scale transformation coefficients in test equating under the NRM. The delta method is more attractive than the bootstrap method and the multiple imputation method because it is significantly less computationally intensive and time-consuming. For example, in this simulation study, the delta method could be up to 8 times faster than the multiple imputation method which has been shown to be more computationally efficient than the bootstrap method (Zhang, 2020b; Zhang & Zhao, 2019). Hence, the delta method will be much more appealing than the multiple imputation method and the bootstrap method when the sample size is very large or when a complex linking design involving multiple test forms is employed (e.g., chain equating, Battauz, 2013). The delta method developed in the study offers researchers and practitioners a more feasible and computationally efficient procedure for facilitating the reporting of the standard errors for the parameter scale transformation coefficients in practical test equating involving the NRM.

Supplemental Material

Supplemental_Material – Supplemental material for Asymptotic Standard Errors of Parameter Scale Transformation Coefficients in Test Equating Under the Nominal Response Model

Supplemental material, Supplemental_Material for Asymptotic Standard Errors of Parameter Scale Transformation Coefficients in Test Equating Under the Nominal Response Model by Zhonghua Zhang in Applied Psychological Measurement

Footnotes

Acknowledgements

The authors would like to thank the Editor Dr. John R. Donoghue and the anonymous reviewers for their helpful and constructive comments.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Zhonghua Zhang

Supplemental Material

Supplementary material is available for this article online.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

Andersson

(2018). Asymptotic variance of linking coefficient estimators for polytomous IRT models. Applied Psychological Measurement, 42(3), 192–205.

Baker

F. B.

(1993). Equating tests under the nominal response model. Applied Psychological Measurement, 17(3), 239–251.

Baker

F. B.

Kim

S. H.

(2004). Item response theory: Parameter estimation techniques. CRC Press.

Battauz

(2013). IRT test equating in complex linkage plans. Psychometrika, 78(3), 464–480.

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51.

Chalmers

R. P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29.

Chalmers

R. P.

(2018). Numerical approximation of the observed information matrix with Oakes’ identity. British Journal of Mathematical and Statistical Psychology, 71(3), 415–436.

Kim

J. S.

Hanson

B. A.

(2002). Test equating under the multiple-choice model. Applied Psychological Measurement, 26(3), 255–270.

10.

Kim

Lee

W. C.

(2006). An extension of four IRT linking methods for mixed-format tests. Journal of Educational Measurement, 43(1), 53–76.

11.

Kolen

M. J.

Brennan

R. L.

(2014). Test equating, scaling, and linking: methods and practices (3rd ed.). Springer Science & Business Media.

12.

Ogasawara

(2001). Standard errors of item response theory equating/linking by response function methods. Applied Psychological Measurement, 25(1), 53–67.

13.

Paek

Cai

(2014). A comparison of item parameter standard error estimation procedures for unidimensional and multidimensional item response theory modelling. Educational and Psychological Measurement, 74(1), 58–76.

14.

Preston

Reise

Cai

Hays

R. D.

(2011). Using the nominal response model to evaluate response category discrimination in the PROMIS emotional distress item pools. Educational and Psychological Measurement, 71(3), 523–550.

15.

Preston

K. S. J.

Parral

S. N.

Gottfried

A. W.

Oliver

P. H.

Gottfried

A. E.

Ibrahim

S. M.

Delany

(2015). Applying the nominal response model within a longitudinal framework to construct the Positive Family Relationships Scale. Educational and Psychological Measurement, 75(6), 901–930.

16.

Zhang

Z. H.

(2020a). Asymptotic standard errors of equating coefficients using the characteristic curve methods for the graded response model. Applied Measurement in Education. Advance online publication. https://doi.org/10.1080/08957347.2020.1789142

17.

Zhang

Z. H.

(2020b). Estimating standard errors of IRT true score equating coefficients using imputed item parameters. The Journal of Experimental Education. Advance online publication. https://doi.org/10.1080/00220973.2020.1751579

18.

Zhang

Z. H.

Zhao

(2019). Standard errors of IRT parameter scale transformation coefficients: Comparison of bootstrap method, delta method, and multiple imputation Method. Journal of Educational Measurement, 56(2), 302–330.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.21 MB