Detection Rates of the M 2 Test for Nonzero Lower Asymptotes Under Normal and Nonnormal Ability Distributions in the Applications of IRT

Abstract

When considering the two-parameter or the three-parameter logistic model for item responses from a multiple-choice test, one may want to assess the need for the lower asymptote parameters in the item response function and make sure the use of the three-parameter item response model. This study reports the degree of sensitivity of an overall model test M₂ to detecting the presence of nonzero asymptotes in the item response function under normal and nonnormal ability distribution conditions.

Keywords

3PL model guessing lower asymptote nonnormality of ability distribution

When item response theory (IRT) is used for research and practice, the two-parameter logistic (2PL) and three-parameter logistic (3PL) models using the marginal maximum likelihood (MML) estimation where population ability distribution is assumed to be normal (MML-normal in short) are popular. If item responses are from a multiple-choice test, one may want to assess the necessity of pseudo guessing parameters in the item response function (IRF) and make sure the use of the 3PL model given the difficulty of the 3PL model estimation compared to the 2PL model. This study examines the sensitivity of an overall model–data fit test M₂ (Joe & Maydeu-Olivares, 2010; Maydeu-Olivares & Joe, 2005, 2006) to detecting the presence of nonzero lower asymptotes in the IRF via simulation. (Compared to the popular likelihood ratio test for nested models as a relative model fit test (Haberman, 1977), M₂ is an overall (or absolute) model fit test known as a limited information test and was proposed to overcome the problem of sparse data in the well-known full-information overall model–data fit tests such as Pearson’s $χ^{2}$ and the $G^{2}$ .) In addition, the population ability distribution shapes were systematically varied in the investigation of the M₂ performance to detect the nonzero lower asymptotes. Past research indicates that M₂ does not have enough power to detect the misspecification of the nonnormality of ability distribution (e.g., Hansen, Cai, Monroe, & Li, 2014; Li & Cai, 2012). Although the nonnormality of ability distribution is included in this study, the major focus is on the sensitivity of the M₂ to the presence of the nonzero lower asymptote parameters as the ability distribution changes from normal to skewed distributions. If M₂ is not sensitive to the violation of normality of ability distribution, but more sensitive to the misspecification of the IRF, we conjectured that a nonnormal distribution, specifically a positively skewed (PS) distribution, could provide a favorable environment for M₂, increasing its power to detect the nonzero lower asymptotes because more data at the lower ability range are present by a PS ability distribution than other distributions such as normal. Maydeu-Olivares and Montano (2013) evaluated the power of M₂ empirically and analytically, but their evaluation did not include fitting the 2PL to the 3PL model data through which a direct evaluation of M₂ performance on detecting nonzero lower asymptotes can be made.

The key to determine the performance of M₂ for detecting the nonzero lower asymptotes when the 2PL model is fitted to the data following the 3PL model is how much discrepancy the misspecification of the nonzero lower asymptotes makes between observed and model-expected first- and the second-order marginals. Typically item response data near the low or very low ability range are not as many as near the middle ability range. This means that the discrepancy due to the nonzero asymptotes may not be clearly manifest enough for M₂ to have high power to detect the nonzero asymptotes with an ability distribution such as normal. It is conjectured that performance of M₂ for detecting the nonzero lower asymptotes could be better with a PS ability distribution than a normal or a negatively skewed (NS) ability distribution.

Study Design

The data simulation factors were test length (TL = 15 and 30), sample size (N = 500 and 1,500), IRF forms (2PL or 3PL model), and five different ability distributions: positively skewed nonnormal2 (PS-nonnormal2), positively skewed nonnormal (PS-nonnormal), standard normal, negatively skewed nonnormal (NS-nonnormal), and negatively skewed nonnormal2 (NS-more-nonnormal2). The four factors resulted in 40 data generation conditions $(2 \times 2 \times 2 \times 5)$ . In each of the conditions, 500 replications were made. Each replication data set was fitted by the 2PL model with the MML-normal estimation. Statistical significance of M₂ was evaluated under the nominal significance of .05. The statistical computing language R (R Core Team, 2015) and the “mirt” (Chalmers, 2012) package in R were used to generate item response data, estimate the 2PL model, and calculate M₂. Item slope (a), difficulty $(b)$ , and lower asymptote $(g)$ parameters were generated from truncated normal distributions: $a ~ N (1.5, 0 . 5^{2})$ with $a \in (0.5, 2.5)$ ; $b ~ N (0, 1)$ with $b \in (- 2, 2)$ ; and $g ~ N (0.23, 0 : 1^{2})$ with $g \in (0.15, 0.3)$ . Person abilities were drawn from the five distributions previously mentioned. The four nonnormal distributions were specified by linear transformations of $X$ , $Y = AX + B$ , where $X$ follows a chi-square distribution with $ν$ degrees of freedom, $X ~ χ^{2} (ν)$ . For the PS-nonnormal2, $v = 8$ , $A = 0.25$ , and $B = - 2$ . For the PS-nonnormal, $v = 8$ , $A = 0.175$ , and $B = - 1.9$ . The NS-nonnormal2 and the NS-nonnormal were obtained by multiplying −1 to the PS-nonnormal2 and the PS-nonnormal, respectively. Under these linear transformations, $E (Y) = - 0.5$ and $\sqrt{Var (Y)} = 0.7$ for PS-nonnormal2, $E (Y) = 0$ and $\sqrt{Var (Y)} = 1$ for both PS-nonnormal and NS-nonnormal, and $E (Y) = 0.5$ and $\sqrt{Var (Y)} = 0.7$ for NS-nonnormal2. The mean, standard deviation (SD), skewness, and excessive kurtosis of the simulated abilities for the nonnormal distributions are summarized in Table 1.

Table 1.

Mean, Standard Deviation (SD), Skewness, and Excessive Kurtosis for Simulated Abilities.

	M	SD	Skewness	Excessive kurtosis
PS-more-nonnormal	–0.5	0.7	0.99	1.46
PS-nonnormal	0	1	0.99	1.45
Normal	0	1	0	–0.01
NS-nonnormal	0	1	–0.99	1.45
NS-more-nonnormal	0.5	0.7	–0.99	1.45

Note. PS = positively skewed; NS = negatively skewed.

Results

The convergence of the model estimation was achieved for all replicated data sets for most of the conditions, passing the second-order test which indicates if the model solution is a potential local maximum. Six of the total 40 conditions had one through four nonconverged cases out of 500 replications. Those nonconverged estimation results were excluded in the calculation of the rejection rates by M₂. The rejection rates of M₂ are given in Table 2. (Per reviewer’s request, a separate simulation under the same data generating conditions was conducted for the likelihood ratio test for nested models comparing the 2PL and 3PL models. The results of this simulation are not included here, but available upon request from the authors.)

Table 2.

Rejection Rates of M_2.

	Test length = 15		Test length = 30
Ability distribution	N = 500	N = 1,500	N = 500	N = 1,500
Zero lower asymptote (2PL IRF model data)
PS-nonnormal2	0.042	0.04	0.034	0.014
PS-nonnormal	0.048	0.064	0.056	0.1
Normal	0.052	0.034	0.062	0.044
NS-nonnormal	0.058	0.066	0.064	0.086
NS-nonnormal2	0.028	0.036	0.014	0.028
Nonzero lower asymptote (3PL IRF model data)
PS-nonnormal2	0.058	0.074	0.104	0.136
PS-nonnormal	0.136	0.204	0.411	0.604
Normal	0.098	0.316	0.206	0.506
NS-nonnormal	0.058	0.104	0.122	0.16
NS-nonnormal2	0.038	0.062	0.04	0.064

Note. PL = parameter logistic; IRF = item response function; PS = positively skewed; NS = negatively skewed.

The power of M₂ to detect nonnormality was poor, that is, around the nominal significance level of 5% for most conditions and never exceeding 10% (see the results in the 2PL IRF model data in Table 2). The power of M₂ to detect the nonzero lower asymptotes (i.e., results for the 3PL IRF model data in Table 2) was not satisfactory overall. Precisely speaking, the rejection rates for the 3PL IRF model data were the results due to both nonnormality of ability distributions and the nonzero lower asymptotes, but based on the observation of little or small impact of the nonnormality, we interpreted that the rejection rates of the 3PL IRF model data conditions were the results much more heavily affected by the nonzero lower asymptotes. The results without any degree of confounding by the nonnormal ability distribution are the rejection rates from the normal condition in the 3PL IRF model data. The conditions which exhibited more than 50% power with the nonzero lower asymptote data were only two: when TL is 30, N = 1,500, and the ability distribution is either normal or PS-nonnormal.

The observed weak power of M₂ to detect nonnormality is consistent with the previously cited studies. Our study provided confirming further results for different nonnormal ability distribution conditions. The low power of M₂ in this scenario (i.e., zero lower asymptote with nonnormality) may be linked to and interpreted as robustness of the 2PL model in predicting the first- and the second-order marginal subtables. Future studies that investigate the differences in those lower order marginals in detail could be conducted to corroborate this conjecture. Regarding the major focus of this study on the sensitivity of M₂ to detect nonzero lower asymptotes, the current finding appears to suggest a large sample size and a long test for M₂ to show satisfactory power. For M₂ to exhibit more than half a chance to detect nonzero lower asymptotes with the assumption of normal ability held in the data, more than 30 items in a test and more than a sample size of 1,500 seem to be necessary based on the present study results. Note that the lower asymptotes were simulated from a truncated normal distribution with its mean equal to .23. If the average of lower asymptotes is smaller than this, a much longer test and a much larger sample size would be required. Finally, for the aforementioned conjecture about the effect of a PS ability distribution on the detection of nonzero lower asymptotes, it was observed when the TL is large (30), with which the PS-nonnormal condition showed higher power than the normal condition. However, for the PS-nonnormal2 condition, the PS nature of the nonnormal data did not help to raise the power of M₂ to detect nonzero lower asymptotes. Although the PS-nonnormal2 had the same degree of nonnormality as PS-nonnormal in terms of skewness and kurtosis, the use of different mean and SD values as 0 and 1, respectively, has practically the equivalent design effect that produces more difficult items and higher discriminating items compared to the conditions having the mean and SD values equal to 0 and 1, respectively. Thus, in the PS-nonnormal2 condition, the effect of the PS distribution seems to be offset by this design effect in terms of the manifestation of the guessing effect in the lower order marginals. Research on adjusting the M₂ test or alternative tests or procedures to improve the detection of nonzero lower asymptotes in IRT applications could be explored further.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Chalmers

P. R.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. Retrieved from http://www.jstatsoft.org/v48/i06/

Haberman

S. J.

(1977). Log-linear models and frequency tables with small expected cell counts. The Annals of Statistics, 5, 1148-1169.

Hansen

Cai

Monroe

(2014). Limited-information goodness-of-fit testing of diagnostic classification item response theory models (CRESST Report 840). Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing, University of California.

Joe

Maydeu-Olivares

(2010). A general family of limited information goodness-of-fit statistics for multinomial data. Psychometrika, 75, 393-419.

Cai

(2012, July). Summed score based fit indices for testing latent variable distribution assumption in IRT. Paper presented at the 2012 International Meeting of the Psychometric Society, Lincoln, NE.

Maydeu-Olivares

Joe

(2005). Limited-and full-Information estimation and goodness-of-fit testing in 2p contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009-1020.

Maydeu-Olivares

Joe

(2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71, 713-732.

Maydeu-Olivares

Montano

(2013). How should we assess the fit of Rasch-type models? Approximating the power of goodness-of-fit statistics in categorical data analysis. Psychometrika, 78, 116-133.

R Core Team. (2015). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from https://www.R-project.org/