Applied Psychological Measurement

In this study, the estimation of extremely large or extremely small proficiency levels, given the item parameters of a logistic item response model, is investigated. On one hand, the estimation of proficiency levels by maximum likelihood (ML), despite being asymptotically unbiased, may yield infinite estimates. On the other hand, with an appropriate prior distribution, the Bayesian approach of maximum a posteriori (MAP) yields finite estimates, but it suffers from severe estimation bias at the extremes of the proficiency scale. As a first step, a simple correction to the MAP estimator is proposed to reduce this estimation bias. The correction factor is determined through a simulation study and depends only on the length of the test. In a second step, some additional simulations emphasize that the corrected estimator behaves like the ML estimator and outperforms the standard MAP method for extremely small or extremely large abilities. Although based on the Rasch model, the method could be adapted to other logistic item response models.

In many practical testing situations, alternate test forms from the same testing program are not strictly parallel to each other and instead the test forms exhibit small psychometric differences. This article investigates the potential practical impact that these small psychometric differences can have on expected classification accuracy. Ten different sets of tests were assembled by minimizing the differences in test information at five θ locations. The impact of the psychometric differences between the assembled test forms was quantified for two different groups of simulated examinees across a range of possible cut scores. Results indicated that using sequential or simultaneous test assembly is preferred to random test assembly. Analyses also implied that the small differences in the psychometric properties between tests produced differences in overall classification accuracy that were less than 1.5%. The biggest differences in classification accuracy were found when the test information functions were not as well matched in regions where there were more examinees. Although these differences were fairly small, they may have the potential to have a practically significant impact on decision making. It is suggested that classification accuracy critically depends on the differences in test information, the location of the cut score, and the groups of examinees considered.

DETECT is a nonparametric methodology to identify the dimensional structure underlying test data. The associated DETECT index, D_max , denotes the degree of multidimensionality in data. Conditional covariances (CCOV) are the building blocks of this index. In specifying population CCOVs, the latent test composite θ _TT is used as the conditional variable. In estimating the CCOVs, the total test score of all items in the test (T) or the rest score of remaining items (S) are generally used as estimates of the latent composite θ _TT . However, estimated CCOVs are biased when using T or S as a proxy for θ _TT . Some type of correction is needed to adjust this bias. This study was an investigation of different ways to estimate the DETECT index based on the conditional scores T and S, and additional bias adjustments, resulting in six different estimates, D ₁ through D ₆. These six indices were investigated in simulated settings, varying the test length, sample size, and the degree of multidimensionality (108 in all). The results showed that indices D ₁, D ₂, and D ₅ are not acceptable as they displayed highly inflated D _max values. No statistically significant differences were found between indices D ₃ and D ₆. Overall comparison of indices D ₃ with D ₄ showed that even though they differed significantly in D _max values, there was no practically meaningful difference between the performance of these two indices. For these reasons, it is recommended that the current index D ₃ be retained even though index D ₄ displayed slightly better results in the current study.

Differential item functioning (DIF) occurs when an item on a test, questionnaire, or interview has different measurement properties for one group of people versus another. One way to test items with ordinal response scales for DIF is likelihood ratio (LR) testing using item response theory (IRT), or IRT-LR-DIF. Despite the various advantages of IRT-LR-DIF, one disadvantage is that the latent variable is usually assumed to be normally distributed. If this normality assumption is violated, nonparametric alternatives such as the Mantel test, generalized Mantel—Haenszel (GMH) test, and poly-SIBTEST may be preferable. Simulations were carried out to compare IRT-LR-DIF to poly-SIBTEST and the GMH and Mantel tests when the latent density is nonnormal for both groups but presumed normal for IRT-LR-DIF. Results indicated that latent nonnormality detrimentally affected all three procedures, but IRT-LR-DIF was surprisingly more robust to latent nonnormality than all of the nonparametric approaches.

This article refutes a recent claim that computer-based tests produce biased scores for very proficient test takers who make mistakes on one or two initial items and that the ‘‘bias’’ can be reduced by using a four-parameter IRT model. Because the same effect occurs with pattern scores on nonadaptive tests, the effect results from IRT scoring, not from adaptive testing. Because very proficient test takers rarely err on items of middle difficulty, the so-called bias is one of selective data analysis. Furthermore, the apparently large score penalty for one error on an otherwise perfect response pattern is shown to result from the relative stretching of the IRT scale at very high and very low proficiencies. The recommended use of a four-parameter IRT model is shown to have drawbacks.