Applied Psychological Measurement

The validation of cognitive attributes required for correct answers on binary test items or tasks has been addressed in previous research through the integration of cognitive psychology and psychometric models using parametric or nonparametric item response theory, latent class modeling, and Bayesian modeling. All previous models, each with their advantages and disadvantages, require item score information and do not focus on conditional validation of cognitive attributes across ability levels and individual test items. This study proposes a method of estimating the probability of correct performance on cognitive attributes across fixed ability levels. The proposed method, referred to here as the least squares distance method (LSDM), is based on the minimization of matrix norms using the Euclidean least squares distance. The LSDM does not require raw or trait scores of examinees as long as IRT estimates of the item parameters are available.

This study explores the effects of ignoring item interaction on item parameter estimation and the efficiency of using the local dependence index Q3 and the SAS NLMIXED procedure to detect item interaction under the three-parameter logistic model and the generalized partial credit model. Through simulations, it was found that ignoring positive item interaction led to overestimation for the discrimination parameters, underestimation for the difficulty parameters, and a Q3 much smaller than zero. As the guessing parameters approached zero, the overestimation for the discrimination parameters became more serious. In contrast, ignoring negative item interaction led to underestimation for the discrimination parameters, overestimation for the difficulty parameters, and a Q3 much larger than zero. As the guessing parameters approached zero, the underestimation for the discrimination parameters became less serious. A modification of posterior predictive p value for Q3 was proposed to detect item interaction and was found to work very well. Direct modeling of item interaction using NLMIXED was demonstrated.

In a randomized experiment (n = 515), a computerized and a computerized adaptive test (CAT) are compared. The item pool consists of 24 polytomous motivation items. Although items are carefully selected, calibration data show that Samejima's graded response model did not fit the data optimally. A simulation study is done to assess possible consequences of model misfit. CAT efficiency was studied by a systematic comparison of the CAT with two types of conventional fixed length short forms, which are created to be good CAT competitors. Results showed no essential administration mode effects. Efficiency analyses show that CAT outperformed the short forms in almost all aspects when results are aggregated along the latent trait scale. The real and the simulated data results are very similar, which indicate that the real data results are not affected by model misfit.

This article highlights issues associated with the use of the differential functioning of items and tests (DFIT) methodology for assessing measurement invariance (or differential functioning) with Likert-type data. Monte Carlo analyses indicate relatively low sensitivity of the DFIT methodology for identifying differential item functioning (DIF) under some conditions of differential functioning with previously recommended significance values. The differential test functioning index was extremely insensitive to differential functioning under all study conditions. The authors recommend alternative noncompensatory DIF cutoff values used to evaluate the significance of DIF for different DIF effect sizes. Additionally, contrasts between polytomous and dichotomous data are drawn, and problems with determining measurement invariance at the scale, rather than item, level for Likert scale data are highlighted.