Journal of Educational and Behavioral Statistics

The detection of differential item functioning (DIF) is important in psychological and behavioral sciences. Standard DIF detection methods perform an item-by-item test iteratively, often assuming that all items except the one under investigation are DIF-free. This article proposes a Bayesian adaptive Lasso method to detect DIF in graded response models (GRMs), where the DIF effects for all items can be identified simultaneously. The multiple-group GRMs are specified, and the possible DIF effects for each item are reparameterized using the increment components. Then, a Bayesian adaptive Lasso procedure is developed for parameter estimation, in which DIF effects can be automatically obtained. Our method is evaluated and compared with the commonly used likelihood ratio test method in a simulation study. The results show that our method can recover most model parameters well and has better control of false positive rates in almost all conditions. An application is presented using data from the National Longitudinal Study of Adolescent to Adult Health (Add Health).

The issue of model uncertainty has been gaining interest in education and the social sciences community over the years, and the dominant methods for handling model uncertainty are based on Bayesian inference, particularly, Bayesian model averaging. However, Bayesian model averaging assumes that the true data-generating model is within the candidate model space over which averaging is taking place. Unlike Bayesian model averaging, the method of Bayesian stacking can account for model uncertainty without assuming that a true model exists. An issue with Bayesian stacking, however, is that it is an optimization technique that uses predictor-independent model weights and is, therefore, not fully Bayesian. Bayesian hierarchical stacking, proposed by Yao et al. further incorporates uncertainty by applying a hyperprior to the stacking weights. Considering the importance of multilevel models commonly applied in educational settings, this paper investigates via a simulation study and a real data example the predictive performance of original Bayesian stacking and Bayesian hierarchical stacking along with two other readily available weighting methods, pseudo-BMA and pseudo-BMA bootstrap (PBMA and PBMA+). Predictive performance is measured by the Kullback–Leibler divergence score. Although the differences in predictive performance among these four weighting methods in Bayesian stacking are small, we still find that Bayesian hierarchical stacking performs as well as conventional stacking, PBMA, and PBMA+ in settings where a true model is not assumed to exist.

Research in education and behavioral sciences often involves the use of latent variable models that are related to indicators, as well as related to covariates or outcomes. Such models are subject to interpretational confounding, which occurs when fitting the model with covariates or outcomes alters the results for the measurement model. This has received attention in models for continuous observable variables but to date has not been examined in the context of discrete variables. This work demonstrates that interpretational confounding can occur in models for discrete variables, and develops a multistage Bayesian estimation approach to deal with this problem. The key features of this approach are that it is (a) measurement preserving, in that it precludes the possibility of interpretational confounding, and (b) uncertainty preserving, in that the uncertainty from the earlier stage of estimating the measurement model is propagated to the second stage of estimating the relations between the latent variable(s) and any covariates or outcomes. Previous work on these methods had only considered models for continuous observed variables, and software was limited to models with a single latent variable and either covariates or outcomes. This work extends the approach and software to a more general class of solutions, including discrete variables, illustrating the procedures with analyses of real data. Functions for conducting the analyses in widely available software are provided.

Popular item response theory (IRT) models are considered complex, mainly due to the inclusion of a random factor variable (latent variable). The random factor variable represents the incidental parameter problem since the number of parameters increases when including data of new persons. Therefore, IRT models require a specific estimation method and large samples for accurate parameter estimation. The two-parameter IRT model is redefined by analytically integrating out the random person factor in the latent response formulation of the model to make it suitable for small sample applications. This IRT Bayesian covariance structure model (IRT-BCSM) describes the clustering of (latent) responses by persons through a structured covariance matrix in which a common covariance parameter represents the dependence implied by a unidimensional random factor variable. The IRT-BCSM is a latent-variable-free model and consists of (fixed) item parameters and a common covariance parameter, where person parameters can be post-hoc sampled. An efficient Gibbs sampler is proposed for parameter estimation. In simulation studies, the performance of the IRT-BCSM is compared to two-parameter IRT models for small samples, and results show an optimal performance of the IRT-BCSM even in sample sizes as small as 50 to 100 persons and 5 to 10 items. Generalizations of the IRT-BCSM show that a redefinition of more complex IRT models also lead to much more efficient parameterizations, which can broaden the scope of IRT applications.

Item compromise has long posed challenges in educational measurement, jeopardizing both test validity and test security of continuous tests. Detecting compromised items is therefore crucial to address this concern. The present literature on compromised item detection reveals two notable gaps: First, the majority of existing methods are based upon a non-Bayesian framework; second, many of these approaches exclusively rely on examinees’ responses for detection, neglecting valuable data such as response times. In this study, we propose a Bayesian change-point method that integrates both responses and response times to detect compromised items in continuous tests. This two-phase approach is designed for iterative use. The accuracy and efficiency of the proposed method are assessed in three simulations and an operational data example. The results demonstrate the method’s effectiveness in accurately and efficiently detecting compromised items. Additionally, the incorporation of response times significantly enhances both detection accuracy and efficiency.

This study proposes a Bayesian method for diagnostic classification models (DCMs) for a partially known Q-matrix setting between exploratory and confirmatory DCMs. This Q-matrix setting is practical and useful because test experts have pre-knowledge of the Q-matrix but cannot readily specify it completely. The proposed method employs priors for the Bayesian variable selection to simultaneously estimate the effects of active and nonactive attributes, and the simulations lead to appropriate attribute recovery rates. Furthermore, the proposed method recovers the attribute mastery of individuals at the same as for a fully known Q-matrix. In addition, the proposed methods can be used to estimate the unknown Q-matrix part. A real data example indicates that the proposed Bayesian estimation method for the partially known Q-matrix fits better than a fully specified Q-matrix. Finally, extensions and future research directions are discussed.