Abstract
Recently, there has been increasing interest in adopting the forced-choice (FC) test format in non-cognitive assessments, as it demonstrates faking resistance when well-designed. However, traditional or manual pairing approaches to FC test construction are time- and effort- intensive and often involve insufficient considerations. To address these issues, we developed the new open-source autoFC R package to facilitate automated and optimized item pairing strategies. The autoFC package is intended as a practical tool for FC test constructions. Users can easily obtain automatically optimized FC tests by simply inputting the item characteristics of interest. Customizations are also available for considerations on matching rules and the behaviors of the optimization process. The autoFC package should be of interest to researchers and practitioners constructing FC scales with potentially many metrics to match on and/or many items to pair, essentially exempting users from the burden of manual item pairing and reducing the computational costs and biases induced by simple ranking methods.
Forced-choice (FC) tests have gained increasing attention and popularity among researchers and practitioners, for their faking-resistance when well-designed (Cao & Drasgow, 2019). A well-designed FC test often entails: (1) each item within a block measures a different latent trait, and (2) items within a block have similar level of social desirability (Zhang et al., 2020). Recently, Pavlov et al. (2021) suggested that items within a block should be matched on inter-item agreement of their social desirability ratings to resolve potential issues of matching based on mean desirability ratings. Depending on the scoring models, FC developers may also need to maximize factor loading differences (Brown & Maydeu-Olivares, 2011) or minimize item location differences (Cao & Drasgow, 2019) for items within a block for statistical reasons. Deciding on what items should be assigned to the same block—item pairing or matching—is thus critical to the quality of an FC test. Item pairing can be essentially seen as an optimization process, which is currently carried out manually. However, given that we often need to simultaneously meet multiple objectives (e.g., minimized desirability differences and maximized loading differences), manual pairing becomes impractical and even infeasible once the number of latent traits and/or the number of items per trait are relatively large.
Description of the Package
The autoFC package is developed to facilitate automatic FC test construction. It offers users the functionality to (1) customize one or more item pairing criteria and calculate a composite pairing index, termed "energy", with user-specified weights for each criterion, (2) automatically optimize the energy for the whole test by sequentially or simultaneously optimizing each matching rule, through the exchange of items among blocks or replacement with unused items, and (3) construct parallel forms of the same test following the same pairing rules. Users can create an FC test of any block size (e.g., pairs, triplets, quadruplets).
Functions of autoFC have default settings, which allows users to obtain automatically constructed assembled tests by only providing item characteristics for each item. However, users are encouraged to manipulate arguments to change default values. Argument customizations are enabled for matching rules related to each item characteristic, their relative weights an initial paired test for further improvement, and person-level item responses if items are matched based on inter-item agreement metrics (Pavlov et al., 2021). Users can also control the behavior of optimization by modifying the corresponding arguments in the function.
Availability and Distribution
autoFC is an open-source package written in the R language (R Core Team, 2021) and is compatible with all major platforms (i.e., Windows, Linux, and MacOS). All source code and documentation files are freely available from the Comprehensive R Archive Network (CRAN) via https://CRAN.R-project.org/package=autoFC. The development version of the package can be tracked at https://github.com/tspsyched/autoFC.
Footnotes
Author’s Notes
We thank Dr. Fritz Drasgow at the University of Illinois at Urbana-Champaign for his valuable suggestions during the development of this project.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
