A Seed Usage Issue on Using catR for Simulation and the Solution

Computerized adaptive testing (CAT) is attractive to test developers because of many benefits brought by its adaptive algorithm which tailors test questions to the ability of a test taker. Such benefits include, but are not limited to, CAT being more secure, more accurate, and more efficient compared to linear testing. Because of the use of a computer and the complexity of testing constraints (e.g., exposure control, content balance), simulation plays an important role in CAT research and practice. Researchers and test developers have been using the R package catR (Magis & Raîche, 2011) for CAT simulation—examples are Graf et al. (2017), Oppl et al. (2017), and Seo and Choi (2018), just to name a few. The R package catR (as of version 3.16), however, has a flaw that can potentially invalidate related simulation results.

A seed is typically used in simulation so the researcher is able to reproduce the same results when running the simulation again. The randomCAT function in catR has a parameter genSeed to specify a seed to generate the same response pattern. Passing a numerical value to genSeed does provide the same response pattern each time randomCAT is run, although the response pattern is biased which, in turn, leads to biased ability estimates.

The left panel in Figure 1 shows the results from randomCAT for a fixed length test with 100 items. Three theta ability levels (−1, 0, and 1) were simulated. As can be seen from this figure, randomCAT produced biased ability estimates for all three conditions. The problem of randomCAT comes from another function genPattern which resets the seed each time it is called. Resetting a seed typically leads to a loss of randomness, especially when the seed is reset to the same value as in the case of genPattern.

OPEN IN VIEWER

Figure 1.

A comparison between the randomCAT function in catR and the revised function.

The author revised genPattern, nextItem, and randomCAT. The simulation was repeated with the revised functions. The right panel in Figure 1 shows that the revised functions produced reasonable ability estimates. Appendix shows the R codes to replicate the results. The source file with the revised codes, catR-Cui.R, can be found in the online Supplement on the journal’s website. Although the genSeed argument was preserved in the randomCAT2 function for the purpose of comparison, a better way dealing with seeds is to avoid using seeds inside a function. Instead of setting seeds inside a function, a seed should be set through an external call to the set.seed function. In other words, with catR, a safer solution would be to simply fix the seed outside the functions and to completely avoid the seed usage inside the functions.

Although the potential risk of improper use of a seed is illustrated using the catR package, other packages might have the same issue. Package developers are advised to examine their packages to avoid the problem. Researchers should also be careful in selecting seeds in any simulation study that involves random numbers. Unfortunately, few research studies have reported seed usage. For example, Cui and Kolen (2008) did a simulation study to compare the parametric bootstrap method and the nonparametric bootstrap method. Their study involved multiple steps of drawing random sample sets. No seed usage, however, was described. Had the same seed been used for different random sample sets, they would correlate with each other, which would be likely to lead to problematic results.

In summary, one should be cautious in conducting simulation, especially when a seed is used. Given the results from this study, researchers and practitioners are encouraged to revisit their studies if they have used catR for simulation. Before a fix is implemented in catR, using the three aforementioned functions—explicitly or implicitly—may result in invalid results from such simulation. Researchers are advised to be careful in selecting seeds in any simulation study that involves a random number. A safe way to use a seed is to set the seed in the beginning and only once for the entire simulation.