R Package cacIRT

The R package cacIRT computes classification accuracy (CA) and consistency (CC) following the approach proposed by Lee (2010) or the approach proposed by Rudner (2005). While implementations of both approaches are available from the respective authors as stand-alone programs, cacIRT provides both in a unified framework within R. In addition, procedures based on Rudner’s approach are extended beyond the sample-based CA index by including a CC index and an option for distributional marginalization (called the D method, see Lee, 2010).

The two approaches differ in many ways but both are based on item response theory (IRT). The Lee approach (see documentation and examples by typing ?class.Lee in the R console) makes classifications on the total score scale but uses an IRT model to calculate the probabilities of each total score. A recursive algorithm, which returns the probabilities of each possible total score conditional on ability, is useful outside of CA and CC estimation and is directly accessible (see ?recursive.raw). The Rudner approach uses the IRT-based ability estimates to classify examinees (see ?class.Rud). The classification indices are based on conditional normal distributions centered at the ability estimate with standard deviation equal to the conditional standard error of measurement. Both the Lee and Rudner approaches perform well in simulation studies (Lathrop & Cheng, 2013).

For tests following the 1-, 2-, and 3-parameter dichotomous IRT models, the user supplies the cut scores (on the appropriate scale—the total score scale if using the Lee approach and the latent trait scale if using the Rudner approach), the item parameters, and one of following: the ability estimates, the response data matrix, or the population ability distribution (if the D method is desired). If the response data matrix is given, ability estimates and associated standard errors are calculated internally with functions from the R package irtoys (Partchev, 2012). For polytomous and mixed format tests, or if the user desires more control than the internal defaults, the Rudner approach can be supplied the ability estimates and the associated standard errors. To use the Lee approach in this situation, the user supplies an R object of response probabilities; see ?class.Lee for details and examples.

cacIRT is freely available on CRAN the Comprehensive R Archive Network at http://www.cran.r-project.org or can be installed through R with install.packages(“cacIRT”). All codes are executed within R and so will run on any system that can run R. All functions are fully documented with examples; a good starting point after installing and loading the package is to run example(class.Lee) and example(class.Rud) in the R console.