Advancing and Evaluating IRT Model Data Fit Indices in Organizational Research

Scale Construction and Evaluation

One of the most common uses of IRT is for scale construction and evaluation. Here, IRT is particularly useful for examining the quality of an existing measure or developing new measures that can assess organizational constructs more effectively. IRT has several unique advantages for scale construction and evaluation over alternative methods. First, IRT can be used to evaluate the quality of a scale regardless of the sample that is used to validate it. In IRT, the quality of items can be evaluated using the item parameters estimated from the model (see below for a more comprehensive discussion of relevant item parameters), which typically include the item difficulty (i.e., how much of the latent trait is needed to endorse an item or get it correct) and discrimination (i.e., how well the item discriminates between high and low levels of the latent trait). A key assumption of IRT models is that these parameters are invariant across different subpopulations of examinees, meaning that they can be readily compared after a linear transformation that puts them on a common scale. In other words, the item difficulty and discrimination will be the same regardless of the sample they are estimated in after applying a linear transformation. This is not the case with traditional methods based in classical test theory (CTT), which involve alternative estimates of difficulty (i.e., the proportion of people endorsing an item) and discrimination (e.g., item-total correlations). Therefore, a set of items in a measure can appear highly discriminating and include a range of difficulties (i.e., indicating a high quality assessment) in one sample but then appear more or less discriminating and difficult in another sample using CTT techniques.

Other advantages of IRT for scale construction and evaluation involve the types of assessments that can be created and constructs that can be assessed effectively. For example, given the assumption of subpopulation invariance described above, IRT can be used to create computer adaptive tests (CAT), which have been shown to be useful for employment testing (Drasgow et al., 2012; Stark et al., 2014) and for performance appraisals (Borman et al., 2001). With CAT, the items that an individual sees will depend on his or her answers to previous items in the assessment. In other words, the test is adaptive in the sense that test-takers are administered items with characteristics that match their level of the latent trait. Because each individual sees a different set of items, this type of assessment cannot be administered using traditional scale development methods. In addition to their advantages for individual assessment, CATs can also improve the efficiency of the assessment by allowing shorter tests, with some research indicating that adaptive tests can be approximately 50% shorter than static nonadaptive tests while maintaining the same quality of measurement (Stark, Chernyshenko, Drasgow, & White, 2012).

Finally, some types of constructs can be assessed more effectively using IRT than traditional scale development methods. Most measures in the organizational literature have been developed using what is known as a dominance response process. Dominance models assume that the probability of individuals endorsing an item increases monotonically with corresponding increases in their level of the latent trait and the distance between their standing on the trait and the location of the item. Dominance models are consistent with CTT and factor analytic approaches to scale development and are widely used in the organizational literature (Chernyshenko, Stark, Drasgow, & Roberts, 2007). Drasgow, Chernyshenko, and Stark (2010) argued that dominance models are most appropriate for measures that assess an individual’s maximal performance and ability in a particular domain. As such, cognitive ability provides a compelling example of a construct that fits within the dominance framework. Individuals are more likely to answer items on a cognitive ability test correctly if their ability is higher than the item’s location on the continuum of the latent trait.

Despite the popularity of dominance models, they do not necessarily provide the best description of the response process for all constructs. For measures that do not assess maximal performance and ability, the concept of dominance and the assumptions of this framework may not be appropriate. In these cases, another class of IRT models, known as ideal point models, may provide a better fit to the data. In some areas of research (e.g., political science), the term “ideal point” is used to refer to an individual’s standing on the latent trait in the dominance framework (Bafumi, Gelman, Park, & Kaplan, 2005). In the current study, we focus on ideal point models as a separate class of models that are distinct from dominance models and describe a different response process. Ideal point models assume that individuals will tend to endorse an item if the item’s location on the latent trait continuum matches their own standing on the latent trait. In other words, if an item is too extreme, or not extreme enough, to match their standing on the trait measured by the assessment, then individuals are less likely to endorse the item.

Drasgow et al. (2010) suggested that ideal point models are best suited for measures that require introspection and self-reflection. The concept behind ideal point models was first proposed by Thurstone (1928) in the context of attitude measurement, which requires individuals to reflect on their thoughts and feelings about a particular topic. ¹ Although Thurstone proposed this idea nearly 90 years ago, ideal point models have only recently gained traction in organizational research. Part of the reason for the growing interest in ideal point models is the relatively recent research demonstrating the advantages of ideal point models for a number of constructs. For example, ideal point models have been shown to be useful for understanding employee personality (Carter et al., 2014; Stark et al., 2006), vocational interests (Tay et al., 2009), person-organization fit (Chernyshenko, Stark, & Williams, 2009), performance ratings (Borman et al., 2001), and job attitudes (Carter & Dalal, 2010). In addition, as suggested by Drasgow et al. (2010), these methods will be applicable to any constructs that require employees to think and respond about themselves.

Although the distinction between dominance and ideal point models may seem trivial, past research has suggested that the choice between these models can have important effects on research results and applied decisions. For example, fitting a dominance model to ideal point data can affect the scale development process by influencing the types of items that are retained for a measure (Chernyshenko et al., 2007; Roberts, Laughlin, & Wedell, 1999). Applying an inappropriate model can also result in inaccurate estimates of the latent trait that will affect the rank order of individuals in a sample (Roberts et al., 1999) and correlations between variables (Carter et al., 2014). In addition, applying an incorrect model can also reduce the utility of employee selection decisions (Dalal & Carter, 2015). Therefore, using an inappropriate model for scale construction can have important implications for organizational research and the conclusions that are drawn from it.

Identify Item and Test Bias

Another advantage of IRT for organizational research is the ability to detect item/test bias. In the IRT literature, these methods are known as differential item functioning (DIF) and differential test functioning (DTF). Recent research has provided an extensive review of these techniques (e.g., Tay et al., 2015) and, therefore, we do not discuss the methodology again here. However, we do note several advantages of the IRT approach over alternative methods that are commonly used in the organizational literature.

First, it is important to note that observed mean-level differences do not necessarily reflect test bias and conclusions about these differences will be confounded with any bias in the measure when using CTT techniques. Stark, Chernyshenko, and Drasgow (2004) noted that observed differences between groups are equal to the sum of both bias and impact. Here, impact refers to true differences between groups. For example, if comparing job satisfaction across samples from different countries, any differences that are observed could be due to true differences in satisfaction across countries or due to bias in the measure. Without first testing for bias, the conclusions drawn from these comparisons may be misleading. Moreover, because of these relationships, significant mean differences do not necessarily indicate that there is bias in the measure. Nye and Drasgow (2011) showed that bias in a measure can inflate mean differences or even reverse the direction of these differences in organizational research.

Given the relationship between observed mean differences and bias, IRT provides a more appropriate way of identifying and testing biases in the measure. However, another alternative to IRT is to use confirmatory factor analysis (CFA). The CFA approach is widely used in the organizational literature (see, e.g., Vandenberg & Lance, 2000) and is more commonly known given that it can be estimated using widely available structural equations modeling (SEM) software. Despite this popularity, IRT also has several advantages over CFA tests of bias. For example, one advantage of IRT over the CFA approach is that methods of examining DTF are more widely developed. With DTF, bias at the item level can cancel out when aggregated to the level of the test. This occurs when different items are biased in opposite directions (i.e., one item is biased against one group and another item is biased against the other group). Therefore, DTF examines whether the test as a whole is biased against either group. This application is particularly useful for employee selection testing where researchers may be interested in whether a particular test will result in biased selection decisions if used in this context. Although the concept of DTF is well developed in IRT (see, e.g., Meade, 2010; Stark et al., 2004), similar tests are not available in CFA approaches to detecting bias.

Faking and Insufficient Effort Responding

A substantial amount of recent research has examined the issue of insufficient effort responding (IER; also known as careless responding) in the organizational literature (Huang, Curran, Keeney, Poposki, & DeShon, 2012; Huang, Liu, & Bowling, 2015; Meade & Craig, 2012). This research has identified a number of ways of detecting IER. However, the recommendations for identifying this type of responding suggest methods that may not always be available in organizational research. For example, one recommendation is to use response times to detect careless responding. However, this method will only be available for computerized testing and may not be available at all for archival data. Similarly, another promising approach is to use an index of consistent responding that examines the correlations between items that are theoretically negatively related (Huang et al., 2012). Nevertheless, this approach requires different scales that are negatively correlated to be assessed. There are other options available but the results have been mixed with some studies suggesting that these indices identify individual differences rather than random responding (Goldberg & Kilkowski, 1985) and other studies showing that the various indices generally do not agree on identifying insufficient effort or are insensitive to careless responding (Huang et al., 2012; Meade & Craig, 2012). Finally, these indices are only useful for detecting careless responding and may not be able to detect other types of response biases due to acquiescence, socially desirable responding or faking (i.e., resulting in spuriously high test scores), or spuriously low responding (e.g., due to faking bad, misreading test items, or random response errors such as mistyping an answer).

In contrast, IRT provides another method of detecting IER. A number of studies have been conducted on person-fit indices in the IRT literature. Although a number of indices have been proposed (see Meijer & Sijtsma, 2001, for a review), these indices generally use IRT models to estimate how individuals should be responding given their levels of the latent trait and then compare their predicted response patterns to their observed patterns of responses. Drasgow, Levine, and McLaughlin (1987) conducted a simulation study and found that some IRT person-fit indices could detect aberrant responding nearly 100% of the time, though the power to detect these effects did vary across conditions. Similarly, Meijer and Sijtsma (2001) conducted a qualitative review of IRT person-fit indices and found that detection rates were above 75% for some indices. More recently, Stark et al. (2017) conducted a simulation study that evaluated person-fit indices for specifically detecting IER and faking. Again, these authors found that IRT aberrance indicators could detect both IER and faking nearly 100% of the time in many conditions. In all of these studies, the ability to detect aberrant responding was substantially higher for the IRT indices than in research on traditional methods of detecting IER.

Step 1: Identify an Appropriate IRT Model

The first step to using IRT for organizational research is to identify an appropriate IRT model for the measures used in a study. To do so, researchers must first determine which model is conceptually most appropriate given the data that will be collected and the response format of the measure. As described above, there are two broad classes of IRT models that are widely used in the organizational literature. Probably the most widely used class of models are dominance models. Again, these models assume that the probability of individuals endorsing an item increases monotonically with corresponding increases in their level of the latent trait and the distance between their standing on the trait and the location of the item.

A number of dominance models are available for modeling the response process. For dichotomous responses, one of the most widely used models is the three-parameter logistic model (3PL). This model is described by the equation below:

P i ( θ ) = c i + ( 1 − c i ) 1 1 + e − D a i ( θ − b i )

where P_i (θ) is the probability of endorsing item i at a given level of the latent trait (θ). In addition, a_i is the discrimination parameter, b_i is the difficulty (or location) parameter, c_i is the guessing parameter, and D is the scaling constant 1.702. This model can be simplified by assuming that there is no guessing (i.e., c_i = 0), which results in the two-parameter logistic (2PL) model.

For polytomous items (i.e., items with more than two response options like Likert-type scales), another dominance model is Samejima’s (1969) graded response model (GRM). This model indicates the probability of endorsing a particular response option using the following equation:

P i k ( θ ) = 1 1 + e − a i ( θ − b i k − 1 ) − 1 1 + e − a i ( θ − b i k )

where P i k ( θ ) is the probability that an individual with a given level of the latent trait (θ) will endorse option k on item i, and b i k is the location (or difficulty) parameter for the corresponding option. ² For this model, there will be C – 1 b i k parameters, where C is the total number of response options.

In contrast to dominance models, ideal point models assume that individuals will tend to endorse an item if the item’s location on the latent trait continuum matches their own standing on the latent trait. In the organizational literature, the most widely used ideal point model is the generalized graded unfolding model (GGUM; Roberts, Donoghue, & Laughlin, 2000), which has been used in several organizational studies (e.g., Carter et al., 2014; Stark et al., 2006; Tay et al., 2009) and is potentially applicable to a number of other areas in organizational research (Drasgow et al., 2010). For dichotomous items (readers are referred to Roberts et al., 2000, for a description of GGUM for polytomous items), this model is defined as

P ( X i = 1 | θ ) = exp ( a i [ ( θ − δ i ) − τ i 1 ] ) + exp ( a i [ 2 ( θ − δ i ) − τ i 1 ] ) 1 + exp ( a i [ 3 ( θ − δ i ) ] ) + exp ( a i [ ( θ − δ i ) − τ i 1 ] ) + exp ( a i [ 2 ( θ − δ i ) − τ i 1 ] )

where a_i is the discrimination parameter for item i, δ i is the location of item i on the latent trait continuum, and τ i 1 is the location of the response threshold on the latent trait continuum.

Step 2: Collecting Data and Fitting IRT Models

Once an appropriate IRT model has been identified, the next step is to collect the data and/or estimate the IRT parameters based on the model being evaluated. In this stage of the process, there are a number of factors that need to be considered to obtain accurate estimates of IRT parameters. One issue that needs to be considered is the sample size. IRT estimation generally requires larger sample sizes than other approaches. However, there is no one answer for how large the sample size needs to be. Instead, the sample size requirements will depend on a number of factors, including the model that is being estimated. Nevertheless, some have suggested that at least 500 individuals are needed to get accurate estimates of the item parameters for the 2PL and GRM models (Embretson & Reise, 2000; Stark et al., 2006). These sample size requirements are often quite a bit larger than recommended sample sizes for other approaches such as CFA.

In addition to larger sample sizes, IRT models also generally make several assumptions about the measures that are being analyzed. One assumption is that the data have the appropriate dimensionality for the IRT model. All of the models described above assume the response data are unidimensional. However, multidimensional generalizations are available for most of these models (Forero & Maydeu-Olivares, 2009; Reckase, 2009). Therefore, it is important to first examine the dimensionality of a measure prior to estimating IRT parameters. This can be done using standard software packages for estimating a CFA model. However, research has also indicated that the data do no need to be strictly unidimensional (see Foster et al., 2017, for a review of this research). Instead, the data just need to be “unidimensional enough.” For example, Reckase (1979) showed that IRT estimates were generally accurate when the first factor accounted for at least 20% of the variance. Similarly, Drasgow and Lissak (1983) found that IRT estimates are still accurate as long as the latent factors are highly correlated (e.g., above .50).

All of the IRT models described above also assume local independence. Local independence means that all of the items being assessed are uncorrelated after accounting for the latent trait they assess. In other words, the only reason that the items are correlated is because of their relationship with the latent trait. In practice, this assumption suggests that there are no correlated errors among the items. Consequently, this assumption can also be tested using common SEM software to estimate a CFA model. ³

After ensuring a sufficient sample size for analyses and verifying that the data meet the assumptions of the IRT model, item parameters can be estimated using a number of different software packages. In the past, estimating IRT models required specialized software packages that were primarily used for IRT modeling and did not estimate other more commonly used statistical models. This was a major disadvantage of using IRT for organizational research because few people were familiar with these software packages or their use. However, this is no longer the case and IRT models can now be estimated in several widely used statistical packages, including Mplus (Muthén & Muthén, 1998-2017) and R, which are also used for other statistical analyses. Therefore, researchers who are familiar with this software can easily apply IRT models using widely used statistical packages. Example R code for estimating the IRT models examined in the present study is provided in the online supplemental material and at the following website: https://psychology.psy.msu.edu/pers_nye/IRT_syntax/.

Step 3: Evaluate IRT Model Fit

After identifying an appropriate IRT model and estimating IRT parameters, the next step is to evaluate the fit of the model to the data. Although evaluating model fit is ubiquitous in the SEM literature, this step is frequently ignored when applying IRT to organizational research. For example, in their review of the literature, Foster et al. (2017) noted that over 40% of articles using IRT in the organizational literature have not reported any information about fit. However, given the number of IRT models available, identifying the correct model for characterizing the response process in organizational research is particularly important (Chernyshenko, Stark, Chan, Drasgow, & Williams, 2001). In fact, past research has demonstrated that selecting an inappropriate IRT model can have serious consequences for scale construction (Chernyshenko et al., 2007; Roberts et al., 1999), test score equating (e.g., Bolt, 1999; Kaskowitz & De Ayala, 2001), parameter estimation (e.g., DeMars, 2005), adaptive testing (e.g., De Ayala, Dodd, & Koch, 1992), and differential item functioning (e.g., Bolt, 2002). As noted above, applying an inappropriate IRT model can also result in inaccurate estimates of the latent trait that will affect the rank order of individuals in a sample (Roberts et al., 1999) and correlations with external variables (i.e., validity; Carter et al., 2014). Therefore, testing for IRT model fit is critical because the benefits of IRT described above cannot be realized if the model does not fit the data well.

A number of fit indices have been developed for testing IRT model fit. One way to operationalize model fit is to examine the extent to which the observed proportions of correct answers on a measure match the model predicted proportions using a chi-square fit statistic. One such index, known as Yen’s Q₁ (Y-Q₁ ; Yen, 1981), is defined as

Y - Q 1 i = ∑ r = 1 10 N r ( O i r − E i r ) 2 E i r ( 1 − E i r ) .

Building on Bock’s (1972) chi-square index, Y-Q₁ is calculated by rank ordering examinees using their trait estimates (i.e., θ ^ ) and then dividing them into 10 score groups of approximately equal size. N_r is the number of examinees in cell r, O_ir is the observed proportion of examinees in cell r that correctly answers item i, and E_{i r} is the proportion of examinees in cell r that are predicted to answer item i correctly. As such, Y-Q₁ is essentially the sum of standardized residuals and follows a χ² distribution with 10 − g degrees of freedom, where g is the number of item parameters estimated.

Although Yen (1984) found that Y-Q₁ had some difficulty identifying misfit when the 2PL model was fit to 3PL data, her study also indicated that Y-Q₁ is sensitive to other forms of model misfit. However, in the context of the present study, Y-Q₁ has two notable limitations. First, some have questioned the appropriateness of dividing scores into an arbitrary number of groups based on trait estimates. The problem is that this approach can provide inaccurate estimates of the differences between the observed and expected proportions. The reason for these inaccuracies is that allocating individuals to score groups of equal size will depend on the sample being used for estimation because the cut points and number of intervals that are used will vary depending on the distribution of the latent trait. This can result in high Type I error rates under some conditions (Orlando & Thissen, 2000). A second limitation of this index is that it can result in inaccurate estimates of fit for some types of models. As Roberts et al. (2000) pointed out, the ideal point response process can result in small expected frequencies for some response categories. Although small expected frequencies can be a problem for any IRT model, this issue is particularly salient for ideal point models because these models assume that individuals tend to endorse items that are close to their level of the latent trait. As a result, the pattern of these small expected frequencies makes it difficult to correct this issue by combining response categories, which is a common way to address this problem for other IRT models. Given these limitations, more research is needed to evaluate Y-Q₁ as an index of IRT model fit.

Drasgow, Levine, Tsien, Williams, and Mead (1995) suggested another family of χ² statistics that does not require examinees to be allocated to separate score groups. Instead, observed and expected frequencies are summed over the number of response options. Using this formulation, the χ² for item singles and doubles can be calculated. For m items, m and ( m 2 ) χ² statistics can be computed for item singles and doubles, respectively. As described by Drasgow et al., the expectations for response option k, where k = 0,1, on item i and item double, i, j, are given by

E i ( k ) = N ∫ P ( u i = k | θ ) f ( θ ) d θ

and

E i , j ( k , k ) = N ∫ P ( u i = k | θ ) P ( u j = k | θ ) f ( θ ) d θ ,

respectively. Here, f is the ability density, which is usually taken as normal (0,1), and u denotes a scored response. The χ² is then computed with the usual formula

χ i 2 = ∑ k = 0 1 [ O i ( k ) − E i ( k ) ] 2 E i ( k ) ,

and the χ² for doubles is computed using the formula for a 2 x 2 contingency table. This family of chi-squares can be extended to triples of items and beyond. However, the expected and observed frequencies of many cells quickly diminish as the size of the contingency table increases and the power of the chi-square would be expected to decline as a result. The chi-squares for item singles, doubles, and triples are typically evaluated by examining the ratio of the chi-square to the degrees of freedom (df; Drasgow et al., 1995). As a useful heuristic, values of the χ²/df ratio less than 3 are considered indicative of good model-data fit (Chernyshenko et al., 2001).

The chi-squares for singles, doubles, and triples have been used to assess the fit of both dominance and ideal point models in previous research (Chernyshenko et al., 2001; Drasgow et al., 1995; Stark et al., 2006) and some models have been found to fit and others misfit. In addition, two recent simulation studies have examined the accuracy of these chi-square indices. In one study, Tay, Ali, Drasgow, and Williams (2011) examined the χ²/df ratios for item doubles and triples and compared the efficacy of these indices for detecting misfit. Their results indicated that the χ²/df ratios were generally effective at identifying the correct IRT model but, in some cases, had trouble identifying misfit when the GGUM was fit to data generated from the 2PL model. In another study, Tay and Drasgow (2012) examined the χ²/df ratios for detecting misfit in dominance models across a range of conditions and found that this index had difficulty distinguishing between some dominance models (e.g., 2PL and 3PL).

One limitation of chi-square fit indices in general is that they are severely influenced by sample size. To address this issue for the chi-square indices proposed by Drasgow et al. (1995), some have suggested examining chi-square values that have been adjusted to a smaller sample size when N is large (e.g., greater than 3,000; Chernyshenko et al., 2001; Tay et al., 2011). Although these adjustments can help to address problems with the increasing chi-square values that result from large sample sizes, some research has indicated that these adjusted χ²/df ratios may still result in high Type I error rates for smaller sample sizes (Tay & Drasgow, 2012).

Problems with chi-square fit indices have been encountered in the structural equation modeling (SEM) literature as well. To address these issues in SEM, alternative fit indices have been proposed and are commonly used to evaluate model-data fit. However, until recently, similar fit indices were not available for IRT models. Recent work has focused on addressing this issue and developing new IRT fit indices that can supplement the chi-square fit statistics and address their potential limitations. For example, Maydeu-Olivares and Joe (2014) focused on developing indices of approximate fit and developed a standardized root mean square residual (SRMSR). ⁴ Based on a set of simulations, these authors suggested that an SRMSR ≤ .05 indicates that the model is an acceptable approximation to the data.

Another limitation of chi-square fit indices is that the number of potential response patterns increases substantially as the number of items in a measure increases. For dichotomous items, there will be 2 ^m potential response patterns, where m is the number of items in the measure. As a result, it is nearly impossible to calculate a chi-square to compare observed to empirical probabilities for a full contingency table with 2 ^m cells. To address this issue, most chi-square fit indices examine lower-order approximations (e.g., for single items, doubles, or triples) of the full contingency table. However, another alternative is to use a Bayesian approach to model-checking that does not require examining such lower-order approximations. For example, an approach called the posterior predictive model check (PPMC; Rubin, 1984) has shown promise in recent research (Li, Bolt, & Fu, 2006; Sinharay, Johnson, & Stern, 2006; Zhu & Stone, 2012). This approach examines model-data fit by comparing observed data to replicated data that are generated from the model being evaluated. The replicated data are then used to evaluate the probability of the observed data by calculating posterior predictive p- (PPP) values to determine the extremity of the observed data given the posterior distribution. In other words, the question is whether the observed data look like the replicated data generated from the model. The fit of multiple models can be evaluated by comparing the number of items (or item doubles) with PPP values near .50, which indicates that there is no systematic difference between the observed and replicated data (Sinharay et al., 2006). A model that results in more items with PPP values near .50 compared to other models would be considered a good fitting model.

Another approach to examining model data fit is to examine Akaike’s information criterion (AIC; Akaike, 1973) and the Bayesian information criterion (BIC; Schwarz, 1978). Both the AIC and BIC have been widely used as model fit indices and provide information about model fit relative to the number of parameters in the model. Although the AIC and BIC have shown some promise for correctly identifying the fit of the 1PL and 2PL IRT models (Kang, Cohen, & Sung, 2009), these indices have some limitations for some types of IRT models. For example, these indices should not be used with MCMC estimation because this method uses a prior distribution which creates a dependence between the parameters that makes estimating the number of parameters difficult. Because the calculation of AIC and BIC depend on the number of parameters in the model, these indices may be inaccurate with MCMC estimation (Spiegelhalter, Best, Carlin, & Van Der Linde, 2002). This is particularly important given that recent research has demonstrated that MCMC estimation can provide more accurate parameter estimates for some ideal point models (de la Torre, Stark, & Chernyshenko, 2006; Wang, 2014).

Generating Parameters

The item parameters used to generate the data from specific IRT models were randomly generated from distributions of parameters obtained from previous simulation and substantive IRT studies (e.g., Roberts et al., 2000; Stark et al., 2006). For dichotomous conditions, the discrimination parameters used to generate the data for the 2PL, 3PL, GGUM, and M2PL models were obtained from a random uniform distribution [0.5, 2]. The location parameters used to generate the data for these models were obtained from a random uniform distribution [–2, 2]. Also, the generating threshold parameter ( τ ) for the GGUM and the guessing parameter, c, for the 3PL model were obtained from random uniform distributions [–1.4, 0.4] and [0, 0.3], respectively. For polytomous conditions, data were simulated with four response categories and the distribution for generating the discrimination parameter was the same as in the dichotomous conditions. In addition, the threshold parameters (b ₁, b ₂, b ₃) for generating data from the GRM and MGRM models were obtained from random uniform distributions [–2, –0.5], [–0.5, 0.5], and [0.5, 2], respectively. The ranges of these uniform distributions were chosen based on past simulation work with the GRM (e.g., Kieftenbeld & Natesan, 2012). For the GGUM, the highest threshold parameter ( τ 3 ) was randomly sampled from a uniform distribution [–1.5, –0.5]; then the thresholds for τ 2 and τ 1 were generated using a recursive formula, wherein a constant of –.25 and a random error sampled from a N(0, 0.04) distribution were added to τ 3 and τ 2 , respectively. This recursive method mirrors the threshold parameters that have been used in previous GGUM simulation studies (e.g., Roberts et al., 2000). The person parameters ( θ ) used to generate the data were randomly selected from a N(0, 1) distribution for the unidimensional models. For the multidimensional models, the person parameters were generated from a multivariate normal distribution with 0 mean vector, variances of 1, and a covariance of 0.5.

Response Data Generation

Using the item parameters described above, response data were generated by comparing randomly sampled numbers from a uniform (0, 1) distribution to the response probabilities for the respective items. For dichotomous conditions, response data were scored 1 if the response probability was greater than the random uniform number and 0 otherwise. For the polytomous conditions, the response data were scored 3 if a randomly sampled uniform number was greater than the sum of the probabilities for categories 0, 1, and 2. If the random number was less than the sum of the probabilities for categories 0, 1, and 2 but greater than the sum for categories 0 and 1, then the response was scored 2, and so forth.