Using the Graded Response Model to Control Spurious Interactions in Moderated Multiple Regression

Abstract

Recent simulation research has demonstrated that using simple raw score to operationalize a latent construct can result in inflated Type I error rates for the interaction term of a moderated statistical model when the interaction (or lack thereof) is proposed at the latent variable level. Rescaling the scores using an appropriate item response theory (IRT) model can mitigate this effect under similar conditions. However, this work has thus far been limited to dichotomous data. The purpose of this study was to extend this investigation to multicategory (polytomous) data using the graded response model (GRM). Consistent with previous studies, inflated Type I error rates were observed under some conditions when polytomous number-correct scores were used, and were mitigated when the data were rescaled with the GRM. These results support the proposition that IRT-derived scores are more robust to spurious interaction effects in moderated statistical models than simple raw scores under certain conditions.

Keywords

graded response model item response theory polytomous models simulation

Operationalizing a latent construct such as an attitude or ability is a common practice in psychological research. Stine (1989) described this process as the creation of a mathematical structure (scores) that represents the empirical structure (construct) of interest. Typically, researchers will use simple raw scores (e.g., either as a sum or a mean) from a scale or test as the mathematical structure for a latent construct. However, much debate regarding the properties of such scores has ensued since S. S. Stevens’s classic publication of the nominal, ordinal, interval, and ratio scales of measurement (Stevens, 1946). Although it is beyond the scope of this article to enter the scale of measurement foray, an often agreed-on position is that simple raw scores for latent constructs do not exceed an ordinal scale of measurement. This scale imbues such scores with limited mathematical properties and permissible transformations that are necessary for the appropriate application of parametric statistical models. Nonparametric, or distribution-free, statistics have been proposed as a solution for the scale of measurement problem. However, many researchers are reluctant to use nonparametric techniques because they are often associated with a loss of information pertaining to the nature of the variables (Gardner, 1975). McNemar (1969) articulated this point by saying, “Consequently, in using a non-parametric method as a short-cut, we are throwing away dollars in order to save pennies” (p. 432).

Assuming that simple raw scores are limited to the ordinal scale of measurement and researchers typically prefer parametric models to their nonparametric analogues, the empirical question regarding the robustness of various parametric statistical models to scale violations arises. Davison and Sharma (1988) and Maxwell and Delaney (1985) demonstrated through mathematical derivations that there is little cause for concern when comparing mean group differences in the independent samples t test when the assumptions of normality and homogeneity of variance are met. However, Davison and Sharma (1990) subsequently demonstrated that scaling-induced spurious interaction effects could occur with ordinal-level observed scores in multiple regression analyses. These findings suggest that scaling may become a problem when a multiplicative interaction term is introduced into a parametric statistical model.

Scaling and Item Response Theory (IRT)

An alternative solution to the scale of measurement issue for parametric statistics is to rescale the raw data itself into an interval-level metric, and a variety of methods for this rescaling have been proposed (see Embretson, 2006; Granberg-Rademacker, 2010; Harwell & Gatti, 2001). A potential method for producing scores with near interval-level scaling properties is the application of IRT models to operationalize number-correct scores into estimated theta scores—the IRT-derived estimate of an individual’s ability or latent construct standing. Conceptually, the attractiveness of this method rests with the invariance property in IRT scaling, and such scores may provide a more appropriate metric for use in parametric statistical analyses.¹ Reise, Ainsworth, and Haviland (2005) stated that

Trait-level estimates in IRT are superior to raw total scores because (a) they are optimal scalings of individual differences (i.e., no scaling can be more precise or reliable) and (b) latent-trait scales have relatively better (i.e., closer to interval) scaling properties. (p. 98, italics in original)

In addition, Reise and Haviland (2005) gave an elegant treatment of this condition by demonstrating that the log-odds of endorsing an item and the theta scale form a linearly increasing relationship. Specifically, the rate of change on the theta scale is preserved (for all levels of theta) in relation to the log-odds of item endorsement.

Empirical Evidence of IRT Scaling

In a simulation testing the effect of scaling and test difficulty on interaction effects in factorial analysis of variance (ANOVA), Embretson (1996) demonstrated that Type I and Type II errors for the interaction term could be exacerbated when simple raw scores are used under nonoptimal psychometric conditions. Such errors occurred primarily due to the ordinal-level scaling limitations of simple raw scores, and the ceiling and floor effects imposed when an assessment is either too easy or too difficult for a group of individuals—a condition known as assessment inappropriateness (see Figure 1). Embretson fitted the one-parameter logistic (Rasch) model to the data and was able to mitigate the null hypothesis errors using the estimated theta scores rather than the simple raw scores. These results illuminated the usefulness of IRT scaling for dependent variables in factorial models, especially under suboptimal psychometric conditions. Embretson argued that researchers are often unaware when these conditions are present and can benefit from using appropriately fitted IRT models to generate scores that are more appropriate for use with parametric analyses.

Figure 1.

A representation of the latent construct distribution and test information (reliability) distributions for appropriate assessments (top) and inappropriate assessments (bottom)

An important question that now arises is whether these characteristics extend to more complex IRT models such as the two- and three-parameter logistic models (dichotomous models with a discrimination and guessing parameter, respectively) and polytomous models. Although the Rasch model demonstrates desirable measurement characteristics (i.e., true parameter invariance; Embretson & Reise, 2000; Fischer, 1995; Perline, Wright, & Wainer, 1979), it is sometimes too restrictive to use in practical contexts. However, the consensus regarding the likelihood that non-Rasch models could achieve interval-level scaling properties is “yes” (Embretson & Reise, 2000; Hambleton, Swaminathan, & Rogers, 1991; Harwell & Gatti, 2001; Reise et al., 2005). Investigations into the scaling properties of these more complex IRT models are thus necessary.

In one extension of this sort, Kang and Waller (2005) simulated the scaling properties of simple raw scores, estimated theta scores derived from a two-parameter logistic IRT model, and assessment appropriateness with the interaction term in a moderated multiple regression (MMR) analysis. Similar to the findings of Embretson (1996), Kang and Waller discovered that using simple raw scores to operationalize a latent construct resulted in substantial inflations of the Type I error rate (>50% or p > .50) for the interaction term in MMR under conditions of assessment inappropriateness. However, the IRT-derived theta score estimates were found to mitigate the Type I error rate to acceptable levels (<10% or p < .10) under the same conditions. This extension demonstrated that the estimated theta scores from a non-Rasch IRT model could be used to better fit the assumptions of parametric statistical models involving an interaction term. Finally, Harwell and Gatti (2001) investigated the congruence of estimated (theta) and actual construct scores using a popular polytomous IRT model, the graded response model (GRM; Samejima, 1969, 1996). The authors posited that if the estimated construct (theta) scores were sufficiently similar to the actual construct (theta) scores, which can be defined (albeit arbitrarily) as a metric with interval-level scaling properties, then the GRM results in scores that are sufficiently interval level. The results of their study supported this relationship. However, a concrete endorsement of the scaling properties should be made with caution due to the inherently arbitrary metric of the theta scale in most (if not all) IRT models.

The preceding theoretical and simulation evidence suggests that scale of measurement violations accompanied with suboptimal psychometric conditions (i.e., assessment inappropriateness) may have nonnegligible effects on the accuracy of common parametric analyses. However, this evidence still has limited generalizability to the majority of psychological research due to the nature of the simulated data. Specifically, Embretson (1996) and Kang and Waller (2005) simulated dichotomous data and fit logistic IRT models appropriate for such data. However, the majority of psychological research uses polytomous data, or data with multiple response options, such as Likert-type scales. Therefore, the purpose of the current study is to extend the understanding of scaling and assessment appropriateness on parametric statistical analyses by simulating polytomous data and fitting an appropriate polytomous IRT model. The authors’ primary null hypothesis against which the performance of number-correct scores and estimated theta scores are being compared is that there is no significant interaction on the actual theta scale. Thus, any significant interaction identified in the simulation results represents a spurious observed effect (Type I error).

Method

This study used a Monte Carlo simulation to identify the psychometric conditions that lead to an elevated risk of Type I errors for interaction effects in MMR when the theta scale is considered to be the true metric. This simulation was similar to the simulation conducted by Kang and Waller (2005) and extends this work into polytomous scales indicative of those commonly used in applied psychological research.

The GRM

The GRM (Samejima, 1969, 1996) is an IRT model suitable for modeling data with ordered categories such as Likert-type scales, and is an extension of the two-parameter logistic model. The GRM is considered a difference family model and was developed specifically to model polytomous data that represent the psychological processes underlying multicategory decision making (Ostini & Nering, 2006). In addition, theta estimates derived using the GRM may show evidence of interval-level scaling properties (Harwell & Gatti, 2001).

Using the GRM, an individual’s likelihood of responding in a particular response category is derived using a two-step process. First, category boundary functions (CBRFs) are calculated to determine boundary decision probabilities of j− 1 response categories for each item. The CBRFs in the GRM can be derived with Equation 1 (adapted from Embretson & Reise, 2000).

P_{ix} * (θ) = \frac{e^{[a_{i} (θ - b_{ij})]}}{1 + e^{[a_{i} (θ - b_{ij})]}},

(1)

In Equation 1, P_ix∗ (θ) is the probability that an individual with a trait (construct) level θ will respond positively at the boundary of category j for item i where x = j = 1 . . . m_i. Theta (θ) represents the individual’s trait (construct) level, a_i represents the item discrimination or slope, and b_ij represents the category location or difficulty parameter with respect to the trait continuum. Importantly, the values of b_ij should be successive integers reflecting increased difficulty in progressing through the response options in well-functioning items.

In the second step of the GRM, the probability of responding in a particular category is determined using category response functions (CRFs), which are derived by subtracting P_ix∗ (θ) from the following category. This process is illustrated in Equation 2 (adapted from Embretson & Reise, 2000).

P_{ix} (θ) = P_{ix} * (θ) - P_{i (x + 1)} * (θ),

(2)

Determining the first category is done by simply subtracting P_i1∗ (θ) from 1.0, and the last category is simply P_im∗ (θ). The GRM was used as a model for the number-correct score algorithm as well as to estimate theta scores in this study.

Independent Variables

The independent variables in this study were respondent sample size (n: two levels), scale length (k: two levels), item discrimination (a_i: two levels), item difficulty (b_i,_{1 . . . j}₋₁: three levels), scale bandwidth (fidelity: two levels), and the regression coefficients (β₁ and β₂: two levels). The structure of this study was therefore a 2 × 2 × 2 × 3 × 2 × 2 design comprising 96 conditions.

Sample size (n)

Two respondent sample sizes were simulated according to recent evidence of the stability of parameter estimates in polytomous IRT, and actual sample sizes in MMR studies in applied psychology. Ostini and Nering (2006) reported that stable estimates for polytomous IRT models could be obtained with as few as 250 individuals, but that samples between 500 and 1,000 are still considered to be desirable. In addition, Aguinis Beaty, Boik, and Pierce (2005) indicated that the average sample size for MMR studies in applied psychological research is ${\bar{x}}_{n} = 272$ with an average standard deviation of $s_{n} = 434$ . These results indicate that the simulation outcomes for the n = 250 sample size will be the most relevant for the majority of applied psychological research; however, some studies do achieve sample sizes upward of n = 1,000 (see, for example, Witt, 1998; n = 979). Therefore, sample sizes included two levels of n = 250 and n = 750 respondents to maximize the generalizability for the majority of empirical MMR studies in applied psychology as well as for typical IRT studies.

Scale length (k)

Two scale lengths of k = 15 and k = 30 items were simulated in this study to model typical scales used in applied psychological research. In a review of validated scales used in applied psychological research, Fields (2002) indicated a modal scale length of 15 items with a mean of 15.43 and a standard deviation of 10.43 for validated scales in applied psychology. The distribution related to these values is also slightly positively skewed, indicating the existence of several very long scales.

Discrimination (a_i)

To derive the highest level of generalizability from this study, item parameter values were randomly selected from specified distributions as opposed to using constant values. Following the structure of Kang and Waller (2005), item discrimination values were selected from a uniform distribution between the values of 0.31 to 0.58 for moderate discrimination and 0.58 to 1.13 for high discrimination. Estimating discrimination values from a uniform distribution has been demonstrated to appropriately represent empirically determined item discrimination values (Reise & Waller, 2003), and the particular cutoff values of 0.31, 0.58, and 1.13 were demonstrated to appropriately represent low, moderate, and high factor loadings for items (Kang & Waller, 2005; Takane & De Leeuw, 1987). Because the GRM is a polytomous extension of the two-parameter logistic model, these values can be deemed appropriate for use in this study. Furthermore, the decision to retain the values from the Kang and Waller (2005) study was made to maintain a basis of comparison for the extension to polytomous data.

Item difficulty (b_i,_{1 . . . j−1})/assessment appropriateness

Three item difficulty conditions were simulated to represent a “difficult,”“moderate,” and “easy” scale with respect to the simulated distribution of construct scores (see Figure 1). The item difficulty conditions are also analogous to the assessment appropriateness conditions with the “difficult” and “easy” conditions representing assessment inappropriateness and the “moderate” condition representing assessment appropriateness. Item difficulty values were randomly selected from a N(−1.5, 1.0) distribution for the easy (inappropriate) conditions, a N(0.0, 1.0) distribution for the moderate (appropriate) conditions, and a N(1.5, 1.0) distribution for the difficult (inappropriate) conditions.

Four item difficulty parameters were randomly selected from the appropriate distribution for each item to represent the j– 1 CBRFs specified in Equation 1 for the GRM. An important aspect of the difficulty parameters in polytomous IRT models is that the difficulty parameter for each CBRF is sequentially ordered. Therefore, the difficulty parameters were modeled with the sequential ordering restriction imposed similar to that implemented by Meade, Lautenschlager, and Johnson (2007).

Scale fidelity

An assessment’s fidelity is measured as the inverse of variability (i.e., bandwidth) in the difficulty of the items (Stocking, 1987). Fidelity contributes to assessment appropriateness by either restricting (high fidelity) or expanding (low fidelity) the width of the item difficulty distribution. The high-fidelity conditions were simulated in this study by generating a second set of item difficulty values from more restricted normal distributions with a mean and standard deviation of N(−1.50, 0.50) for easy scales, N(0.00, 0.50) for moderate scales, and N(1.50, 0.50) for difficult scales. These restricted distributions will create the high-fidelity and low-bandwidth situation in which Kang and Waller (2005) observed the highest prevalence of Type I errors. As in the previous difficulty parameter selection, there were four (j− 1) difficulty values for each item sampled from within the specified distribution with the sequential ordering restriction imposed.

Regression weights

In accordance with Kang and Waller (2005), regression weights were set at a value of 0.30 or 0.50 for both β₁ and β₂. An intercept of 0 is used and therefore omitted from the regression models. It should be noted that these regression weights are fixed only for the purposes of simulating the dependent variables.

Fixed Effects

Item response categories (j)

Five item response categories were used to simulate a five-category Likert-type response scale. Fields (2002) identified 134 validated construct assessments that are used in applied psychological research of which five-category Likert-type response scales were the most common (n = 57).

Regression models

The purpose of this study was to observe the prevalence of Type I errors in MMR in three different pairs of models. In the first regression model pair, actual latent trait scores θ will be analyzed (see Equations 3a and 3b). In the second regression model pair, number-correct scores (X) will be analyzed (see Equations 4a and 4b). In the third regression model pair, estimated theta scores $\hat{θ}$ will be analyzed (see Equations 5a and 5b). These three model pairs were expressed in accordance with Kang and Waller (2005) as follows:

θ_{3} = β_{1} θ_{1} + β_{2} θ_{2} + ε,

(3a)

θ_{3} = β_{1} θ_{1} + β_{2} θ_{2} + β_{3} θ_{1} θ_{2} + ε,

(3b)

X_{3} = β_{1} X_{1} + β_{2} X_{2} + ε,

(4a)

X_{3} = β_{1} X_{1} + β_{2} X_{2} + β_{3} X_{1} X_{2} + ε,

(4b)

{\hat{θ}}_{3} = β_{1} {\hat{θ}}_{1} + β_{2} {\hat{θ}}_{2} + ε,

(5a)

{\hat{θ}}_{3} = β_{1} {\hat{θ}}_{1} + β_{2} {\hat{θ}}_{2} + β_{3} {\hat{θ}}_{1} {\hat{θ}}_{2} + ε .

(5b)

The first model of each pair is the additive model, and the second model in each pair contains a multiplicative (interaction) term. Each model pair was structured as a hierarchical regression analysis where the interaction term is entered at the second step (Aiken & West, 1991; Cohen, Cohen, West, & Aiken, 2003). A significant change in variance accounted for (ΔR²) between the first and second model indicated the existence of a spurious interaction effect based on the null hypothesis that the data were created with no significant interaction on the actual theta scale.

Regression Main Effects

Two continuous predictor variables were simulated for each regression model specified in Equations 3 through 5b. Predictor variables θ₁ and θ₂ were randomly selected for the number of observations (n) from normal distributions with a mean and standard deviation equal to N(0.00, 1.00). These variables served as the main effect scores in the regression models. It is important to note that θ₁ and θ₂ were sampled from identical but independent distributions; thus, no multicollinearity was modeled.

Regression Criterion Variables

One continuous criterion variable was calculated for each regression model specified in Equations 3 through 5b. In accordance with Kang and Waller (2005), the general form of the criterion variables is given by the following equation, which represents a multiple regression model with two significant main effects and no interaction on the actual theta scale.

θ_{3} = β_{1} θ_{1} + β_{2} θ_{2} + \sqrt{1 - (β_{1}^{2} + β_{2}^{2})} \times ε,

(6)

In Equation 6, β₁ and β₂ are the simulated regression weights and ε is an error term. Note that the intercept term, β₀, was set to equal 0 and thus omitted from the model. The term $\sqrt{1 - (β_{1}^{2} + β_{2}^{2})}$ was included to represent an appropriate error variance component for each level of β. See the Appendix for a derivation of the error term.

Number-correct scores

To generate the number-correct scores, X₁, X₂, and X₃, the values of the previously defined construct scores θ₁, θ₂, and θ₃ were entered into the GRM algorithm (Equations 1 and 2) for each simulated participant.

A matrix of response scores was generated by reporting the number-correct score (1, 2, 3, 4, or 5) corresponding to the highest-category response likelihood for each simulated participant on each item. These values were derived using an algorithm written by the first author in the R language based on response probabilities calculated in Equations 1 and 2. Actual number-correct score responses were generated by comparing a randomly selected value from a uniform distribution, U(0.0, 1.0), with the relative response probabilities that are generated for each level of theta (or individual) and each item. This process can be thought of as determining the relative likelihood of a category response given the item and person parameters with a realistic level of decision-making error (Kang & Waller, 2005; Stone, 1992). This integration of response error is important so as to not assume perfect responding by simulated individuals. A mean score for X₁, X₂, and X₃ for each simulated individual was calculated from the number-correct score response matrices for analysis in the regression models.

Estimated Theta Scores

Finally, theta scores ${\hat{θ}}_{1}$ , ${\hat{θ}}_{2}$ , and ${\hat{θ}}_{3}$ were estimated from the simulated raw data, using PARSCALE 4.1 (Muraki & Bock, 2003). PARSCALE was set to derive the person (latent construct scores) and item parameters using the expected a posteriori (EAP) method. This method calculates ${\hat{θ}}_{1}$ , ${\hat{θ}}_{2}$ , and ${\hat{θ}}_{3}$ as the modal value of the posterior distribution, which is the most likely value of theta for the observed response pattern (Baker & Kim, 2004) and is a preferred estimation method for assessments that are moderate to short in item length (Mislevy & Stocking, 1989).

Iterations

For the purposes of estimating Type I error rates in Monte Carlo studies, Robey and Barcikowski (1992) specify that approximately 1,000 iterations will achieve a power equal to .90 when approximating an alpha level of α = .05 and using the interval of $α \pm \frac{1}{2} α$ as a robustness interval. Therefore, 1,000 iterations per condition were conducted. This allowed for adequate reduction in sampling variance for the IRT parameter estimates (Harwell, Stone, Hsu, & Kirisci, 1996), achieves a power of .90 around the interval $. 025 \leq α \leq . 075$ (Robey & Barcikowski, 1992), and doubles the number of iterations used by Kang and Waller (2005).

Simulation-Dependent Variables

Type I errors

The primary dependent variable for this study was the empirical Type I error rate (π) that is observed for the interaction term of the MMR models. The specific value of π was identified in a three-step process. In each iteration of the simulation, the variance in θ₃ accounted for by θ₁ and θ₂ was recorded as the R² value for the additive and multiplicative regression models specified in Equations 3 through 5b. Second, the significance of the change in variance accounted for, ΔR², between the respective additive and multiplicative models was tested at an alpha level of $p \leq . 05$ , and recorded as 1 for a significant result and 0 for a nonsignificant result. Finally, the empirical alpha level π was recorded as the proportion $(\frac{x}{1, 000})$ of iterations resulting in a significant ΔR² for the actual latent trait scores θ₃, the number-correct scores X₃, and the estimated theta scores ${\hat{θ}}_{3}$ .

Procedure

The simulation for the current study was conducted in the R environment (version 2.9.0; Ihaka & Gentleman, 1996; R Development Core Team, 2008) using a series of functions written by the authors, contributed code from Kang and Waller (2005), and PARSCALE 4.1 (Muraki & Bock, 2003). For ease of interpretation, four separate simulations were conducted. The four simulations were separated based on sample size (n = 250, 750) and scale fidelity (normal, high). In each simulation, the independent variables of scale length, regression weights, discrimination, and difficulty will be systematically varied. Therefore, the summary statistics for each simulation are included in four tables, each with 24 rows.

Each simulation was run using the following process. First, using the pseudorandom number generator in R, theta vectors were sampled from a standard normal distribution N(0.0, 1.0) for θ₁ and θ₂. Next, corresponding vectors for θ₃ were calculated using Equation 6. These vectors were saved as the actual latent construct scores. To calculate the number-correct score matrices, X₁, X₂, and X₃, each of these three score vectors were evaluated in an algorithm written by the first author that implements Equation 1 and Equation 2 to determine the probability of a category response. Final number-correct score values were determined by the comparison of a randomly selected value from a uniform distribution as previously described. Finally, the estimated theta scores ${\hat{θ}}_{1}$ , ${\hat{θ}}_{2}$ , and ${\hat{θ}}_{3}$ were derived using PARSCALE 4.1 (Muraki & Bock, 2003). To accomplish this task, the number-correct score matrices were “batched” out to PARSCALE with an accompanying syntax file following the structure identified by Gagne, Furlow, and Ross (2009). The estimated theta scores from PARSCALE were then returned to R as the vectors ${\hat{θ}}_{1}$ , ${\hat{θ}}_{2}$ , and ${\hat{θ}}_{3}$ .

Finally, the nine score vectors to be entered into the corresponding additive and multiplicative regression models specified in Equations 3 through 5b and the change in variance accounted for between the two corresponding models was recorded. The final summary statistics and tables were generated using portions of code provided by Niels Waller and used in the Kang and Waller (2005) study.

Results

Using the $α \pm \frac{1}{2} α$ criterion, the results indicated meaningfully inflated Type I error rates in 53 of 96 conditions (55%) when number-correct scores were used to operationalize the latent constructs, and in 33 of 96 conditions (34%) when estimated theta scores were used to operationalize the latent constructs (see conditions marked with an asterisk under the columns labeled π _X and $π_{\hat{θ}}$ , respectively, in Tables 1-4). In addition, a binomial test was conducted as a measure of statistically significant departures from the nominal (α = .05) alpha level for each scoring method. The results of the binomial test were slightly more conservative and indicated significantly inflated Type I error rates in 63 of 96 conditions (66%) when number-correct scores were used to operationalize the latent constructs and in 44 of 96 conditions (46%) when estimated theta scores were used to operationalize the latent constructs (see conditions marked with dagger under the columns labeled π _X and $π_{\hat{θ}}$ , respectively, in Tables 1-4).

Table 1.

Results of Simulation 1 (Normal Fidelity, Distribution of Latent Construct Scores = Standard Normal N(0, 1))

c	n	b_i,j ₋ ₁	a_i	β	k	$π_{θ}$	$Δ R_{θ}^{2}$	$π_{x}$	$Δ R_{x}^{2}$	$π_{\hat{θ}}$	$Δ R_{\hat{θ}}^{2}$	α	RMSE	$S W_{θ}$	$S W_{x}$	$S W_{\hat{θ}}$	$s k_{x}$	$s k_{\hat{θ}}$
1	250	Easy	Low	.3	15	0.062	0.018	0.055	0.021	0.047	0.022	.66	0.86	0.96	0.06	0.78	0.54	0.23
2	250	Easy	Low	.3	30	0.062	0.018	0.068^a	0.020	0.052	0.023	.80	0.84	0.96	0.06	0.77	0.56	0.22
3	250	Easy	Low	.5	15	0.062	0.011	0.088^a,b	0.018	0.058	0.018	.66	0.86	0.96	0.23	0.84	0.54	0.24
4	250	Easy	Low	.5	30	0.062	0.011	0.113^a,b	0.017	0.061	0.017	.80	0.83	0.96	0.30	0.82	0.57	0.24
5	250	Easy	High	.3	15	0.062	0.018	0.055	0.021	0.047	0.022	.66	1.44	0.96	0.06	0.78	0.54	0.23
6	250	Easy	High	.3	30	0.061	0.018	0.105^a,b	0.022	0.082^a,b	0.016	.92	1.44	0.96	0.00	0.41	1.01	0.34
7	250	Easy	High	.5	15	0.061	0.011	0.296^a,b	0.020	0.089^a,b	0.012	.86	1.44	0.96	0.01	0.57	0.99	0.34
8	250	Easy	High	.5	30	0.062	0.011	0.372^a,b	0.019	0.098^a,b	0.014	.92	1.44	0.96	0.02	0.61	1.01	0.34
9	250	Moderate	Low	.3	15	0.062	0.018	0.046	0.020	0.056	0.020	.69	0.70	0.96	0.83	0.86	0.02	0.01
10	250	Moderate	Low	.3	30	0.062	0.018	0.042	0.021	0.048	0.020	.82	0.70	0.96	0.86	0.86	0.02	0.02
11	250	Moderate	Low	.5	15	0.062	0.011	0.041	0.016	0.052	0.017	.69	0.70	0.96	0.91	0.86	0.02	0.02
12	250	Moderate	Low	.5	30	0.062	0.011	0.037	0.015	0.037	0.018	.82	0.70	0.96	0.93	0.86	0.02	0.03
13	250	Moderate	High	.3	15	0.062	0.018	0.051	0.019	0.047	0.020	.88	0.53	0.96	0.43	0.88	0.04	0.01
14	250	Moderate	High	.3	30	0.062	0.018	0.048	0.019	0.059	0.019	.94	0.52	0.96	0.50	0.88	0.03	0.00
15	250	Moderate	High	.5	15	0.062	0.011	0.041	0.014	0.050	0.014	.88	0.53	0.96	0.86	0.91	0.04	0.00
16	250	Moderate	High	.5	30	0.061	0.011	0.039	0.013	0.055	0.014	.94	0.52	0.96	0.89	0.91	0.03	0.00
17	250	Difficult	Low	.3	15	0.061	0.018	0.064^a	0.020	0.053	0.021	.65	0.89	0.96	0.03	0.72	0.58	0.21
18	250	Difficult	Low	.3	30	0.061	0.018	0.067^a	0.021	0.052	0.022	.79	0.88	0.96	0.04	0.70	0.60	0.22
19	250	Difficult	Low	.5	15	0.061	0.011	0.106^a,b	0.019	0.075^a	0.018	.65	0.89	0.96	0.15	0.79	0.58	0.21
20	250	Difficult	Low	.5	30	0.061	0.011	0.138^a,b	0.017	0.055	0.019	.79	0.88	0.96	0.23	0.78	0.60	0.20
21	250	Difficult	High	.3	15	0.061	0.018	0.112^a,b	0.023	0.098^a,b	0.011	.85	1.58	0.96	0.00	0.30	1.06	0.37
22	250	Difficult	High	.3	30	0.061	0.018	0.123^a,b	0.022	0.070^a	0.014	.92	1.59	0.96	0.00	0.26	1.08	0.38
23	250	Difficult	High	.5	15	0.061	0.011	0.317^a,b	0.021	0.120^a,b	0.011	.85	1.59	0.96	0.00	0.49	1.06	0.38
24	250	Difficult	High	.5	30	0.061	0.011	0.386^a,b	0.020	0.102^a,b	0.012	.92	1.59	0.96	0.01	0.47	1.08	0.38

Note: c = condition; n = number of individuals; b_{i j}_{− 1} = item category difficulty distribution, Easy (assessment inappropriateness) = N(−1.5, 1), Moderate (assessment appropriateness) = N(0, 1), Difficult (assessment inappropriateness) = N(1.5, 1); a_i = item discrimination distribution, Low = U(.31,.58), High = U(.58,1.13); β = regression weight; k = number of items; $π_{θ}$ = empirical Type I error rate for actual theta scores; $Δ R_{θ}^{2}$ = average effect size for significant interactions for actual theta scores; $π_{x}$ = empirical Type I error rate for number-correct scores; $Δ R_{x}^{2}$ = average effect size for significant interactions for number-correct scores; $π_{\hat{θ}}$ = empirical Type I error rate for estimated theta scores; $Δ R_{\hat{θ}}^{2}$ = average effect size for significant interactions for estimated theta scores; α = average internal consistency for the number-correct scores; RMSE = root mean square error for the estimated theta scores; $SW_{θ}$ = proportion of n.s. Shapiro–Wilk tests for the actual theta scores; $SW_{x}$ = proportion of n.s. Shapiro–Wilk tests for the number-correct scores; ${SW}_{\hat{θ}}$ = proportion of n.s. Shapiro–Wilk tests for the estimated theta scores; $sk_{x}$ = |skewness| for the number-correct scores (abs. value); $sk_{\hat{θ}}$ = |skewness| for the estimated theta scores. Iterations per condition = 1,000.

Significant Type I Error rate based on the results of a binomial test.

Significant Type I Error rate based on the alpha +/- .5alpha criterion.

Table 2.

Results of Simulation 2 (Normal Fidelity, Distribution of Latent Construct Scores = Standard Normal N(0, 1))

c	n	b_i,j−1	a_i	β	k	$π_{θ}$	$Δ R_{θ}^{2}$	$π_{x}$	$Δ R_{x}^{2}$	$π_{\hat{θ}}$	$Δ R_{\hat{θ}}^{2}$	α	RMSE	$S W_{θ}$	$S W_{x}$	$S W_{\hat{θ}}$	$s k_{x}$	$s k_{\hat{θ}}$
25	750	Easy	Low	.3	15	0.049	0.006	0.074^a	0.007	0.056	0.007	.66	0.82	0.95	0.00	0.44	0.55	0.23
26	750	Easy	Low	.3	30	0.049	0.006	0.069^a	0.007	0.057	0.007	.80	0.84	0.95	0.00	0.45	0.57	0.24
27	750	Easy	Low	.5	15	0.049	0.004	0.167^a,b	0.007	0.089^a,b	0.006	.66	0.82	0.95	0.00	0.65	0.55	0.22
28	750	Easy	Low	.5	30	0.049	0.004	0.222^a,b	0.006	0.079^a,b	0.006	.80	0.84	0.95	0.01	0.66	0.56	0.24
29	750	Easy	High	.3	15	0.049	0.006	0.162^a,b	0.008	0.084^a,b	0.006	.86	1.31	0.95	0.00	0.11	1.00	0.31
30	750	Easy	High	.3	30	0.049	0.006	0.142^a,b	0.007	0.066^a	0.006	.92	1.31	0.95	0.00	0.11	1.01	0.32
31	750	Easy	High	.5	15	0.049	0.004	0.627^a,b	0.009	0.173^a,b	0.006	.86	1.31	0.95	0.00	0.52	1.00	0.31
32	750	Easy	High	.5	30	0.049	0.004	0.710^a,b	0.009	0.158^a,b	0.006	.92	1.31	0.95	0.00	0.52	1.00	0.32
33	750	Moderate	Low	.3	15	0.049	0.006	0.056	0.006	0.052	0.006	.69	0.68	0.95	0.32	0.76	0.02	0.01
34	750	Moderate	Low	.3	30	0.049	0.006	0.046	0.007	0.043	0.007	.82	0.66	0.95	0.44	0.77	0.02	0.02
35	750	Moderate	Low	.5	15	0.049	0.004	0.046	0.006	0.055	0.005	.69	0.68	0.95	0.69	0.82	0.02	0.02
36	750	Moderate	Low	.5	30	0.049	0.004	0.043	0.005	0.038	0.006	.82	0.66	0.95	0.81	0.82	0.02	0.02
37	750	Moderate	High	.3	15	0.049	0.006	0.050	0.006	0.053	0.006	.88	0.51	0.95	0.01	0.69	0.03	0.00
38	750	Moderate	High	.3	30	0.049	0.006	0.044	0.006	0.044	0.006	.94	0.53	0.95	0.01	0.71	0.03	0.01
39	750	Moderate	High	.5	15	0.049	0.004	0.065^a	0.005	0.068^a	0.005	.88	0.50	0.95	0.51	0.79	0.03	0.00
40	750	Moderate	High	.5	30	0.049	0.004	0.056	0.004	0.055	0.005	.94	0.53	0.95	0.67	0.80	0.03	0.02
41	750	Difficult	Low	.3	15	0.048	0.006	0.075^a	0.007	0.058	0.007	.66	0.86	0.95	0.00	0.34	0.58	0.21
42	750	Difficult	Low	.3	30	0.049	0.006	0.081^a,b	0.008	0.043	0.007	.79	0.87	0.95	0.00	0.36	0.60	0.20
43	750	Difficult	Low	.5	15	0.049	0.004	0.164^a,b	0.007	0.075^a	0.007	.66	0.86	0.95	0.00	0.56	0.58	0.18
44	750	Difficult	Low	.5	30	0.049	0.004	0.269^a,b	0.007	0.059	0.006	.79	0.87	0.95	0.00	0.60	0.60	0.19
45	750	Difficult	High	.3	15	0.049	0.006	0.159^a,b	0.008	0.066^a	0.007	.85	1.40	0.95	0.00	0.07	1.07	0.33
46	750	Difficult	High	.3	30	0.049	0.006	0.180^a,b	0.008	0.065^a	0.007	.92	1.41	0.95	0.00	0.07	1.09	0.33
47	750	Difficult	High	.5	15	0.049	0.004	0.635^a,b	0.010	0.193^a,b	0.006	.85	1.40	0.95	0.00	0.49	1.07	0.33
48	750	Difficult	High	.5	30	0.049	0.004	0.765^a,b	0.010	0.192^a,b	0.006	.92	1.41	0.95	0.00	0.48	1.09	0.33

Note: c = condition; n = number of individuals; b _{i j} _{− 1} = item category difficulty distribution, Easy (assessment inappropriateness) = N(−1.5, 1), Moderate (assessment appropriateness) = N(0, 1), Difficult (assessment inappropriateness) = N(1.5, 1); a_i = item discrimination distribution, Low = U(.31,.58), High = U(.58,1.13); β = regression weight; k = number of items; $π_{θ}$ = empirical Type I error rate for actual theta scores; $Δ R_{θ}^{2}$ = average effect size for significant interactions for actual theta scores; $π_{x}$ = empirical Type I error rate for number-correct scores; $Δ R_{x}^{2}$ = average effect size for significant interactions for number-correct scores; $π_{\hat{θ}}$ = empirical Type I error rate for estimated theta scores; $Δ R_{\hat{θ}}^{2}$ = average effect size for significant interactions for estimated theta scores; α = average internal consistency for the number-correct scores; RMSE = root mean square error for the estimated theta scores; $SW_{θ}$ = proportion of n.s. Shapiro–Wilk tests for the actual theta scores; $SW_{x}$ = proportion of n.s. Shapiro–Wilk tests for the number-correct scores; $SW_{\hat{θ}}$ = proportion of n.s. Shapiro–Wilk tests for the estimated theta scores; $sk_{x}$ = |skewness| for the number-correct scores (abs. value); $sk_{\hat{θ}}$ = |skewness| for the estimated theta scores. Iterations per condition = 1,000.

Significant Type I Error rate based on the results of a binomial test.

Significant Type I Error rate based on the alpha +/- .5alpha criterion.

Table 3.

Results of Simulation 3 (High Fidelity, Distribution of Latent Construct Scores = Standard Normal N(0, 1))

c	n	b_i,j−1	a_i	β	k	$π_{θ}$	$Δ R_{θ}^{2}$	$π_{x}$	$Δ R_{x}^{2}$	$π_{\hat{θ}}$	$Δ R_{\hat{θ}}^{2}$	α	RMSE	$S W_{θ}$	$S W_{x}$	$S W_{\hat{θ}}$	$s k_{x}$	$s k_{\hat{θ}}$
49	250	Easy	Low	.3	15	0.062	0.018	0.067^a	0.021	0.055	0.020	.64	0.7	0.96	0.01	0.78	0.64	0.23
50	250	Easy	Low	.3	30	0.062	0.018	0.078^a,b	0.020	0.054	0.022	.78	0.69	0.96	0.01	0.75	0.67	0.24
51	250	Easy	Low	.5	15	0.062	0.011	0.099^a,b	0.018	0.064^a	0.018	.64	0.70	0.96	0.10	0.89	0.64	0.23
52	250	Easy	Low	.5	30	0.062	0.011	0.132^a,b	0.018	0.057	0.019	.78	0.69	0.96	0.15	0.83	0.67	0.24
53	250	Easy	High	.3	15	0.062	0.018	0.128^a,b	0.024	0.079^a,b	0.022	.84	1.57	0.96	0.00	0.03	1.34	0.76
54	250	Easy	High	.3	30	0.061	0.018	0.152^a,b	0.023	0.085^a,b	0.021	.91	1.56	0.96	0.00	0.04	1.38	0.76
55	250	Easy	High	.5	15	0.061	0.011	0.390^a,b	0.023	0.215^a,b	0.019	.84	1.57	0.96	0.00	0.26	1.34	0.77
56	250	Easy	High	.5	30	0.062	0.011	0.467^a,b	0.023	0.224^a,b	0.019	.91	1.56	0.96	0.00	0.27	1.37	0.76
57	250	Moderate	Low	.3	15	0.062	0.018	0.047	0.020	0.044	0.021	.68	0.56	0.96	0.77	0.96	0.01	0.00
58	250	Moderate	Low	.3	30	0.062	0.018	0.044	0.020	0.058	0.020	.81	0.56	0.96	0.80	0.97	0.01	0.00
59	250	Moderate	Low	.5	15	0.062	0.011	0.041	0.016	0.050	0.016	.68	0.56	0.96	0.89	0.97	0.01	0.00
60	250	Moderate	Low	.5	30	0.062	0.011	0.040	0.015	0.050	0.017	.81	0.56	0.96	0.90	0.96	0.01	0.00
61	250	Moderate	High	.3	15	0.062	0.018	0.047	0.019	0.056	0.018	.88	0.38	0.96	0.11	0.93	0.02	0.01
62	250	Moderate	High	.3	30	0.062	0.018	0.042	0.020	0.059	0.019	.93	0.39	0.96	0.14	0.93	0.02	0.01
63	250	Moderate	High	.5	15	0.062	0.011	0.031	0.014	0.042	0.014	.88	0.38	0.96	0.79	0.95	0.02	0.01
64	250	Moderate	High	.5	30	0.061	0.011	0.032	0.014	0.054	0.014	.93	0.39	0.96	0.86	0.95	0.02	0.01
65	250	Difficult	Low	.3	15	0.061	0.018	0.075^a	0.021	0.050	0.020	.63	0.72	0.96	0.01	0.74	0.66	0.24
66	250	Difficult	Low	.3	30	0.061	0.018	0.066^a	0.022	0.049	0.021	.78	0.71	0.96	0.01	0.73	0.69	0.24
67	250	Difficult	Low	.5	15	0.061	0.011	0.115^a,b	0.020	0.071^a	0.018	.63	0.72	0.96	0.07	0.83	0.66	0.24
68	250	Difficult	Low	.5	30	0.061	0.011	0.150^a,b	0.018	0.067^a	0.018	.78	0.70	0.96	0.11	0.83	0.69	0.24
69	250	Difficult	High	.3	15	0.061	0.018	0.141^a,b	0.025	0.098^a,b	0.022	.83	1.65	0.96	0.00	0.03	1.39	0.80
70	250	Difficult	High	.3	30	0.061	0.018	0.155^a,b	0.024	0.093^a,b	0.022	.91	1.65	0.96	0.00	0.02	1.42	0.80
71	250	Difficult	High	.5	15	0.061	0.011	0.411^a,b	0.025	0.246^a,b	0.021	.83	1.65	0.96	0.00	0.22	1.40	0.81
72	250	Difficult	High	.5	30	0.061	0.011	0.488^a,b	0.025	0.235^a,b	0.020	.91	1.65	0.96	0.00	0.24	1.43	0.81

Note: c = condition; n = number of individuals; b _i,j _{− 1} = item category difficulty distribution, Easy (assessment inappropriateness) = N(−1.5, 0.5), Moderate (assessment appropriateness) = N(0, 0.5), Difficult (assessment inappropriateness) = N(1.5, 0.5); a_i = item discrimination distribution, Low = U(.31,.58), High = U(.58,1.13); β = regression weight; k = number of items; $π_{θ}$ = empirical Type I error rate for actual theta scores; $Δ R_{θ}^{2}$ = average effect size for significant interactions for actual theta scores; $π_{x}$ = empirical Type I error rate for number-correct scores; $Δ R_{x}^{2}$ = average effect size for significant interactions for number-correct scores; $π_{\hat{θ}}$ = empirical Type I error rate for estimated theta scores; $Δ R_{\hat{θ}}^{2}$ = average effect size for significant interactions for estimated theta scores; α = average internal consistency for the number-correct scores; RMSE = root mean square error for the estimated theta scores; $SW_{θ}$ = proportion of n.s. Shapiro–Wilk tests for the actual theta scores; $SW_{x}$ = proportion of n.s. Shapiro–Wilk tests for the number-correct scores; $SW_{\hat{θ}}$ = proportion of n.s. Shapiro–Wilk tests for the estimated theta scores; $sk_{x}$ = |skewness| for the number-correct scores (abs. value); $sk_{\hat{θ}}$ = |skewness| for the estimated theta scores. Iterations per condition = 1,000.

Significant Type I Error rate based on the results of a binomial test.

Significant Type I Error rate based on the alpha +/- .5alpha criterion.

Table 4.

Results of Simulation 4 (High Fidelity, Distribution of Latent Construct Scores = Standard Normal N(0, 1))

c	n	b_i,j−1	a_i	β	k	$π_{θ}$	$Δ R_{θ}^{2}$	$π_{x}$	$Δ R_{x}^{2}$	$π_{\hat{θ}}$	$Δ R_{\hat{θ}}^{2}$	α	RMSE	$S W_{θ}$	$S W_{x}$	$S W_{\hat{θ}}$	$s k_{x}$	$s k_{\hat{θ}}$
73	750	Easy	Low	.3	15	0.049	0.006	0.086^a,b	0.007	0.066^b	0.007	.64	0.65	0.95	0.00	0.16	0.65	0.26
74	750	Easy	Low	.3	30	0.049	0.006	0.080^a,b	0.007	0.056	0.007	.78	0.65	0.95	0.00	0.16	0.67	0.26
75	750	Easy	Low	.5	15	0.049	0.004	0.183^a,b	0.007	0.100^a,b	0.006	.64	0.65	0.95	0.00	0.54	0.65	0.26
76	750	Easy	Low	.5	30	0.049	0.004	0.268^a,b	0.006	0.086^a,b	0.006	.78	0.65	0.95	0.00	0.55	0.67	0.26
77	750	Easy	High	.3	15	0.049	0.006	0.216^a,b	0.008	0.109^a,b	0.007	.84	1.36	0.95	0.00	0.00	1.35	0.62
78	750	Easy	High	.3	30	0.049	0.006	0.199^a,b	0.009	0.095^a,b	0.007	.91	1.36	0.95	0.00	0.00	1.38	0.62
79	750	Easy	High	.5	15	0.049	0.004	0.740^a,b	0.011	0.407^a,b	0.007	.84	1.36	0.95	0.00	0.07	1.35	0.62
80	750	Easy	High	.5	30	0.049	0.004	0.842^a,b	0.012	0.388^a,b	0.007	.91	1.35	0.95	0.00	0.08	1.38	0.62
81	750	Moderate	Low	.3	15	0.049	0.006	0.047	0.006	0.049	0.006	.68	0.56	0.95	0.15	0.90	0.01	0.00
82	750	Moderate	Low	.3	30	0.049	0.006	0.045	0.007	0.046	0.006	.81	0.56	0.95	0.23	0.89	0.01	0.00
83	750	Moderate	Low	.5	15	0.049	0.004	0.043	0.005	0.047	0.005	.68	0.56	0.95	0.61	0.93	0.01	0.00
84	750	Moderate	Low	.5	30	0.049	0.004	0.046	0.005	0.042	0.006	.81	0.56	0.95	0.74	0.94	0.01	0.00
85	750	Moderate	High	.3	15	0.049	0.006	0.043	0.006	0.049	0.006	.88	0.39	0.95	0.00	0.70	0.02	0.00
86	750	Moderate	High	.3	30	0.049	0.006	0.041	0.006	0.042	0.006	.93	0.39	0.95	0.00	0.72	0.02	0.00
87	750	Moderate	High	.5	15	0.049	0.004	0.045	0.005	0.054	0.004	.88	0.39	0.95	0.32	0.92	0.02	0.00
88	750	Moderate	High	.5	30	0.049	0.004	0.040	0.004	0.047	0.005	.93	0.39	0.95	0.46	0.92	0.02	0.00
89	750	Difficult	Low	.3	15	0.048	0.006	0.080^a,b	0.008	0.059	0.007	.64	0.68	0.95	0.00	0.12	0.67	0.26
90	750	Difficult	Low	.3	30	0.049	0.006	0.094^a,b	0.007	0.041	0.007	.78	0.67	0.95	0.00	0.12	0.69	0.26
91	750	Difficult	Low	.5	15	0.049	0.004	0.180^a,b	0.007	0.094^a,b	0.007	.64	0.68	0.95	0.00	0.50	0.67	0.26
92	750	Difficult	Low	.5	30	0.049	0.004	0.315^a,b	0.007	0.076^a,b	0.006	.78	0.67	0.95	0.00	0.51	0.69	0.27
93	750	Difficult	High	.3	15	0.049	0.006	0.199^a,b	0.009	0.106^a,b	0.008	.84	1.46	0.95	0.00	0.00	1.40	0.66
94	750	Difficult	High	.3	30	0.049	0.006	0.236^a,b	0.009	0.107^a,b	0.008	.91	1.46	0.95	0.00	0.00	1.43	0.65
95	750	Difficult	High	.5	15	0.049	0.004	0.734^a,b	0.012	0.436^a,b	0.008	.84	1.46	0.95	0.00	0.06	1.40	0.66
96	750	Difficult	High	.5	30	0.049	0.004	0.849^a,b	0.013	0.404^a,b	0.008	.91	1.46	0.95	0.00	0.05	1.43	0.65

Note: c = condition; n = number of individuals; b _i,j ₋₁ = item category difficulty distribution, Easy (assessment inappropriateness) = N(−1.5, 0.5), Moderate (assessment appropriateness) = N(0, 0.5), Difficult (assessment inappropriateness) = N(1.5, 0.5); a_i = item discrimination distribution, Low = U(.31,.58), High = U(.58,1.13); β = regression weight; k = number of items; $π_{θ}$ = empirical Type I error rate for actual theta scores; $Δ R_{θ}^{2}$ = average effect size for significant interactions for actual theta scores; $π_{x}$ = empirical Type I error rate for number-correct scores; $Δ R_{x}^{2}$ = average effect size for significant interactions for number-correct scores; $π_{\hat{θ}}$ = empirical Type I error rate for estimated theta scores; $Δ R_{\hat{θ}}^{2}$ = average effect size for significant interactions for estimated theta scores; α = average internal consistency for the number-correct scores; RMSE = root mean square error for the estimated theta scores; $SW_{θ}$ = proportion of n.s. Shapiro–Wilk tests for the actual theta scores; $SW_{x}$ = proportion of n.s. Shapiro–Wilk tests for the number-correct scores; $SW_{\hat{θ}}$ = proportion of n.s. Shapiro–Wilk tests for the estimated theta scores; $sk_{x}$ = |skewness| for the number-correct scores (abs. value); $sk_{\hat{θ}}$ = |skewness| for the estimated theta scores. Iterations per condition = 1,000.

Significant Type I Error rate based on the results of a binomial test.

Significant Type I Error rate based on the results of the alpha +/- .5alpha criterion.

In addition, Figure 2 represents the frequency of the empirical Type I error rates for the number-correct scores and estimated theta scores. For the number-correct scores, these data indicate a positively skewed distribution (skew = 2.04) ranging from 3.1% to 84.9%, with a mean empirical Type I error rate of 17.5%, median of 8.7%, and a standard deviation of 20%. Restricting the summary to only those occurrences outside of the $α \pm \frac{1}{2} α$ interval, the mean empirical Type I error rate was 27.6% with values ranging from 7.8% to 84.9%. The distribution for the estimated theta scores was also positively skewed (skew = 2.97) and ranged from 3.7% to 43.6% with a mean empirical Type I error rate of 9.0%, median of 5.9%, and a standard deviation of 8.0%. Restricting the summary to only those occurrences outside of the $α \pm \frac{1}{2} α$ interval, the mean empirical Type I error rate was 15.9%, with values ranging from 7.6% to 43.6%.

Figure 2.

Distribution of spurious interactions for number-correct scores and estimated theta scores

These results indicate that there were instances of spurious interactions regardless of the scoring method. However, it is clear that the number-correct scores performed much worse than the estimated theta scores in comparable conditions. Finally, an important finding to highlight is that, of the conditions with meaningfully inflated Type I error rates for the estimated theta scores, none were unique with regard to the number-correct scores (see Tables 1-4). In other words, no meaningful inflations existed for the estimated theta scores that did not also exist for the number-correct scores.

Assessment Appropriateness and Type I Errors

The results of this simulation also clearly indicate the anticipated effect of scoring method and assessment appropriateness on the occurrence of Type I errors for the interaction term of a MMR analysis. Figure 3 represents the mean and maximum Type I error rates for each scoring method collapsed across the 32 assessment appropriateness conditions. Under these conditions, there is no significant departure from the nominal Type I error rate, regardless of whether one uses simple raw scores or estimated theta scores in the MMR analysis. These results are consistent with previous findings related to scaling effects on Type I error rates for moderated statistical models (Davison & Sharma, 1990; Embretson, 1996; Kang & Waller, 2005).

Figure 3.

Empirical Type I error rates for the interaction term of a simulated moderated multiple regression model under conditions of assessment appropriateness

However, striking differences in the empirical Type I error rate can be observed for each scoring method when the assessment is inappropriate for the individuals. Figure 4 represents the mean and maximum Type I error rates for each scoring method collapsed across the 64 assessment inappropriateness (easy/difficult) conditions. Number-correct scores resulted in empirical Type I error rates that were above the acceptable interval in 53 of the 64 (83%) inappropriate assessment conditions. At the iteration level, a direct logistic regression analysis indicated that the likelihood of committing a Type I error was 8.13 times greater when number-correct scores from inappropriate assessments were used, χ²(1, N = 96,000) = 5,008.55, p < .001, odds ratio = 8.13. In addition, estimated theta scores resulted in empirical Type I error rates that were above the acceptable interval in 33 of the 64 (51%) inappropriate assessment conditions. A direct logistic regression analysis indicated that the likelihood of committing a Type I error was 2.4 times greater when estimated theta scores from inappropriate assessments were used, χ²(1, N = 96,000) = 918.15, p < .001, odds ratio = 2.40.

Figure 4.

Empirical Type I error rates for the interaction term of a simulated moderated multiple regression model under conditions of assessment inappropriateness

Impact of the Independent Variables on the Empirical Type I Error Rate

Table 5 represents the mean empirical Type I error rate as well as direct logistic regression tests for the levels of each independent variable. The dependent variable in the logistic regression analyses was the occurrence of a Type I error and the iteration level, and was coded as a 1 if the ΔR² between the additive and multiplicative model was significant (p < .05) or a 0 if it was not significant. All of the independent variables were entered into the model simultaneously as categorical predictors. A general pattern can be identified in these results such that higher empirical Type I error rates were observed for the stronger level of each independent variable. This pattern would indicate that each psychometric characteristic that was varied in the simulations had an overall effect on the empirical Type I error rates for the interaction term.

Table 5.

Impact of Individual Predictors on Empirical Type I Error Rates

		M			Direct logistic regression for number-correct score Type I errors^a				Direct logistic regression for estimated theta score Type I errors^b
		$π_{θ}$	$π_{x}$	$π_{\hat{θ}}$	Wald ^c2	df	B	OR^c	Wald χ²	df	B	OR^c
Appropriateness (difficulty)	Appropriate	0.06	0.04	0.05	5,008.55***	1	2.096	8.13	918.15***	1	0.875	2.40
	Inappropriate	0.06	0.24	0.11
Discrimination	Low	0.06	0.10	0.06	4,437.19***	1	1.339	3.82	1,185.01***	1	0.836	2.30
	High	0.06	0.25	0.12
Sample size	250	0.06	0.16	0.09	154.09***	1	0.234	1.26	9.93***	1	0.073	1.08
	750	0.06	0.19	0.09
Fidelity	Normal	0.06	0.16	0.07	165.42***	1	0.242	1.27	364.77***	1	0.446	1.56
	High	0.06	0.19	0.11
Items	15	0.06	0.16	0.09	154.09***	1	0.234	1.26	9.93***	1	0.073	1.08
	30	0.06	0.19	0.09
Beta weights	.3	0.06	0.09	0.06	4,876.13***	1	1.417	4.12	881.98***	1	0.710	2.03
	.5	0.06	0.26	0.12

Note: OR = odds ratio.

Omnibus full model, χ² (1, N = 96,000) = 17,157.51, p < .001, R² = .27.

Omnibus full model, χ² (1, N = 96,000) = 3,571.47, p < .001, R² = .08.

In each case, the OR reported corresponds to increases in the predictor variable (e.g., increased assessment inappropriateness results in higher likelihoods of Type I errors, increases in discrimination results in higher likelihoods of Type I errors, etc.).

***

p < .001.

Several important findings can be identified from these results. First, the psychometric characteristics that were manipulated in this simulation had a stronger overall effect on Type I errors when the variables were operationalized as number-correct scores when compared with estimated theta scores. These results suggest that number-correct scores are more sensitive to measurement effects in parametric analyses than are IRT-derived theta estimates. For both dependent variables, assessment appropriateness was the most impactful predictor of Type I errors, followed by item discrimination and regression weights. This result confirms and extends the effects of assessment appropriateness identified by Kang and Waller (2005), as well as arguments raised by Busemeyer (1980) on the role of assessment difficulty in parametric statistics.

Strength of Spurious Interaction Effects

Finally, the authors were interested in understanding how assessment appropriateness affected the strength of spurious interactions for the different scoring methods. The columns labeled with ΔR² for each respective scoring method in Tables 1 through 4 indicate the average strength of the interaction when a spurious interaction was identified. Because sample size is known to affect the strength of interaction effects, the authors used a multivariate analysis of covariance (MANCOVA) to determine the effect of assessment appropriateness using sample size as a covariate. After adjusting for the effects of sample size, the results indicated a significant effect of assessment appropriateness on the strength of spurious interaction effects for number-correct scores F(1, 93) = 51.92, p < .001, partial η² = .36 such that the average interaction strength in the inappropriate assessment conditions (M = 0.015, SD = 0.007) was significantly greater than the appropriate assessment conditions (M = 0.011, SD = 0.006). A similar, albeit much weaker, result was also identified for estimated theta scores F(1, 93) = 4.47, p < .05, partial η² = .05 such that the average interaction strength in the inappropriate assessment conditions (M = 0.013, SD = 0.006) was significantly greater than the appropriate assessment conditions (M = 0.012, SD = 0.006). No significant difference was identified for actual theta scores. These results indicate that assessment appropriateness has an effect on the strength of spurious interaction effects for number-correct scores and estimated theta scores and that the effect is considerably stronger in the number-correct score conditions.

Discussion

Theoretical and empirical evidence has emerged to suggest that using IRT to operationalize an individual’s standing on a latent construct has important measurement implications over the use of number-correct scores (Borsboom, 2008; Embretson, 1996, 2006; Embretson & DeBoeck, 1994; Harwell & Gatti, 2001; Kang & Waller, 2005; Perline et al., 1979; Reise & Haviland, 2005; Reise et al., 2005; Wainer, 1982). Specifically, IRT-derived theta scores have been demonstrated to be resistant to inflated Type I error rates in moderated statistical models potentially due to achieving an interval, or nearly interval, scale of measurement (Embretson, 1996; Kang & Waller, 2005). This previous work has been limited to applications of dichotomous data and restrictive IRT models. Therefore, the authors’ goal was to extend their understanding of these potentially beneficial measurement properties by modeling multicategory data and implementing a polytomous IRT model. These studies represent a generalizability trend such that each successive study branches further away from the measurement ideal and into the realities of psychological data.

It is imperative to point out that under certain conditions, significantly inflated Type I error rates were observed for both the estimated theta scores and the number-correct scores. This result for the number-correct scores was expected; however, this result for the estimated theta scores was somewhat unexpected. Strong psychometric influences still resulted in inflated Type I error rates for the estimated theta scores from the GRM. However, it was often the case that the Type I error rate of the number-correct scores far exceeded that of the estimated theta scores. For example, in conditions 7, 8, 23, and 24 in Table 1, the Type I error rates for the number-correct scores ranged from .296 to .386 whereas the respective Type I error rates for the estimated theta scores ranged from .089 to .120. Clearly, given the alternative, the estimated theta scores would be more attractive to researchers in these conditions. In addition, in cases where the Type I error rates were grossly inflated, such as conditions 31, 32, 47, and 48 in Table 2 and 79, 80, 95, and 96 in Table 4, the Type I error rate for the number-correct scores was approximately 200% to 450% higher than the Type I error rate for the estimated theta scores. An examination of Figure 2 clearly reveals these disparities between the number-correct scores and estimated theta scores. Although the estimated theta scores did not perform perfectly within the acceptable limits, these results demonstrate a clear preference for their use in applied research when certain psychometric conditions exist.

Another prominent result was that of the role of assessment appropriateness on Type I error rates for both number-correct scores and estimated theta scores. Assessment appropriateness is defined as the congruence between the reliability of an assessment and the latent construct distribution of the individuals responding to an assessment (see Figure 1). The results of these simulations demonstrated that, under conditions of assessment appropriateness, there is no concern as to unacceptable Type I error rates for any psychometric condition or scoring technique. The complimentary result, that spurious interactions were only observed under some conditions of assessment inappropriateness, was also true. Embretson (1996) and Kang and Waller (2005) also identified assessment appropriateness and inappropriateness as the primary factor in their simulations of spurious interaction effects. Embretson (1996) determined that the degree and direction of the inappropriateness fully accounted for the nature of the interaction with regard to treatment groups in a simulated factorial ANOVA. Prior to these studies, Maxwell and Delaney (1985) demonstrated how various distributional shapes of latent constructs can interact with assessment difficulty (appropriateness) to result in artificial group mean differences when the observed scores and latent scores were related through a nonlinear, monotonic relationship. However, the expectation that this effect would not occur when using estimated theta scores was not fully supported. This finding suggests that certain conditions may reduce the level of linearity in the theta–estimated theta relationship. Specifically, an examination of the root mean square error (RMSE) values in these conditions indicates less precise parameter recovery. This would suggest that the degree of congruence between the actual and estimated theta scores was being eroded in exceedingly easy or difficult assessments. Ferrando (2009) explored the issue of measurement inappropriateness in which the appropriateness of an item can be constrained to a particular range of the trait continuum, thus degrading the fit of a linear model above and below the ceiling and floor limits. Indices for determining this range are provided for binary items in a unidimensional, two-parameter model as well as in multidimensional cases and can provide useful information for determining an acceptable range of measurement appropriateness.

These results also suggest a more complex relationship underlying data structures and the assessment of moderators in MMR analyses than perhaps previously thought. Paunonen and Jackson (1988) conducted a simulation in which Type I error rates were compared between ordinary least squares regression (OLS) and principal components regression (PCR) in relation to the multicollinearity of the predictors. Their results indicated that OLS performed much better than did PCR with regard to accurate moderator detection, and that linear transformations of the data had little effect on the Type I error rates for either procedure. In their study, the researchers simulated random effects data from normal distributions just as in this study, but did not investigate any influences of psychometric characteristics on the data (i.e., difficulty, discrimination, assessment appropriateness, etc.). Conceptually, Paunonen and Jackson (1988) generated data as if they were able to collect actual theta scores. It is not surprising, therefore, that their results were well within the normal Type I error rate for MMR. An examination of the empirical Type I error rates for the actual theta scores in Tables 1 through 4 replicate these findings. The generalizability of the results from Paunonen and Jackson’s simulation are therefore limited because actual latent trait scores are unobservable (directly). The results of this study indicate that psychometric characteristics such as assessment appropriateness and the operationalization of the scores have a significant influence on the performance of MMR analyses.

Limitations and Extensions

Violations of the normality assumption in MMR models were present in these simulations, lending some caution to the interpretation of the results. Specifically, skewness was observed in the simulated dependent variables, and the magnitude of the skew was positively related to the empirical Type I error rate for both number-correct scores and estimated theta scores. In addition, the results of the Shapiro–Wilk tests indicated that cases of residual nonnormality also increased with the empirical Type I error rate for both number-correct scores and estimated theta scores (see Tables 1-4). To examine these results more closely, score distributions were generated for three separate conditions in which the skew of the scores appeared to be strongly related to their respective Type I Error rate. Figure 5 contains distributions for the actual theta scores and the derived scores (estimated theta and number-correct scores) from conditions 12, 44, and 96. These conditions represent cases in which number-correct scores and theta scores performed equally well (condition 12), number-correct scores performed poorly but theta scores performed well (condition 44), and neither number-correct scores nor theta scores performed well (condition 96). The commonality that can be observed here is that the poor performance is clearly aligned with the nonnormal score distributions, and the greatest amount of nonnormality is observed under conditions of assessment inappropriateness in which the latent construct distribution is poorly matched with the test information function (reliability) of the assessment.

Figure 5.

Score distributions for conditions 12, 44, and 96

These findings create a potential confound such that the empirical Type I error rates may be partially related to these violations of normality, which were observed as a result of the simulations rather than a controlled factor. Kang and Waller (2005) identified a similar pattern in their study involving dichotomous data, and in a brief investigation, the researchers were able to correct some of the effects of nonnormality using the Box-Cox transformation. Specifically, moderate empirical Type I error rates responded well to this correction, but higher rates were not fully mitigated to the nominal level of .05 (Kang & Waller, 2005).

It is conceivable that variations in observed nonnormality are a result of variations in nonlinearity between the overall scale (test) response function and the latent trait. A preliminary investigation into scatterplots of individual iterations confirmed this relationship (results available on request). Thus, these relationships could be used as an indicator of a risk for spurious interactions.² An additional indication of this effect is the pattern of parameter recovery values (RMSE) reported in Tables 1 through 4. As nonnormality increased (and in inappropriate assessment conditions), the IRT parameter estimation of the theta scores worsened.

Finally, it is pertinent to discuss the inherent arbitrariness of the scaling metrics and the use of the GRM to generate the response score matrices. The actual theta scores were generated without a significant interaction and derived scores were tested for a significant interaction. Although this practice is common in measurement simulations involving the translation from IRT scores to other scoring methods (Embretson, 1996; Harwell et al., 1996; Kang & Waller, 2005), an argument can be made that the authors are favoring the GRM scores as a nonarbitrary trait scale and implicitly satisfying the assumptions of the IRT scale but not the number-correct score scale. A limiting factor that may arise here is that the results only extend to empirical situations in which the theta scale is regarded as the true metric. This limitation highlights the possibility that the interaction null hypothesis is scale specific and the findings may not generalize to situations in which another metric is considered to be the true measurement scale.³

Conclusion

Overall, the results of this study provide support for the use of IRT for the operationalization of latent constructs. However, this support is not ubiquitous and the direct utility of IRT-derived theta scores for parametric statistics is limited to suboptimal psychometric conditions (i.e., assessment inappropriateness). These conditions resulted in ceiling and floor effects, which can be identified in preliminary data analyses; however, some evidence indicates that extreme cases will not be fully corrected with typical score transformations.

If a researcher can demonstrate that the reliability of the assessment and the distribution of construct scores for the individuals are reasonably matched, there is no evidence here that the Type I error rate will reach an unacceptable level for any scoring technique. However, Embretson (1996) cautioned that these conditions are difficult to anticipate, and the use of theta estimates could be justified as a default measurement technique to assuage any concerns. Furthermore, De Boeck and Wilson (2004) suggested that experimental hypotheses can be tested within an appropriate IRT model. This integration of measurement and experimental testing could provide another avenue for greater decision-making accuracy.

The investigation of the performance of polytomous IRT models in a variety of contexts is still an important avenue of measurement research. Harwell and Gatti (2001) specifically identified a need for a deeper understanding of the properties of the scores generated from complex models such as the GRM as well as various item and scale properties that can influence those scores. The simulations conducted in this study represent an important step forward in filling these needs.

Footnotes

Appendix

The error variance term can be derived in the following manner. First, the predictor variables θ₁ and θ₂ and the criterion variable θ₃ are normally distributed with a mean and standard deviation equal to N(0.00, 1.00). Because the standard deviation is simply the square root of the variance, the variance of the predictor and criterion variables is equal to one. Given these conditions, the following derivation gives the error term for the regression models:

(A1a)

σ_{θ_{3}}^{2} = β_{1}^{2} σ_{θ_{1}}^{2} + β_{1}^{2} σ_{θ_{2}}^{2} + σ_{e}^{2},

where $σ_{θ_{1}}^{2}$ , $σ_{θ_{2}}^{2}$ , and $σ_{θ_{3}}^{2}$ are all equal to 1.00; therefore,

(A1b)

1 = β_{1}^{2} + β_{1}^{2} + σ_{e}^{2},

(A1c)

1 - β_{1}^{2} - β_{1}^{2} = σ_{e}^{2},

(A1d)

1 - (β_{1}^{2} + β_{1}^{2}) = σ_{e}^{2},

(A1e)

\sqrt{1 - (β_{1}^{2} + β_{1}^{2})} = σ_{e} .

Finally, given that the two levels of regression weights β₁ and β₂ were simulated as .3 and .5, Equation 6 can be further reduced to the following form.

(A2a)

θ_{3} = . 5 (θ_{1}) + . 5 (θ_{2}) + . 7071 (ε),

(A2b)

θ_{3} = . 3 (θ_{1}) + . 3 (θ_{2}) + . 9055 (ε) .

In the simulation, the criterion variable θ₃ was operationalized with Equations A2a and A2b for the two levels of the regression weights, .3 and .5, respectively. An alternative way of specifying this derivation would be to say that the error associated with each regression model is being sampled from a N(0.00, 0.7071) and N(0.00, 0.9055) distribution for each level of β.

Acknowledgements

The authors would like to acknowledge Niels Waller for his assistance with the R programming code for portions of the simulation. We would additionally like to thank Dr. Paula Popovich, Dr. Jeff Vancouver, and Dr. Victor Heh for their comments on a previous version of this manuscript.

The data presented in this study were generated as a part of the first author’s doctoral dissertation.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Aguinis

Beaty

J. C.

Boik

R. J.

Pierce

C. A.

(2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90, 94-107.

Aiken

L. S.

West

S. G.

(1991). Multiple regression: Testing and interpreting interactions. Thousand Oaks, CA: SAGE.

Baker

F. B.

Kim

S. H.

(2004). Item response theory parameter estimation techniques (2nd ed.). New York, NY: Marcel Dekker.

Borsboom

(2008). Latent variable theory. Measurement, 6, 25-53.

Busemeyer

J. R.

(1980). Importance of measurement theory, error theory, and experimental design for testing the significance of interactions. Psychological Bulletin, 88, 237-244.

Cohen

West

S. G.

Aiken

L. S.

(2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.

Davison

M. L.

Sharma

(1988). Parametric statistics and levels of measurement. Psychological Bulletin, 104, 137-144.

Davison

M. L.

Sharma

(1990). Parametric statistics and levels of measurement: Factorial designs and multiple regression. Psychological Bulletin, 107, 394-400.

De Boeck

Wilson

(2004). A framework for item response models. In De Boeck

Wilson

(Eds.), Explanatory item response models: A generalized linear and non-linear approach (pp. 3-39). New York, NY: Springer.

10.

Embretson

S. E.

(1996). Item response theory models and spurious interaction effects in factorial ANOVA designs. Applied Psychological Measurement, 20, 201-212.

11.

Embretson

S. E.

(2006). The continued search for non-arbitrary metrics in psychology. American Psychologist, 61, 50-55.

12.

Embretson

S. E.

DeBoeck

(1994). Latent trait theory. In Sternberg

R. J.

(Ed.), Encyclopedia of Intelligence (p. 644–647). New York, NY: MacMillan.

13.

Embretson

S. E.

Reise

S. P.

(2000). Item response theory for psychologists. Mahwah, NJ: Earlbaum.

14.

Ferrando

P. J.

(2009). Difficulty, discrimination, and information indices in the linear factor analysis model for continuous item responses. Applied Psychological Measurement, 33, 9-24.

15.

Fields

D. L.

(2002). Taking the measure of work: A guide to validated scales for organizational research and diagnosis. Thousand Oaks, CA: SAGE.

16.

Fischer

G. H.

(1995). Derivations of the Rasch model. In Fischer

Molenaar

(Eds.), Rasch models: Foundations, recent developments, and applications (pp. 15-38). New York, NY: Springer-Verlag.

17.

Gagne

Furlow

Ross

(2009). Increasing the number of replications in item response theory simulations. Educational and Psychological Measurement, 69, 79-84.

18.

Gardner

P. L.

(1975). Scales and statistics. Review of Educational Research, 45, 43-57.

19.

Granberg-Rademacker

J. S.

(2010). An algorithm for converting ordinal scale measurement data to interval/ratio scale. Educational and Psychological Measurement, 70, 74-90.

20.

Hambleton

R. K.

Swaminathan

Rogers

H. J.

(1991). Fundamentals of item response theory. Newbury Park, CA: SAGE.

21.

Harwell

Gatti

G. G.

(2001). Rescaling ordinal data to interval data in educational research. Review of Educational Research, 71, 105-131.

22.

Harwell

Stone

C. A.

Hsu

Kirisci

(1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20, 101-125.

23.

Ihaka

Gentleman

(1996). R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics, 5, 299-314.

24.

Kang

Waller

(2005). Moderated multiple regression, spurious interaction effects, and IRT. Applied Psychological Measurement, 29, 87-105.

25.

Maxwell

Delaney

(1985). Measurement and statistics: An examination of construct validity. Psychological Bulletin, 97, 85-93.

26.

McNemar

(1969). Psychological statistics (4th ed.). New York, NY: Wiley.

27.

Meade

A. W.

Lautenschlager

G. J.

Johnson

E. C.

(2007). A Monte Carlo examination of the sensitivity of the differential functioning of items and tests framework for tests of measurement invariance with Likert data. Applied Psychological Measurement, 31, 430-455.

28.

Mislevy

R. J.

Stocking

M. L.

(1989). A consumer’s guide to LOGIST and BILOG. Applied Psychological Measurement, 13, 57-75.

29.

Muraki

Bock

R. D.

(2003). PARSCALE: IRT item analysis and test scoring for rating-scale data (Version 4.1) [Computer software]. Lincolnwood, IL: Scientific Software International.

30.

Ostini

Nering

M. L.

(2006). Polytomous item response theory models. Thousand Oaks, CA: SAGE.

31.

Paunonen

S. V.

Jackson

D. N.

(1988). Type I error rates for moderated multiple regression. Journal of Applied Psychology, 73, 569-573.

32.

Perline

Wright

B. D.

Wainer

(1979). The Rasch model as additive conjoint measurement. Applied Psychological Measurement, 3, 237-255.

33.

R Development Core Team. (2008). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org

34.

Reise

S. P.

Ainsworth

A. T.

Haviland

M. G.

(2005). Item response theory: Fundamentals, applications, and promise in psychological research. Current Directions in Psychological Science, 14, 95-101.

35.

Reise

S. P.

Haviland

M. G.

(2005). Item response theory and the measurement of clinical change. Journal of Personality Assessment, 84, 228-238.

36.

Reise

S. P.

Waller

(2003). How many IRT parameters does it take to model psychopathology items? Psychological Methods, 8, 164-184.

37.

Robey

R. R.

Barcikowski

R. S.

(1992). Type I error and the number of iterations in Monte Carlo studies of robustness. British Journal of Mathematical and Statistical Psychology, 45, 283-288.

38.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika (Monograph Suppl. 17), 35, 139.

39.

Samejima

(1996). The graded response model. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 85-100). New York, NY: Springer.

40.

Stevens

S. S.

(1946). On the theory of scales of measurement. Science, 103, 677-680.

41.

Stine

W. W.

(1989). Meaningful inference: The role of measurement in statistics. Psychological Bulletin, 105, 147-155.

42.

Stocking

M. L.

(1987). Two simulated feasibility studies in computerised adaptive testing. Applied Psychology: An International Review, 36, 263-277.

43.

Stone

C. A.

(1992). Recovery of marginal maximum likelihood estimates in the two-parameter logistic response model: An evaluation of MULTILOG. Applied Psychological Measurement, 16, 1-6.

44.

Takane

De Leeuw

(1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52, 392-408.

45.

Wainer

(1982). Robust statistics: A survey and some prescriptions. In Keren

(Ed.), Statistical and methodological issues in psychology and social sciences research (pp. 187-214). Hillsdale, NJ: Erlbaum.

46.

Witt

L. A.

(1998). Enhancing organizational goal congruence: A solution to organizational politics. Journal of Applied Psychology, 83, 666-674.

Using the Graded Response Model to Control Spurious Interactions in Moderated Multiple Regression

Abstract

Keywords

Scaling and Item Response Theory (IRT)

Empirical Evidence of IRT Scaling

Method

The GRM

Independent Variables

Sample size (n)

Scale length (k)

Discrimination (ai)

Item difficulty (bi,1 . . . j−1)/assessment appropriateness

Scale fidelity

Regression weights

Fixed Effects

Item response categories (j)

Regression models

Regression Main Effects

Regression Criterion Variables

Number-correct scores

Estimated Theta Scores

Iterations

Simulation-Dependent Variables

Type I errors

Procedure

Results

Assessment Appropriateness and Type I Errors

Impact of the Independent Variables on the Empirical Type I Error Rate

Strength of Spurious Interaction Effects

Discussion

Limitations and Extensions

Conclusion

Footnotes

Appendix

Acknowledgements

Notes

References

Discrimination (a_i)

Item difficulty (b_i,_{1 . . . j−1})/assessment appropriateness