Assessing Dimensionality of Noncompensatory Multidimensional Item Response Theory With Complex Structures

Abstract

The purpose of this study was to investigate the effect of complex structure on dimensionality assessment in noncompensatory multidimensional item response models using dimensionality assessment procedures based on DETECT (dimensionality evaluation to enumerate contributing traits) and NOHARM (normal ogive harmonic analysis robust method). Five methods were evaluated: two DETECT-based methods—exploratory and cross-validated—and three NOHARM-based methods: root mean square residual (RMSR), $χ_{G / D}^{2}$ , and approximate likelihood ratio (ALR). The results suggested that the studied methods had varying degree of success in correctly counting the number of dimensions and in meaningfully labeling sets of items as dimension-like. In two-dimensional, shorter tests, $χ_{G / D}^{2}$ and ALR largely outperformed RMSR- and DETECT-based methods in conditions with small and medium sample sizes, across all levels of complexity and correlations. Lengthening of the test in two-dimensional conditions led to most notably improved accuracy in determining the correct number of dimensions for NOHARM-based RMSR, whereas for the remaining methods, increase in the number of items had differential effect. DETECT-based methods, on the other hand, had more success in labeling sets of items as dimension-like in conditions with two dimensions, irrespective of the test length, suggesting the items that ought to be together were more often grouped together. Performance of the methods was further evaluated in conditions with larger dimensionality (i.e., 3) and conditions with the increased number of items (i.e., longer tests).

Keywords

noncompensatory multidimensional item response theory complex structures dimensionality assessment DETECT NOHARM

Introduction

Dimensionality of a test can be defined as “the number of latent variables that account for the correlations among item responses in a particular data set” (Camilli, Wang, & Fesq, 1995, p. 80). A process of determining test dimensionality, dimensionality assessment, is crucial as it forms the basis for statistical analysis of the data (Hambleton, Swaminathan, & Rogers, 1991; Zhang, 2007). Stout et al. (1996) described the assessment of test dimensionality as a two-stage process: (a) verification or refutation of unidimensionality and (b) if necessary, description of the multidimensional test structure. Although many of the item response theory models make assumptions about a single dimension (unidimensionality) and local independence, if multidimensional structure is expected, understanding which items go with which dimension is just as crucial.

Over the past few decades, researchers have provided arguments for supporting dimensionality assessment and understanding the structure of a test as an important step in testing (Hambleton et al., 1991; Jang & Roussos, 2007; Tate, 2003; Zhang, 2007). Failure to accurately determine the number of dimensions and associated dimensional structure (or misalignment of the psychometric model and the data) may lead to severe consequences in various aspects of testing, including inaccurate and imprecise estimates of item and person parameters, problems in test linking and equating, item bias and test assembly, and problematic score reporting (e.g., Ackerman, 1989, 1994; Chen & Thissen, 1997; Reckase, Carlson, Ackerman, & Spray, 1986; Walker & Beretvas, 2003; Way, Ansley, & Forsyth, 1988; Yen, 1985). Consequently, it is argued that given the role of dimensionality assessment in supporting a variety of psychometric endeavors, assessing dimensionality should be a prerequisite to applying most commonly used item response theory models (Childs & Oppler, 2000; Jang & Roussos, 2007; Nandakumar & Yu, 1996; Nandakumar, Yu, Li, & Stout, 1998; Seraphine, 2000).

Dimensionality of the item responses can be conducted using a number of different procedures (for a review of current and emerging methods, see Jasper, 2010; Levy & Svetina, 2010; Tate, 2003). A procedure can be characterized by researchers’ approach to modeling (e.g., exploratory or confirmatory) and other characteristics associated with that particular method, such as distributional assumptions it makes (e.g., parametric or nonparametric) or the modeling paradigm within which it is commonly applied (e.g., item response or factor analytic).

Research in dimensionality assessment has largely focused on examining dimensionality in situations where a compensatory type of a multidimensional model is assumed. In addition, in many of those cases, the structure of the data is assumed to be factorially simple—meaning, each item has primary association with a single dimension. Research has shown that to a large degree, commonly used methods perform well under conditions that align well with the principles on which they were built, such as simple structure and compensatory multidimensional item response theory (MIRT; Finch & Habing, 2005, 2008; Nandakumar, 1991, 1993; Nandakumar et al., 1998; Nandakumar & Stout, 1993; Nandakumar & Yu, 1996; Stout, 1987; Stout et al., 1996; van Abswoude, van der Ark, & Sijtsma, 2004; Zhang, 2007; Zhang & Stout, 1999b). Fewer studies examined conditions that depart from the principles or assumptions, such as noncompensatory MIRT (e.g., Froelich & Habing, 2008; Hattie, Krakowski, Rogers, & Swaminathan, 1996) and/or complex structure (e.g., Froelich & Habing, 2008; Gierl, Leighton, & Tan, 2006).

The current study focuses on how well two of the more commonly used methods work under conditions that do not align with the foundational principles of the tools. Specifically, this study examines in a systematic way how well do dimensionality assessment procedures implemented in DETECT (dimensionality evaluation to enumerate contributing traits) and NOHARM (normal ogive harmonic analysis robust method) perform in conditions where complex structure exists and where the generating (underlying) MIRT model is noncompensatory.

The article is organized as follows. The next section provides an overview of the dimensionality assessment methods, DETECT and NOHARM, used in the study, followed by a brief summary of current research on the performance of those procedures. The Method section outlines the implemented study design and describes the outcome variables used in the study. In the Results section, performance across the procedures based on DETECT and NOHARM output is reported for the studied conditions. The last section discusses study limitations and implications.

Dimensionality Assessment

Overview of DETECT Procedure

Dimensionality evaluation to enumerate contributing traits (Kim, 1994; Zhang & Stout, 1999a, 1999b) is an estimation procedure based on conditional covariances among the items. As a procedure, it is often used as an exploratory tool for dimensionality assessment and is useful when the data are scored dichotomously.

The main goal of DETECT is to partition the items into clusters such that within a cluster the items are most homogeneous, and the clusters themselves are widely separated, reflecting an assumption of approximate simple structure. If approximate simple structure exists, the theoretical index D will be maximized at the correct dimensionality-based cluster partition D*. The maximum possible value of D, denoted as D*, indicates the amount of multidimensionality the test displays (how much departure from being perfectly fitted by a unidimensional model; Zhang & Stout, 1999b). This implies that when the partition matches approximate simple structure, the maximum value of DETECT will be obtained since all of the within-cluster conditional covariances will be positive and all between-cluster conditional covariances will be negative (Zhang & Stout, 1999b). There are a large number of possible ways in which DETECT can partition the items; therefore, to search the space intelligently, the DETECT procedure employs a generic algorithm in addition to hierarchical cluster analysis to limit the search (Roussos, Stout, & Marden, 1998; Zhang & Stout, 1999b).

If the hypothesis of approximate simple structure is supported, the solution may be interpreted in terms of the number of homogeneous item clusters as the number of dominant dimensions. This is possible because DETECT procedure outputs the number of nonoverlapping clusters and items associated with each of the clusters. To the extent where there are clusters with few items or if approximate simple structure does not hold, inferring the number of dominant dimensions should be done with caution (Jang & Roussos, 2007; Zhang & Stout, 1999b). For a complete description of DETECT procedure, see Zhang and Stout (1999a, 1999b).

Overview of NOHARM Procedure

Normal ogive harmonic analysis robust method (Fraser & McDonald, 1988) is a parametric nonlinear factor analytic estimation method. The model fitted in NOHARM can be represented by the nonlinear factor analytic model or its equivalent compensatory MIRT in the form of the following equation:

P [X_{j} = 1 | θ = (θ_{1}, \dots, θ_{m})] = Φ (β_{j 0} + β_{j 1} θ_{1} + \dots + β_{j m} θ_{m}),

where $X_{j}$ is a response to item j, $Φ (\cdot)$ is a normal ogive, θ is a vector of examinee abilities on m dimensions, and $β_{j m}$ are item parameters. As a method, NOHARM allows for various rotations (e.g., oblique, orthogonal) in exploratory analysis to provide approximate independent clusters (McDonald, 2000). The estimation procedure employed in NOHARM is unweighted least squares that allows for analysis of long and highly dimensional tests (Fraser & McDonald, 1988; McDonald, 2000; Reckase, Thompson, & Nering, 1997).

As originally developed, NOHARM does not produce a formal statistic for the model fit. In exploratory analysis, a user specifies the number of factors to be extracted for any one solution. Several approaches or tests have been proposed to examine model fit based on NOHARM output, including a heuristic approach by Tate (2003) and his reduction in root mean square residual (RMSR), and formal statistics approximate $χ_{G / D}^{2}$ proposed by Gessaroli and De Champlain (1996), approximate likelihood ratio (ALR) proposed by Gessaroli, De Champlain, and Folske (1997), and a T statistic (as an alternative to ALR and $χ_{G / D}^{2}$ ), proposed by Maydeu-Olivares (2001). Three of these methods were considered in the current study, including Tate’s RMSR, ALR, and $χ_{G / D}^{2}$ .

Tate’s reduction in RMSR

Tate (2003) suggested evaluating model fit by a degree of improvement via sequential model fitting. Using this approach, if the higher dimensional model produces 10% or more decrease in RMSR over the preceding model, the higher dimensional model should be retained. Let us assume we fit four exploratory models in NOHARM, and obtained RMSR of .00721 for a model with a single factor and RMSRs of .00622, .00541, and .00511 for models with two, three, and four factors, respectively. The resulting decreases in RMSRs from a single-factor solution to the second-, third-, and fourth-dimensional solutions would be 14%, 13%, and 6%, respectively. Following the recommended rule of 10% decrease, the result of adding the fourth dimension (i.e., going from three to four) resulted in a decrease of 6%; thus given the rule, we would conclude that optimal solution was a three-factor solution.

Approximate $χ_{G / D}^{2}$ (Gessaroli & De Champlain, 1996)

A formal test, the goodness of fit of a particular dimensionality solution based on NOHARM output, was introduced by Gessaroli and De Champlain (1996) as a $χ^{2}$ type statistic. The null hypothesis tests that the off-diagonal elements in the residual correlation matrix produced by the factor analysis equal zero (Finch & Habing, 2005, 2007). If the null hypothesis is not rejected, it can be concluded that the fitted model adequately approximates the observed correlations among the items (Finch & Habing, 2005). The approximate $χ^{2}$ statistic can be computed as follows:

χ_{G / D}^{2} = (N - 3) \sum_{j = 2}^{J} \sum_{j' = 1}^{j - 1} z_{j j'}^{2 (r)},

where N is the number of examinees, J is the total number of items, j and j′ serve to index the items to define the unique pairings of items, and

z_{j j'}^{(r)} = . 5 \log (1 + r_{j j'}^{(r)}) - . 5 \log (1 - r_{j j'}^{(r)})

is the Fisher’s z transformation of the residual correlation for a given item pairing; and

r_{j j'}^{(r)} = \frac{p_{j j'}^{(r)}}{\sqrt{p_{j}^{(0)} (1 - p_{j}^{(0)}) p_{j'}^{(0)} (1 - p_{j'}^{(0)})}},

where $p_{j}^{(0)}$ is the observed proportion of examinees getting item j correct and $p_{j j'}^{(r)}$ is the residual covariance between items j and j′. The resulting statistic is compared to the reference $χ_{G / D}^{2}$ distribution with degrees of freedom given by df = 0.5J(J− 1) −t, where t is the number of independent parameters estimated in fitting the model. In exploratory models, t = (1 +m) ×J and m is the number of dimensions (Finch & Habing, 2005, 2007).

Approximate likelihood ratio (Gessaroli et al., 1997)

An alternative statistics to $χ_{G / D}^{2}$ , the ALR, can be expressed as follows:

ALR = \sum_{j = 2}^{J} \sum_{j' = 1}^{j - 1} G_{j j^{'}}^{2},

where

G_{j j'}^{2} = - 2 \sum_{k_{j} = 0}^{1} \sum_{k_{j'} = 0}^{1} p_{k_{j} k_{j'}} \ln (\frac{{\hat{p}}_{k_{j} k_{j'}}}{p_{k_{j} k_{j'}}}),

where $p_{k_{j} k_{j'}}$ and ${\hat{p}}_{k_{j} k_{j'}}$ are the observed and expected (model-implied) proportions of examinees with scores of $k_{j}$ and $k_{j'}$ for items j and j′ (0 or 1), respectively. Given the dichotomous scoring of items, $p_{k_{j} k_{j'}}$ (and ${\hat{p}}_{k_{j} k_{j'}}$ ) yield four combinations: (a) proportion of both items being correctly answered ( $p_{11}$ ), (b) proportion of both items being incorrectly answered ( $p_{00}$ ), (c) proportion where item j is correctly answered but item j′ is not ( $p_{10}$ ), and (d) the proportion where item j is incorrectly answered but item $j'$ is correctly answered ( $p_{01}$ ). The ALR statistic is compared with the reference $χ^{2}$ distribution with the same degrees of freedom as $χ_{G / D}^{2}$ .¹

To determine the optimal number of factors or dimensions, a χ² difference test is adopted. First, for each M-dimensional fitted model, $χ_{G / D}^{2}$ (or ALR) is calculated. The sequential testing begins by subtracting the calculated statistic from the M-dimensional model from the statistic from the (M− 1)-dimensional model. The difference is treated as a χ² variate with degrees of freedom equal to the difference in the number of estimated parameters. If the observed difference is larger than the critical value based on the appropriate χ² distribution, we infer that the M-dimensional model is favored and selection stops. Alternatively, the procedure is repeated by comparing the (M− 1)-dimensional model with the (M− 2)-dimensional model. The optimal factor solution is found when the higher dimensional model does not improve the fit significantly: that is, when p > .05.² Once the optimal solution is selected, NOHARM output (estimates of loadings) of that model is used in computing and reporting results.

Research Using DETECT and NOHARM Procedures

Previous studies generally found support for dimensionality assessment using DETECT- or NOHARM-based methods when compared to other procedures, although varying degrees of superiority were noted. As an example, consider Finch and Habing (2005), who in an extensive simulation study evaluated DETECT’s and NOHARM’s performances. The authors investigated two- and six-dimensional structures using two- and three-parameter logistic models and found that in two-dimensional (2D) cases, DETECT and NOHARM performed equally well in conditions with higher correlation, although DETECT was more successful in recreating dimensional structures in conditions with lower correlations. Neither DETECT nor NOHARM seemed to be affected by the sample size, although test length and skewness of the data had varying impact (it should be noted that Finch and Habing found ALR to be superior over $χ_{G / D}^{2}$ across the studied conditions).

In six-dimensional (6D) conditions, ALR tended to outperform DETECT, although as in the 2D case, the number of subjects did not seem to have an impact on either procedure. Increase in the number of items, however, had opposite effect in 6D than in 2D case; ALR benefited from the increase in items, whereas DETECT’s performance deteriorated in high-dimensional cases with the increased number of items.

Furthermore, Finch and Habing (2005) found that when the procedures erred (i.e., did not successfully recreate the dimensional structure), the DETECT method appeared to separate items that should have been grouped together, whereas the ALR appeared to group items that should have been kept separate. Last, unlike in the Finch and Habing (2003) study, the authors found in the 2005 study that guessing had little effect on either of the methods.

As alluded to earlier, it was often the case that the studies assumed the (approximate) simple structure (e.g., Finch & Habing, 2005; van Abswoude et al., 2004; Zhang & Stout, 1999b), while they investigated a number of different effects on the procedures’ performance, such as the model choice—that is, when guessing was present (e.g., Finch & Habing, 2003, 2005, 2007); the varying number of dimensions; test length; sample size; and distributional characteristics of the data (e.g., Finch & Habing, 2003, 2005, 2007; Gessaroli & De Champlain, 1996; Zhang & Stout, 1999b).

Only a few studies examined the performance of these methods when complex structure was present in the data. Gierl et al. (2006) examined DETECT’s performance in 2D conditions varying the amount of complexity present, whereas De Champlain and Gessaroli (1998) investigated performance of NOHARM-based $χ_{G / D}^{2}$ , in comparison with other factor analytical procedures.

In a simulation study with 2D cases, Gierl et al. (2006) found that DETECT was generally very successful in accurately recovering the dimensional structure some complex structures, particularly when 30% or less items were complex, correlation between the traits was≤.75, and N≥1,000. The authors recommended that in cases when large numbers of items were expected to display complex structure, DETECT should be used for dimensionality analysis with large sample size, N≥1,500, and in situations where latent traits were correlated up to .60. Initial results from De Champlain and Gessaroli (1998) suggested that $χ_{G / D}^{2}$ was relatively unaffected by the sample size, the number of items, the item parameter structure, and correlations between the traits considered in the study.

Although many of the results from these studies seem encouraging, the authors widely acknowledge that positive results should be situated within a restrictive set of studied conditions. Therefore, the current study focuses on how well the two commonly used procedures in dimensionality assessment work under conditions that do not align with the foundational principles of the tools but align with the use of psychometric models that allow for rich and refined inferences—specifically in noncompensatory MIRT when complex structure exists.

Method

Methods to Evaluate Dimensionality

As stated previously, this study examined the performance of DETECT- and NOHARM-based methods. Within the exploratory DETECT, dimensionality assessment was conducted using

exploratory DETECT (DETECT_E) and

cross-validated DETECT (DETECT_CV),

where 50% of the sample was randomly used as a training sample, whereas the remaining 50% was used in cross-validation analysis. In this study, the maximum of clusters to be extracted by DETECT was set at five.

Similarly, in NOHARM, for each condition, exploratory one- through five-factor models were fit and three NOHARM-based methods were computed for each using solutions based on Promax rotation:

RMSR

$χ_{G / D}^{2}$ (Gessaroli & DeChamplain, 1996)

ALR (Gessaroli et al., 1997)

Default options were employed for the procedures.

Determining the (optimal) number of dimensions

Since DETECT output nonoverlapping clusters of items, the optimal number of dimensions was obtained in a straightforward way. The partition that maximized the DETECT index was considered a preferred solution or optimal dimensionality. For example, if DETECT output two clusters of items, the recorded preferred solution was considered to be 2D. For the NOHARM-based models (as with any factor analytic procedure), determining the optimal number of dimensions required an outside evaluation. In the current study, this evaluation was based on formal test of sequential model fitting via difference tests (for $χ_{G / D}^{2}$ and ALR) or heuristic approach via reduction in RMSR; procedures previously discussed.

Study Design

Several factors were manipulated in this study, including

Number of dimensions (two or three)

Structure type of data (0% complexity—simple structure, 10% of items exhibiting complexity, 30% or 50%)

Correlations between dimensions (.00, .30, .60, .75, or .90)

Sample size (small: 500, medium: 1,000, or large: 2,000)

Number of items per dimension (10 items per dimension for shorter tests or 20 items per dimension for longer tests)

Binary item responses were generated to follow a two-parameter normal-ogive noncompensatory MIRT (Sympson, 1978) model:

P (X_{i j} = 1 | θ_{i}, a_{j}, b_{j}) = \prod_{m = 1}^{M} \frac{1}{1 + \exp [a_{j m} (θ_{i m} - b_{j m})]},

where $X_{i j}$ is a response of person i to item j, $b_{j m}$ is the location for item j along dimension m, $a_{j m}$ is the discrimination for item j along dimension m, $θ_{i m}$ is the ability level for person i on dimension m, and m equals the number of dimensions.

Language R Version 2.10 (R Development Core Team, 2010) was used to simulate the binary item responses such that each item response vector conformed to the conditions outlined above. The fully crossed design yielded a total of 240 conditions, where each condition was replicated 500 times. Item parameters were fixed across all conditions, and they ranged in values similar to those used in comparable studies (e.g., Ansley & Forsyth, 1985; Gierl et al., 2006). Item locations ranged from −1.50 to 1.50, with 0.75 increments, whereas item discrimination parameters ranged from 0.80 to 1.60, with increments of 0.20. A complex item had a discrimination parameter on multiple latent traits such that influence of dominant trait was always the highest, and the remaining dimensional influence(s) decreased by 0.20. For example, in the 3D condition, a complex item might have a discrimination parameter of 1.40 on its dominant dimension and 1.20 discrimination parameters on the remaining two dimensions. Person parameters were generated from multivariate normal distributions with an appropriately sized mean vector of 0 and covariance matrix Σ, where the diagonal elements of Σ are all 1 and the off-diagonal elements were given by the covariance for the associated condition.

Outcome Variables

Two outcome variables were used to evaluate the performance of the procedures: (a) the proportion of correct selection of true (modeled) dimensionality and (b) the ability to label sets of items as dimension-like.

The proportion of correct selection of true dimensionality

This variable was computed as the proportion of times across 500 replications within each condition that a “true” dimensional space was favored by a method. A “true” dimensional space was defined by the generating model; either 2D or 3D structures were considered and therefore counted.

The ability to label sets of items as “dimension-like.”

This outcome variable was computed in a two-step process: (a) grouping of the items and (b) labeling of the groups of items. Inclusion of this outcome variable allowed for the emphasis on interpretation of the clusters or sets of items resulted from the analysis. The goal was to examine how often a set of items could be interpreted as adequately representing one of the true underlying dimensions.

Grouping of the items

In DETECT, sets of items were determined and grouped “automatically,” as the procedure output nonoverlapping clusters with specifications to which cluster each item belongs.

In NOHARM, however, some manipulation was required to obtain groupings of items. Items were grouped such that the following criteria were applied to the rotated Promax factor solution selected as optimal by each of the methods. For an item to be considered associated with a particular factor, the item must have an estimated loading of >.40 on that particular factor and the difference between the loading and all its other loadings (i.e., loadings on remaining factors) must be >.20.

If the item had an estimated loading >.40 and the difference between its largest loading and at least one other loading was <.20, the item was labeled as complex (note this complexity was with respect to the fitted factor model, which may not necessarily correspond to whether the item was generated as a factorially complex item). Alternatively, if an item failed to meet either criteria (i.e., its loadings were <.40 on all factors), the item was considered to be unexplained (unexplained here referred to the fact that an item failed to have a substantial loading or association with any dimension).

Labeling of sets of items

After all items were grouped in their respective sets, the labeling of these “item groups” or “item sets” as dimension-like began. A set of items can be labeled as Dimension-1-like, Dimension-2-like, or Dimension-3-like set of items, depending on the true dimensionality of the data (e.g., in 2D conditions, item sets could be labeled as Dimension-1-like or Dimension-2-like).

For a set of items to be labeled as dimension-like, three criteria ought to be met. First, at least 50% of the items in the set must be items that were generated as factorially simple (i.e., ought to reflect a single dominant dimension). Second, these factorially simple items ought to occupy more than half of the set to which they belong (i.e., a majority of items in the set ought to be factorially simple, reflecting a particular dimension) in order to consider all of the items in that set as dimension-like items. Third, the proportion of factorially simple items in the cluster under consideration (e.g., Dimension-1-like items in Cluster 1) must be larger than the proportion of the same type of items in any other cluster (e.g., Dimension-1-like items in Cluster 2 or Cluster 3). These three criteria combined allow for individual clusters to be evaluated separately when optimal solution favors mth cluster solution, without labeling multiple clusters as the same dimension-like (i.e., any two clusters cannot be labeled as the same dimension-like sets).

To illustrate how these three criteria were applied, consider the following situation. Let us assume a 2D condition with 10 items per dimension and 30% complexity in the data. According to the study design, items 1 to 10 belong to Dimension 1 and items 11 to 20 belong to Dimension 2. Furthermore, because data exhibit 30% complexity, a total of 6 items (3 per dimension) are modeled as complex. Therefore, the “true” associations of the items are as follows: Items 3, 7, and 9 associated with Dimension 1 are modeled as factorially complex, whereas Items 1 and 2, 4 to 6, 8, and 10 are modeled as factorially simple items associated with Dimension 1. Likewise, Items 13, 17, and 19 are modeled as factorially complex items, whereas Items 11 and 12, 14 to 16, 18, and 20 are modeled as factorially simple items associated with Dimension 2.

If DETECT_E outputs solution of two clusters such that Items 1 to 9 are grouped into Cluster 1 and Items 10 to 20 are grouped into Cluster 2, our evaluation of the criteria would be as follows:

In Cluster 1, we have ~87% of items modeled as factorially simple (six out of the original seven items), which meets the first criterion to label this set as Dimension-1-like. The second criterion is also met, as the six factorially simple items comprise more than 50% of the total size of cluster 1 (six out of nine items; ~67%). Note that the proportion of factorially simple items in Cluster 1 associated with Dimension 1 is .87, which is greater than the remaining proportion of items in Cluster 2, which is .13. Given that all three criteria are met, we would consider Items 1 through 9 from Cluster 1 as Dimension-1-like.

The same procedure is applied to Cluster 2, which contains Items 10 through 20. Here, we have 1 item (Item 10) that is originally modeled as a factorially simple item associated with Dimension 1; Items 11 and 12, 14 to 16, 18, and 20 are modeled as factorially simple associated with Dimension 2; and Items 13, 17, and 19 are modeled as factorially complex items. Applying the first criterion, we see that all 7 items modeled as factorially simple associated with Dimension 2 are grouped together (100% of items originally modeled as factorially simple are grouped together). The second criterion is also met, as these 7 items comprise more than 50% of the total set of 11 items (7 out of 11). Last, the third criterion of proportionally is also met, as the proportion of items that are of particular modeled dimensionality is larger in Cluster 2 than in Cluster 1 (1.0 > .0). For this one replication (i.e., selected optimal solution), DETECT_E’s ability to label both sets of items as dimension-like would be counted.

It is to be noted that the marginal proportions computed across replications within any one condition ignore the accuracy aspect of the methods. In other words, ability to label sets of items as dimension-like is evaluated for a dimensional solution irrespective of whether that solution is indeed reflective of true, generated, dimensionality. Going back to our example of 2D condition, if DETECT_E provided a three-cluster solution, each cluster of items would be submitted to the three criteria described above and count of the ability to label sets of items as Dimension-1-like and Dimension-2-like would be taken across all three clusters.

Results

Overall, the results are presented in two parts, each corresponding to the outcome variable considered in the study. Within each part, relevant selected results are presented. Complete tables and figures may be obtained from the author by request.

Part 1: Proportion Correct

Shorter tests with 10 items per dimension in 2d structures

In Table 1, the proportion of correctly identified dimensions (i.e., counting of dimensions), henceforth proportion correct, for 2D conditions with 10 items per dimension is presented. Several patterns emerged for the studied procedures. First, the NOHARM-based RMSR seldom identified the correct number of dimensions across any complexity level, sample size, or correlation. Second, the other two NOHARM-based methods— $χ_{G / D}^{2}$ and ALR—outperformed DETECT-based methods in all but two conditions. Third, across all complexity levels, the increase in sample size tended to decrease the proportion correct for $χ_{G / D}^{2}$ and ALR, with one notable exception in conditions with correlation of .90. Similarly, holding sample size constant, the increase in correlation generally resulted in poorer performance for all methods when correlation was ≤.75. Last, when correlation was .90, the increase in complexity levels and the sample size resulted in some of the highest proportion correct rates for $χ_{G / D}^{2}$ and ALR.

Table 1.

Proportion Correct for Methods Where Data Follow 2D Noncompensatory MIRT With 10 Items per Dimension.

		Complexity Level
		0%					10%					30%					50%
ρ	N	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV
.00	500	.71	.83	^b	.32	.16	.72	.80	^b	.28	.14	.76	.81	^b	.06	.05	.77	.88	^b	.13	.11
	1,000	.72	.80	^b	.61	.22	.67	.74	^b	.49	.18	.67	.74	.01	.07	.09	.71	.81	^b	.11	.14
	2,000	.56	.56	^b	.84	.43	.59	.58	^b	.68	.35	.55	.53	.01	.06	.06	.53	.61	^b	.09	.12
.30	500	.72	.82	.02	.11	.07	.69	.78	^b	.08	.06	.72	.81	^b	.01	.04	.82	.88	^b	.06	.08
	1,000	.71	.76	^b	.16	.08	.65	.71	^b	.11	.08	.67	.74	^b	.01	.02	.70	.81	^b	.03	.09
	2,000	.59	.58	^a	.25	.13	.48	.44	^b	.14	.07	.46	.44	^b	^a	.01	.50	.52	^a	.01	.05
.60	500	.67	.76	^b	.01	.02	.67	.70	^b	^b	.02	.55	.58	^b	^b	.01	.59	.62	^b	^b	.05
	1,000	.64	.72	.01	^b	^b	.54	.50	^a	^b	^b	.56	.58	^b	^a	^b	.53	.58	^b	.01	.02
	2,000	.50	.45	^a	^b	.01	.22	.15	^b	^a	.01	.21	.14	^a	^a	^b	.17	.15	^a	^a	.01
.75	500	.58	.61	^b	^b	^b	.54	.59	^b	^a	.01	.49	.47	^b	^a	.03	.44	.44	^b	^b	.07
	1,000	.65	.65	.01	^a	^b	.40	.37	^b	^b	^b	.52	.61	^b	^b	^b	.49	.57	^a	^a	.03
	2000	.36	.28	^b	^a	^a	.08	.02	^a	^b	^a	.27	.25	^b	^a	^b	.15	.13	^a	^a	^b
.90	5,00	.43	.46	.02	^a	.01	.48	.57	^b	^a	.02	.53	.56	^b	^b	.03	.55	.55	.01	.01	.05
	1,000	.53	.56	.02	^a	^b	.64	.78	^b	^a	^b	.68	.88	.01	^a	^b	.75	.90	.01	^b	.03
	2,000	.48	.55	^b	^a	^a	.59	.68	^b	^a	^a	.75	.91	^b	^a	^a	.81	.96	^b	^a	.01

Note. 2D = two-dimensional; MIRT = multidimensional item response theory; ALR = approximate likelihood ratio; NOHARM = normal ogive harmonic analysis robust method; DETECT = dimensionality evaluation to enumerate contributing traits; RMSR = root mean square residual. R stands for NOHARM-based RMSR; D_E stands for DETECT exploratory; D_CV stands for DETECT cross-validated.

Indicates actual zero correct.

Indicates <.01 proportion correct.

Impact of lengthening the 2D test

An increase in the number of items associated with each dimension had differential impact on the methods (see Table 2). Out of the five methods, the greatest impact of lengthening the test was observed for RMSR, where the method yielded some of the highest proportions correct. The greatest improvement across all complexity levels was noted in conditions with small sample sizes across complexity and correlation levels. This result is contrast to that obtained in 2D conditions with shorter tests, in which RMSR was the worst performing method.

Table 2.

Proportion Correct for Methods Where Data Follow 2D Noncompensatory MIRT With 20 Items per Dimension.

		Complexity Level
		0%					10%					30%					50%
ρ	N	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV
.00	500	.58	.21	.87	.65	.21	.85	.18	.87	.58	.20	.97	.23	.83	.26	.14	.95	.24	.84	.25	.17
	1,000	.54	.12	.81	.92	.44	.87	.11	.85	.92	.39	.92	.08	.78	.53	.18	.95	.09	.67	.52	.22
	2,000	.41	.01	.55	1.0	.81	.67	^b	.47	.99	.73	.77	.01	.46	.81	.38	.73	^a	.19	.73	.37
.30	500	.94	.19	.89	.29	.11	.97	.16	.83	.25	.13	.87	.20	.76	.10	.06	.81	.23	.74	.13	.15
	1,000	.89	.08	.83	.64	.15	.91	.04	.67	.50	.15	.94	.11	.56	.14	.09	.90	.04	.53	.17	.17
	2,000	.70	.01	.54	.92	.42	.75	^a	.22	.82	.32	.66	^a	.07	.22	.11	.61	^a	.06	.28	.14
.60	500	.87	.16	.86	.02	.04	.85	.05	.62	.04	.06	.26	.09	.40	.01	.02	.19	.08	.40	.06	.06
	1,000	.92	.08	.76	.05	.05	.89	^b	.18	.04	.04	.56	^b	.06	.01	.02	.43	^a	.06	.03	.09
	2,000	.64	.01	.29	.10	.02	.52	^a	^a	.07	.05	.41	^a	^a	.01	^b	.29	^a	^a	.01	.05
.75	500	.29	.13	.83	.01	.03	.25	.02	.42	.01	.02	.11	.16	.41	^b	.02	.07	.15	.27	.03	.04
	1,000	.67	.02	.55	^b	.01	.59	^b	.02	^b	.02	.30	^a	.04	^a	^b	.27	^a	^b	^b	.02
	2,000	.64	^a	.06	^a	^b	.37	^a	.00	^a	.01	.55	^a	^a	^a	^a	.48	^a	^a	^a	.01
.90	500	.05	.22	.26	^a	.01	.08	.28	.64	^b	.03	.12	.56	.87	^b	.02	.15	.72	.83	.01	.03
	1,000	.10	.13	.45	^a	^a	.29	.20	.61	^a	^b	.37	.58	.90	^a	^b	.47	.63	.79	^a	.01
	2,000	.27	.00	.12	^a	^a	.77	.01	.11	^b	^b	.86	.37	.79	^a	^a	.95	.32	.47	^a	^b

Indicates actual zero correct.

Indicates <.01 proportion correct.

The ALR method was affected by the increase in the number of items in different ways. In conditions with no complexity, positive impact (i.e., higher proportion correct) was noted in conditions with correlations between .30 and .75 across sample sizes. As the complexity level increased, ALR typically yielded higher rates in conditions with correlations at .60 or lower; it is to be noted, however, that in conditions with N = 500 and increased complexity, ALR generally tended to deteriorate in performance.

The $χ_{G / D}^{2}$ method was generally negatively affected by the lengthening of the test across all complexity levels, sample size, and correlations. The proportion correct rates for $χ_{G / D}^{2}$ sharply decreased across most conditions, suggesting the method did not perform well in the conditions with increased number of items and in 2D cases.

DETECT-based methods were particularly positively affected by the increase in the number of items in conditions with correlations of ≤.30, across complexity levels and sample sizes. As the sample size increased, the proportion correct tended to increase for both DETECT-based methods, particularly DETECT_E, in these low(er) correlation conditions. At correlation of ≥.60, proportions correct were generally very low, never exceeding .10 across any complexity levels or sample sizes for either DETECT-based method.

Shorter tests with 10 items per dimension in 3d structures

The proportion correct for 3D conditions with 10 items per dimension is presented in Table 3. Generally, none of the methods performed well in recovering the true underlying number of dimensions, except in conditions with no correlation and large sample size where the NOHARM-based methods $χ_{G / D}^{2}$ and ALR outperformed the remaining methods across all complexity levels. At correlation levels of .60 or higher, none of the methods was successful in correctly counting the number of dimensions across the studied conditions (e.g., the highest rates were obtained by $χ_{G / D}^{2}$ and ALR in condition with zero complexity, correlation of .60, and large sample size; .64 and .62, respectively). The two DETECT-based methods and RMSR were not successful in correctly identifying the number of dimensions across any condition.

Table 3.

Proportion Correct for Methods Where Data Follow 3D Noncompensatory MIRT With 10 Items per Dimension.

		Complexity Level
		0%					10%					30%					50%
ρ	N	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV
.00	500	.65	.71	.03	.07	.10	.70	.66	.02	.10	.15	.17	.23	.06	.04	.16	.26	.33	.11	.11	.23
	1,000	.87	.86	.02	.04	.08	.89	.79	.03	.02	.11	.49	.50	.04	^b	.05	.59	.41	.11	.05	.16
	2,000	.85	.89	.02	.02	.05	.86	.81	.04	.02	.05	.81	.78	.08	.00	.01	.87	.46	.13	.00	.07
.30	500	.42	.49	.03	.03	.11	.41	.36	.03	.07	.17	.08	.15	.08	.03	.23	.19	.32	.16	.11	.34
	1,000	.79	.79	.02	^b	.06	.72	.62	.01	.04	.11	.21	.23	.06	^b	.09	.32	.35	.16	.04	.20
	2,000	.86	.85	.05	^b	.02	.80	.47	^b	^b	.09	.44	.43	.04	.00	.03	.60	.27	.07	.00	.08
.60	500	.13	.17	.06	.03	.15	.12	.24	.03	.06	.19	.07	.13	.11	.04	.25	.14	.33	.18	.15	.39
	1,000	.32	.30	.05	.00	.04	.28	.33	.00	.02	.12	.19	.20	.10	.00	.11	.30	.36	.24	.02	.27
	2,000	.62	.64	.03	.00	.02	.51	.24	.00	^b	.04	.41	.35	.05	.00	.02	.56	.43	.11	.00	.09
.75	500	.07	.09	.07	.01	.13	.10	.20	.04	.04	.19	.11	.16	.13	.03	.32	.18	.34	.25	.12	.39
	1,000	.12	.08	.08	^b	.06	.24	.27	.03	^b	.09	.19	.18	.14	^b	.10	.32	.30	.27	.03	.25
	2,000	.25	.14	.07	.00	.01	.48	.39	^b	.00	.03	.32	.17	.13	.00	.01	.40	.34	.25	.00	.11
.90	500	.05	.10	.08	.04	.16	.11	.22	.05	.05	.26	.13	.18	.14	.05	.32	.19	.31	.23	.15	.43
	1,000	.10	.08	.09	.00	.07	.24	.19	.08	^b	.09	.21	.16	.15	.00	.15	.32	.25	.26	.02	.30
	2,000	.09	.05	.12	.00	^b	.34	.17	.07	^b	.03	.25	.10	.18	.00	.01	.29	.24	.30	.00	.09

Note. 3D = three-dimensional; MIRT = multidimensional item response theory; ALR = approximate likelihood ratio; NOHARM = normal ogive harmonic analysis robust method; DETECT = dimensionality evaluation to enumerate contributing traits; RMSR = root mean square residual. R stands for NOHARM-based RMSR; D_E stands for DETECT exploratory; D_CV stands for DETECT cross-validated.

Indicates actual zero correct.

Indicates <.01 proportion correct.

Impact of lengthening the 3D test

Lengthening the test in 3D conditions had differential, although in some cases, meaningful effect on the methods. As seen in Table 4, the most notable and generally positive effect was found for NOHARM-based RMSR, proportions correct for which increased in many conditions; in particular, high proportions correct were yielded when complexity was 30% or less and correlations ≤.30. The other two NOHARM-based methods, ALR and $χ_{G / D}^{2}$ , were generally negatively affected by the increase in the number of items across complexity levels and sample sizes. Only a handful of exceptions were found, in conditions with 30% complexity and correlations of .75 or higher and 10% complexity and correlation of .90, where the $χ_{G / D}^{2}$ method benefited from the increase in the number of items. Low proportions correct for the ALR method were observed across all complexity levels and correlations. DETECT-based methods generally benefited from the lengthening of the test, in particular in conditions with low(er) correlations across complexity and sample size levels. Between the two DETECT-based methods, DETECT_CV tended to yield higher proportions correct compared to DETECT_E. Compared with the NOHARM-based methods, the DETECT-based methods did not perform well. When the DETECT-based methods outperformed the NOHARM-based methods $χ_{G / D}^{2}$ or RMSR, the rates themselves were not very high (never exceeding .35). In other words, the proportions remained quite low, suggesting that the methods were not successful in identifying the correct number of dimensions across the studied conditions in shorter or longer tests.

Table 4.

Proportion Correct for Methods Where Data Follow 3D Noncompensatory MIRT With 20 Items per Dimension.

		Complexity Level
		0%					10%					30%					50%
ρ	N	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV
.00	500	.16	.39	.93	.20	.34	.16	.30	.95	.31	.34	^b	.27	.92	.18	.25	.01	.08	.87	.34	.30
	1,000	.36	.60	.97	.16	.30	.37	.49	.98	.20	.36	.05	.34	.98	.05	.21	.10	.09	.99	.31	.32
	2,000	.44	.76	.99	.17	.19	.50	.54	.98	.18	.29	.26	.33	.98	^b	.07	.39	.06	.99	.16	.32
.30	500	.04	.33	.96	.19	.25	.01	.15	.94	.27	.31	^a	.21	.26	.12	.24	^b	.04	.11	.27	.34
	1,000	.17	.52	.97	.07	.25	.16	.15	.90	.12	.33	^b	.12	.70	.03	.17	.01	^b	.54	.22	.27
	2,000	.44	.71	.99	.04	.13	.42	.03	.70	.07	.20	.06	.01	.29	^a	.07	.15	^b	.26	.11	.25
.60	500	^a	.27	.63	.10	.19	^a	.09	.35	.19	.24	^a	.28	.04	.08	.27	^a	.10	^b	.16	.29
	1,000	^b	.47	.98	.08	.15	^b	^b	.25	.09	.21	^b	.32	.18	^b	.20	^b	.09	.10	.10	.28
	2,000	.09	.58	.99	^b	.11	.06	^a	^a	.03	.18	.02	.12	.20	^a	.04	.09	.01	.28	.01	.13
.75	500	^a	.23	.07	.07	.15	^a	.24	.12	.08	.28	^a	.38	.02	.09	.30	^b	.14	^b	.18	.29
	1,000	^b	.38	.43	.03	.13	^b	.20	.33	.04	.17	^b	.37	.07	^b	.19	^b	.23	^b	.07	.30
	2,000	^b	.51	.95	^b	.05	.04	.05	.05	.01	.08	^b	.43	.14	^a	.05	.02	.19	.02	^b	.14
.90	500	^a	.22	^a	.05	.20	^b	.27	.03	.07	.26	^a	.34	^b	.11	.31	^b	.25	^a	.20	.35
	1,000	^a	.23	^b	.01	.12	^a	.36	.04	.01	.13	^b	.40	^a	^b	.21	^b	.33	^a	.07	.27
	2,000	^a	.23	^a	^a	.03	.01	.39	.07	^b	.04	.01	.29	^b	^a	.05	.01	.33	^a	^b	.17

Indicates actual zero correct.

Indicates <.01 proportion correct.

Part 2: Ability to Label Sets of Items as Dimension-Like

Two-dimensional structures with shorter tests

In addition to examining the performance of the methods in terms of accurate counting of dimensions, the performance of the methods was examined by their ability to label sets of items as dimension-like. Recall that the purpose of this evaluation was to go beyond the assessment of the number of dimensions present. The purpose here was to evaluate, regardless of the selected solution, how often grouped items can be thought of in some meaningful way as dimensions.³ Using the previously specified criteria, marginal proportions were calculated for each method across the 500 replications. In Table 5, marginal proportions of labeling (both) two sets of items as dimension-like are reported (complete results for labeling any one set of items as dimension-like may be obtained by contacting the author).

Table 5.

Marginal Proportion of Labeling Both Sets of Items as Dimension-Like for 2D Conditions in Short Tests.

		Complexity Level
		0%					10%					30%					50%
ρ	N	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV
.00	500	.89	.92	.32	1.0	.99	.86	.88	.24	1.0	.98	.26	.29	.04	.56	.51	.01	.01	.03	.04	.09
	1,000	.91	.90	.35	1.0	1.0	.86	.86	.25	1.0	1.0	.31	.32	.06	.54	.56	.01	.02	.01	.02	.06
	2,000	.83	.80	.39	1.0	1.0	.81	.77	.28	1.0	1.0	.30	.29	.08	.47	.53	.02	.02	.02	.00	.03
.30	500	.87	.93	.29	1.0	.93	.84	.87	.18	1.0	.91	.25	.27	.03	.33	.29	.01	.01	.02	.09	.09
	1,000	.89	.88	.39	1.0	.99	.82	.82	.21	1.0	.99	.32	.32	.04	.23	.37	.02	.03	.03	.04	.08
	2,000	.81	.79	.35	1.0	1.0	.67	.62	.20	1.0	1.0	.28	.27	.04	.19	.31	.05	.04	.01	.01	.06
.60	500	.75	.79	.17	.97	.73	.67	.68	.10	.94	.59	.06	.08	.01	.11	.17	.01	.01	.02	.11	.07
	1,000	.80	.82	.21	1.0	.90	.64	.59	.09	.99	.77	.11	.10	.02	.06	.14	.05	.05	.02	.11	.11
	2,000	.67	.62	.15	1.0	.99	.31	.26	.09	1.0	.97	.09	.06	.01	.05	.12	.08	.08	.02	.09	.13
.75	500	.42	.45	.07	.78	.48	.27	.30	.05	.70	.30	.02	.02	.00	.05	.09	.00	.00	.01	.14	.04
	1,000	.63	.63	.11	.97	.66	.30	.28	.04	.85	.49	.01	.01	^b	.03	.06	.02	.02	.01	.18	.09
	2,000	.43	.37	.07	1.0	.86	.11	.04	.01	.97	.71	^b	^b	.00	^b	.02	.06	.06	.01	.24	.17
.90	500	.04	.04	.00	.25	.22	.02	.01	^b	.12	.14	^b	^b	.00	.01	.03	.00	^b	.01	.11	.04
	1,000	.05	.04	.01	.30	.24	^b	.00	^b	.13	.14	.00	.00	.00	^b	.02	^b	^b	^b	.15	.09
	2,000	.03	.02	.00	.20	.26	.00	.00	.00	.09	.16	.00	.00	.00	^b	.01	.00	.00	^b	.12	.10

Note. 2D = two-dimensional; ALR = approximate likelihood ratio; NOHARM = normal ogive harmonic analysis robust method; DETECT = dimensionality evaluation to enumerate contributing traits; RMSR = root mean square residual. R stands for NOHARM-based RMSR; D_E stands for DETECT exploratory; D_CV stands for DETECT cross-validated.

Indicates actual zero.

Indicates <.01 proportion.

It was observed that the DETECT-based methods, in particular DETECT_E, had more success in labeling the two sets of items as dimension-like compared with their NOHARM-based counterparts across all conditions. RMSR method did not perform well with respect to labeling both sets of items in any condition.

When the data exhibited 30% or more of complexity, none of the methods was very successful in labeling the two sets of items as dimension-like. In these conditions, out of all of the methods, DETECT_E was the most successful in labeling both sets of items as dimension-like, in particular with lower levels of complexity and correlations of at most .75.

Generally, an increase in correlations between dimensions yielded lower marginal proportions of labeling both sets of items as dimension-like for all complexity levels. Additional, unreported results showed that as the complexity in the data increased, the methods’ ability to label any one set of items as dimension-like also increased. In other words, in conditions with complexity of 30% and 50%, the methods were generally very successful in identifying any one set of items. This was found in conditions across different levels of correlation and sample size.

Impact of lengthening the test

In 2D structures, the increase in the number of items had varying impact on methods’ ability to label the sets of items as dimension-like. As it can be seen in Table 6, the DETECT-based methods tended to be the least affected by the increase in test length and remained the most successful procedures to label both sets of items as dimension-like across conditions with 30% or less complexity, in particular when correlations were ≤.60. In conditions with 50% complexity, none of the five methods was able to label both sets of items as dimension-like.

Table 6.

Marginal Proportion of Labeling Both Sets of Items as Dimension-Like for 2D Conditions in Longer Tests.

		Complexity Level
		0%					10%					30%					50%
ρ	N	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV	ALR	$χ_{G / D}^{2}$	R	D _E	D _CV
.00	500	.60	.70	.96	1.0	1.0	.87	.67	.95	1.0	1.0	.70	.32	.64	.98	.85	^b	.01	^b	.02	.05
	1,000	.56	.58	.91	1.0	1.0	.91	.60	.95	1.0	1.0	.76	.19	.68	.99	.94	.03	.02	.03	^b	.02
	2,000	.49	.41	.76	1.0	1.0	.82	.30	.69	1.0	1.0	.67	.05	.43	1.0	.99	.06	^a	.01	^a	^b
.30	500	.97	.61	.96	1.0	1.0	.99	.59	.93	1.0	.99	.63	.24	.58	.84	.62	.02	.06	.05	^b	.05
	1,000	.91	.47	.90	1.0	1.0	.94	.44	.84	1.0	1.0	.74	.15	.47	.91	.81	.03	.05	.06	^a	.04
	2,000	.82	.28	.72	1.0	1.0	.82	.20	.49	1.0	1.0	.60	.02	.13	.93	.91	.05	.03	.03	^a	^b
.60	500	.88	.42	.90	1.0	.84	.85	.22	.69	1.0	.83	.11	.06	.18	.47	.28	^a	.11	.09	.03	.03
	1,000	.95	.32	.82	1.0	.98	.90	.12	.28	1.0	.99	.31	.01	.04	.58	.44	.02	.16	.17	^b	.04
	2,000	.69	.12	.44	1.0	1.0	.55	.04	.08	1.0	1.0	.28	^b	^b	.50	.54	.13	.13	.16	^b	.02
.75	500	.26	.20	.75	.96	.42	.21	.08	.38	.95	.44	^b	^b	.01	.13	.11	^a	.07	.05	.07	.01
	1,000	.65	.06	.55	1.0	.80	.55	.02	.04	1.0	.76	^b	^a	^a	.15	.15	^a	.13	.13	.05	.08
	2,000	.63	.02	.09	1.0	.98	.36	^b	.01	1.0	.97	^b	^a	^a	.14	.15	.03	.11	.13	.03	.08
.90	500	^a	^b	^b	.23	.09	^b	^b	^b	.23	.10	^a	^a	^a	.02	.03	^a	^b	^b	.05	^b
	1,000	^b	^a	^b	.42	.12	^a	^a	^a	.32	.17	^a	^a	^a	.01	.01	^a	^a	^a	.14	.03
	2,000	^b	^a	^a	.66	.28	^a	^a	^a	.40	.24	^a	^a	^a	^b	.01	^a	^b	^b	.19	.08

Indicates actual zero.

Indicates <.01 proportion.

The increase in test length had dissimilar effects on the NOHARM-based methods. RMSR benefited greatly from the increase in test length: the most out of all five methods. The positive effect was observed for almost all conditions; however, as the complexity and correlations increased, the impact was smaller in magnitude. The directionality of the impact (positive/negative) varied across the other two NOHARM-based methods. For $χ_{G / D}^{2}$ , the increase in test length almost always led to a large decrease in marginal proportions across the studied conditions, in particular when the sample size was large. Only a couple of exceptions were noted in conditions where $χ_{G / D}^{2}$ yielded very low marginal proportions in both shorter and longer tests, such as in 50% complexity conditions.

ALR, however, had differential impact from the test length increase. For example, in conditions with no complexity and no correlation, the increase in the test length had a negative impact on the procedure’s performance. As the complexity levels or correlations increased, lengthening of the test tended to positively affect the ALR’s performance. Lastly, it was observed that the five procedures studied were not successful in labeling all three sets of items as dimension-like in conditions with .90 correlation, except for DETECT_E, which in conditions with no complexity yielded rates of .42 and .66 for medium and large sample size, respectively.

Three-Dimensional Structures With Shorter and Longer Tests

Although marginal proportions for labeling sets of items as dimension-like in 2D conditions were high for the NOHARM-based methods under certain conditions (see Tables 5 and 6), the NOHARM-based methods were not successful in identifying the sets of items as dimension-like that would correspond to all three dimensions across any complexity levels, sample size, or correlations.

For ALR, $χ_{G / D}^{2}$ , and RMSR, the marginal proportions correct for identifying all three sets of items as dimension-like were close to zero across most of the studied conditions; the six highest rates in any of the methods were obtained in conditions with 10% complexity and no correlation across sample sizes—marginal proportions to label all three sets of items as dimension-like ranged between .12 and .24. Outside these six instances, all other proportions were essentially zero across shorter and longer tests across all studied conditions for the three NOHARM-based methods. Overall, the NOHARM-based methods were generally more able to label any two or any one set of item as dimension-like. For this reason, only the results in labeling all three sets of items as dimension-like for the two DETECT-based methods are reported in Table 7.

Table 7.

The Marginal Proportion of Labeling All Three Sets of Items as Dimension-Like in 3D Conditions for DETECT-Based Methods in Shorter (J = 10 per Dimension) and Longer (J = 20 per Dimension) Tests.

		0% Complexity				10% Complexity				30% Complexity				50% Complexity
		DETECT_E		DETECT_CV		DETECT_E		DETECT_CV		DETECT_E		DETECT_CV		DETECT_E		DETECT_CV
ρ	N/J	10	20	10	20	10	20	10	20	10	20	10	20	10	20	10	20
.00	500	.71	.86	.34	.65	.62	.86	.18	.55	.12	.30	.01	.02	.08	.06	^b	^a
	1000	.63	.85	.57	.82	.72	.88	.48	.85	.22	.43	.07	.13	.25	.21	.02	.02
	2000	.52	.83	.71	.81	.66	.89	.73	.87	.22	.43	.18	.40	.49	.34	.15	.13
.30	500	.52	.86	.15	.28	.33	.81	.06	.18	.07	.18	^b	^b	.05	.03	^a	^a
	1000	.59	.84	.36	.75	.50	.85	.18	.58	.13	.42	.02	.05	.24	.21	.02	^a
	2000	.43	.67	.54	.83	.57	.78	.41	.83	.17	.41	.07	.28	.41	.36	.07	.07
.60	500	.19	.42	.06	.04	.06	.24	.02	.01	.01	.03	^b	^a	.03	.01	^b	^a
	1000	.32	.75	.12	.15	.15	.55	.03	.10	.03	.17	.01	^b	.09	.14	.01	^b
	2000	.37	.62	.19	.56	.24	.60	.07	.32	.06	.31	.01	.04	.22	.52	.06	.03
.75	500	.06	.06	.03	^b	.02	.03	^b	^a	^b	^b	^a	^a	.01	^a	^a	^a
	1000	.08	.26	.04	.02	.03	.09	.01	.01	.01	.03	^a	^a	.04	.07	^a	^a
	2000	.12	.51	.07	.10	.03	.22	.02	.04	.01	.09	.01	.01	.06	.29	.02	.01
.90	500	^b	.01	.01	^a	^a	^a	^b	^a	^b	^b	^a	^a	^a	^a	^a	^a
	1000	.01	^a	.01	^b	^a	^a	^b	^a	^b	^a	^a	^a	.01	^b	^a	^a
	2000	.03	.01	.01	^a	^b	^b	^b	^a	^a	^a	^b	^b	.02	.04	.02	^a

Note. 3D = three-dimensional; ALR = approximate likelihood ratio; DETECT = dimensionality evaluation to enumerate contributing traits; RMSR = root mean square residual. DETECT_E stands for DETECT exploratory; DETECT_CV stands for DETECT cross-validated.

Indicates actual zero.

Indicates <.01 proportion.

As illustrated in Table 7, DETECT_E was generally more successful than DETECT_CV in labeling all three sets of items as dimension-like. Both methods were the most successful in conditions with zero and 10% complexity: when correlations among dimensions were .60 or less. As the complexity increased to 30% or higher, in conditions with small and medium sample sizes, the methods yielded low marginal proportions (rarely did the rates rise above .20 for either procedure). With respect to the test length, it was observed that the increase in the number of items tended to positively affect the methods’ performance. The effect was largely noted in conditions with a correlation with of at most .60 and a complexity of 0% or 10%.

Discussion

This study examined the performance of popular tools used in conducting dimensionality assessment of binary data where the underlying MIRT was noncompensatory and the data exhibited varied degrees of complexity. The methodological contribution of this study rests in that little is currently known about dimensionality assessment procedures under these types of conditions, although they indeed may reflect complex assessments. As it was observed in the current study, the use of DETECT- and NOHARM-based methods should be further studied prior to their application in situations that may resemble those of the present study, as only in a small number of instances the procedures performed at what in practice might be considered an acceptable performance.

For example, in 2D conditions, the DETECT-based methods seemed to have difficulty in correctly counting the number of dimensions in shorter tests (Table 1), but they were quite successful in appropriately grouping items together (Table 5). On the other hand, the NOHARM-based methods ALR and $χ_{G / D}^{2}$ performed well in counting the dimensions for 2D conditions with low/moderate correlations and small/moderate sample sizes (Table 1), although they had difficulty in finding the groups of items—especially as the complexity increased (Table 5). Similar conclusions can be drawn by examining the effects of lengthening the test, as methods were not uniformly affected by the increase in items. For instance, in conditions with lower correlations, the NOHARM-based method RMSR did not perform well in 2D shorter tests (Table 1); however, increase in the number of items positively affected its performance in both the counting of the correct number of dimensions (Table 2) and grouping of the items (Tables 5 and 6). Note, however, that this was not the case for 3D conditions. Furthermore, it is suggested that to more fully understand the performance of the methods, Tables 1 through 7 should be used concurrently, as they address related aspects of the methods’ performance.

Even though little is known about performance of dimensionality assessment procedures in situations where noncompensatory MIRT model underlies the data, recognizing the work conducted in compensatory MIRT, we are able to further understand how these procedures operate across a variety of conditions. As discussed above, Finch and Habing (2005) found that when 2D and 6D conditions were examined, in higher dimensional conditions, the increase in the number of items benefited the ALR, whereas DETECT’s performance deteriorated. Lengthening the test had a somewhat different effect in the current study, where in 3D conditions (as compared with 2D cases), the increase in the number of items had a negative effect on the proportion correct for ALR and $χ_{G / D}^{2}$ and a positive effect on RMSR.

Similarly, Gierl et al. (2006) found that DETECT performed quite satisfactorily when using a compensatory model and conditions with at most 30% complexity and at most .60 correlations. The results in this study were often different; DETECT-based methods generally performed poorly, with only a few exceptions where none of the methods accurately identified the correct dimensionality.⁴ Notice, however, that these types of comparison cannot be directly made as the generating MIRT models differed in the current and previous studies.

Although the current study contributes to the literature of dimensionality assessment by using the noncompensatory MIRT with complex data, it has certain limitations. Among those, the methods’ performance is only generalizable by the limits of the studied conditions. Future work should include other possible testing scenarios, such as the use of pseudo-guessing parameters and the analysis of other model structure types (e.g., unbalanced structures where the number of complex items varies across dimensions). In addition, this study only considered 2D and 3D conditions. As Finch and Habing (2005) found, the procedures may indeed perform differently in test structures with a large(r) number of dimensions. As our assessments become more complex and multidimensional, considerations of higher dimensions may be useful.

Furthermore, our understanding of the performance of any method in dimensionality assessment should be approached in a comprehensive manner. In the current study, an attempt was made to further illustrate how groups of items may be considered to meaningfully reflect a dimension. The criteria and choices in operationalization of these outcome variables were made with practical considerations; however, these should be further studied and potentially refined. Additionally, as future research further examines data with complex structure, a deeper examination of the identification and classification of complex items is warranted. This is especially possible with procedures such as NOHARM, which can capture the structural aspect of the test by providing estimates of loadings and other item parameters. Finally, the current study does not directly address the issue of what happens when the methods err. Given the incorrect identification of the dimensionality, methods may be more likely to under- or overfactor. Together with under- or overfactoring, future studies could investigate how procedures group and separate items, in similar fashion to what Finch and Habing examined in their 2005 study. This knowledge will provide a better understanding of how the methods behave, and should be considered for future research.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Ackerman

T. A.

(1989). Unidimensional IRT calibration of compensatory and noncompensatory multidimensional items. Applied Psychological Measurement, 13, 113-127.

Ackerman

T. A.

(1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7, 255-278.

Ansley

R. A.

Forsyth

T. N.

(1985). An examination of the characteristics of unidimensional IRT parameter estimates derived from two-dimensional data. Applied Psychological Measurement, 9, 37-48.

Camilli

Wang

Fesq

(1995). The effects of dimensionality on equating the law school admission test. Journal of Educational Measurement, 32, 79-96.

Chen

Thissen

(1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.

Childs

R. A.

Oppler

S. H.

(2000). Implications of test dimensionality for unidimensional IRT scoring: An investigation of a high-stakes testing program. Educational and Psychological Measurement, 60, 939-955.

De Champlain

Gessaroli

M. E.

(1998). Assessing the dimensionality of item response matrices with small sample sizes and short test lengths. Applied Measurement in Education, 11, 231-253.

Finch

Habing

(2003, April). Performance of DIMTEST and NOHARM based statistics for testing unidimensionality. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

Finch

Habing

(2005). Comparison of NOHARM and DETECT in item cluster recovery: Counting dimensions and allocating items. Journal of Educational Measurement, 42, 149-169.

10.

Finch

Habing

(2007). Performance of DIMTEST- and NOHARM-based statistics for testing unidimensionality. Applied Psychological Measurement, 31, 292-307.

11.

Fraser

McDonald

R. P.

(1988). NOHARM: Least squares item factor analysis. Multivariate Behavioral Research, 23, 267-269.

12.

Froelich

A. G.

Habing

(2008). Conditional covariance-based subtest selection for DIMTEST. Applied Psychological Measurement, 32, 138-155.

13.

Gessaroli

M. E.

De Champlain

A. F.

(1996). Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157-192.

14.

Gessaroli

M. E.

De Champlain

A. F.

Folske . (1997, March). Assessing dimensionality using a likelihood-ratio chi-square test based on a non-linear factor analysis of item response data. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

15.

Gierl

M. J.

Leighton

J. P.

Tan

(2006). Evaluating DETECT classification accuracy and consistency when data display complex structure. Journal of Educational Measurement, 43, 265-289.

16.

Hambleton

R. K.

Swaminathan

Rogers

H. J.

(1991). Fundamentals of item response theory. Newbury Park, CA: Sage.

17.

Hattie

Krakowski

Rogers

H. J.

Swaminathan

(1996). An assessment of Stout’s index of essential unidimensionality. Applied Psychological Measurement, 20, 1-14.

18.

Jang

E. E.

Roussos

(2007). An investigation into the dimensionality of TOEFL using conditional covariance-based nonparametric approach. Journal of Educational Measurement, 44, 1-22.

19.

Jasper

(2010). Applied dimensionality and test structure assessment with the START-M mathematics test. International Journal of Educational and Psychological Assessment, 6, 104-125.

20.

Kim

H. R.

(1994). New techniques for the dimensionality assessment of standardized test data (Doctoral dissertation, University of Illinois at Urbana-Champaign). Dissertation Abstracts International: Section B. Sciences and Engineering, 55(12), 5598.

21.

Levy

Svetina

(2010, May). A framework for characterizing dimensionality assessment and overview of current approaches. Paper presented at the annual meeting of the National Council on Measurement in Education, Denver, CO.

22.

Maydeu-Olivares

(2001). Multidimensional item response theory modeling of binary data: Large sample properties of NOHARM estimates. Journal of Educational and Behavioral Statistics, 26, 51-71.

23.

McDonald

R. P.

(2000). A basis for multidimensional item response theory. Applied Psychological Measurement, 24, 99-114.

24.

Nandakumar

(1991). Traditional dimensionality versus essential dimensionality. Journal of Educational Measurement, 28, 99-117.

25.

Nandakumar

(1993). Assessing essential unidimensionality of real data. Applied Psychological Measurement, 17, 29-38.

26.

Nandakumar

Stout

(1993). Reﬁnement of Stout’s procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18, 41-68.

27.

Nandakumar

(1996). Empirical validation of DIMTEST on nonnormal ability distributions. Journal of Educational Measurement, 33, 355-368.

28.

Nandakumar

Stout

(1998). Assessing unidimensionality of polytomous data. Applied Psychological Measurement, 22, 99-115.

29.

R Development Core Team (2010). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

30.

Reckase

M. D.

Carlson

J. E.

Ackerman

T. A.

Spray

J. A.

(1986, June). The interpretation of unidimensional IRT parameters when estimated from multidimensional data. Paper presented at the annual meeting of Psychometric Society, Toronto, Ontario, Canada.

31.

Reckase

Thompson

Nering

(1997, June). Identifying similar item content clusters on multiple test forms. In Miller

(Chair), High-dimensional simulation of item response data for CAT research. Symposium conducted at the annual meeting of the Psychometric Society, Gatlingburg, TN.

32.

Roussos

Stout

Marden

(1998). Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement, 35, 1-30.

33.

Seraphine

A. E.

(2000). The performance of DIMTEST when latent trait and item difficulty distributions differ. Applied Psychological Measurement, 24, 82-94.

34.

Stout

(1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.

35.

Stout

Habing

Douglas

Kim

H. R.

Roussos

Zhang

(1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20, 331-354.

36.

Svetina

Levy

(2012, April). Dimensionality in compensatory MIRT when complex structure exists: Evaluation of DETECT and NOHARM. Paper presented at the annual meeting of American Educational Research Association, Vancouver, British Columbia, Canada.

37.

Sympson

J. B.

(1978). A model for testing with multidimensional items. In Weiss

D. J.

(Ed.), Proceedings of the 1977 Computerized Adaptive Testing Conference (pp. 82-89). Minneapolis, MN: University of Minneapolis, Department of Psychology, Psychometric Methods Program.

38.

Tate

(2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203.

39.

van Abswoude

A. A.

van der Ark

Sijtsma

(2004). A comparative study of test data dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28, 3-24.

40.

Walker

C. M.

Beretvas

S. N.

(2003). Comparing multidimensional and unidimensional proficiency classifications: Multidimensional IRT as a diagnostic aid. Journal of Educational Measurement, 40, 255-275.

41.

Way

W. D.

Ansley

T. N.

Forsyth

R. A.

(1988). The comparative effects of compensatory and noncompensatory two-dimensional data on unidimensional IRT estimates. Applied Psychological Measurement, 12, 239-252.

42.

Yen

W. M.

(1985). Increasing item complexity: A possible cause of scale shrinkage for unidimensional item response theory. Psychometrika, 50, 399-410.

43.

Zhang

(2007). Conditional covariance theory and detect for polytomous items. Psychometrika, 72, 69-91.

44.

Zhang

Stout

(1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129-152.

45.

Zhang

Stout

(1999b). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.

Assessing Dimensionality of Noncompensatory Multidimensional Item Response Theory With Complex Structures

Abstract

Keywords

Introduction

Dimensionality Assessment

Overview of DETECT Procedure

Overview of NOHARM Procedure

Tate’s reduction in RMSR

Approximate χ G / D 2 (Gessaroli & De Champlain, 1996)

Approximate likelihood ratio (Gessaroli et al., 1997)

Research Using DETECT and NOHARM Procedures

Method

Methods to Evaluate Dimensionality

Determining the (optimal) number of dimensions

Study Design

Outcome Variables

The proportion of correct selection of true dimensionality

The ability to label sets of items as “dimension-like.”

Grouping of the items

Labeling of sets of items

Results

Part 1: Proportion Correct

Shorter tests with 10 items per dimension in 2d structures

Impact of lengthening the 2D test

Shorter tests with 10 items per dimension in 3d structures

Impact of lengthening the 3D test

Part 2: Ability to Label Sets of Items as Dimension-Like

Two-dimensional structures with shorter tests

Impact of lengthening the test

Three-Dimensional Structures With Shorter and Longer Tests

Discussion

Footnotes

Declaration of Conflicting Interests

Funding

Notes

References

Approximate $χ_{G / D}^{2}$ (Gessaroli & De Champlain, 1996)