An Overview of Software for Conducting Dimensionality Assessment in Multidimensional Models

Abstract

An overview of popular software packages for conducting dimensionality assessment in multidimensional models is presented. Specifically, five popular software packages are described in terms of their capabilities to conduct dimensionality assessment with respect to the nature of analysis (exploratory or confirmatory), types of data (dichotomous, ordered polytomous, continuous, missingness), technical details, and statistics used for dimensionality assessment. Following descriptions of existing software packages, several promising potentially broadly applicable approaches are described that have been proposed but are not yet implemented in widely available software.

Keywords

dimensionality assessment multidimensional models current and emerging approaches

The development and emergence of applications of multidimensional item response theory (MIRT) models that specify multiple latent variables (e.g., Ackerman, 1996; Adams, Wilson, & Wang, 1997; Béguin & Glas, 2001; Bolt & Lall, 2003; Embretson, 1997; McDonald, 1997; Reckase, 1997; Walker & Beretvas, 2003; Yao & Boughton, 2007) bring with it the need for supporting data-model fit procedures in such contexts. Included among the areas of data-model fit for MIRT models is dimensionality analysis. Many choices exist with respect to procedures and software for dimensionality assessment of item response data. In this article, the authors review five popular software packages for dimensionality assessment that have established guidelines for use in terms of evaluating test statistics or referencing recommended cutoff values. Each software package is described with regard to the features that can be handled within the standard options of the software. Specifically, each software package is described in terms of its capabilities to conduct dimensionality assessment with respect to the

nature of the analysis, where at the exploratory end of the spectrum the number of latent variables is not specified, and at the confirmatory end of the spectrum the number of latent variables and the pattern of dependence of item responses on the latent variables are specified;

types of data, including dichotomous, ordered polytomous, continuous, and missing data (assuming missing at random);

technical details, including estimation, rotation, and model restrictions and allowances (e.g., of a lower asymptote parameter for dichotomous data); and

statistics used for dimensionality assessment with a focus on their use for multidimensional modeling.

Following descriptions of existing software packages, several promising potentially broadly applicable approaches are described that have been proposed but are not yet implemented in widely available software.

Software Description

Mplus

Nature of Analysis

One of the most popular programs for latent variable modeling, Mplus (Muthén & Muthén, 1998-2010) can be used to conduct either exploratory or confirmatory factor analysis for dimensionality assessment.

Types of Data

Mplus is one of the most flexible software packages available to researchers with respect to the types of data. It can handle dichotomous, polytomous, and continuous data; various types of correlational matrices; and missing data.

Technical Details

Exploratory or confirmatory factor analysis may be conducted using least squares or maximum-likelihood-based estimators. In addition, Mplus contains options for orthogonal (Varimax) and oblique (Promax) rotations of exploratory factor solutions. Mplus does not allow for lower asymptote parameters in neither the computation of the tetrachoric (or polychoric) correlations nor in the estimation of the model parameters in an exploratory or confirmatory approach. If the correlation matrix is not positive definite (as a result from negative factor or residual variances, out of bound correlations, or linear dependency between factors), Mplus will issue a warning and a solution will not be produced.

Statistics for dimensionality assessment

In exploratory factor analysis, the user specifies the number of latent variables to be extracted. Relevant Mplus output for each exploratory dimensionality analysis includes eigenvalues from the tetrachoric or polychoric correlation matrix, the (rotated) solution, the residual correlation matrix, the root mean square residual (RMSR), a χ² statistics with associated degrees of freedom, and the root mean square error of approximation (RMSEA). Although Mplus does not determine the number of dimensions in exploratory mode directly, it can be used by researchers by specifying models of different number of dimensions. As Mplus outputs the previously mentioned relevant output for each requested solution, a user can determine dimensionality by applying certain criterion. For example, a user may wish to evaluate model fit between M and (M+ 1) factor solution, where M is the number of latent variables requested to be extracted. Then, via a χ² difference test, a better fitting model may be retained. Alternatively, a user may examine dimensionality by comparing eigenvalues produced by Mplus with those obtained via parallel analysis (Horn, 1965). In such situations, the number of retained factors equals the number of eigenvalues produced by Mplus that are greater than a mean of simulated random eigenvalues from parallel analysis (e.g., see Crawford et al., 2010; Glorfeld, 1995, and Zwick & Velicer, 1986, for descriptions of various approaches for parallel analysis).

The procedures and statistics produced in confirmatory analyses come primarily from those in structural equation modeling traditions for continuous, polytomous, or dichotomous data. For continuous data, researchers have developed an overall model χ² test statistic (e.g., Bollen, 1989) and a number of model fit indices (e.g., Hu & Bentler, 1999). For discrete data, scaling corrections are available for the usual χ² statistics using likelihood or least squares estimation (Muthén & Muthén, 1998-2010; Satorra & Bentler, 1994). C. Yu and Muthén (2002) introduced a weighted root mean square residual (WRMR) for use with discrete data, suggesting that values <1 indicate adequate fit (Finney & DiStefano, 2006). At the local level, residual correlations or covariances serve to indicate the adequacy with which the model accounts for the item-pair associations. Mplus can produce all these indices based on estimating confirmatory models using least squares estimators based on tetrachoric and polychoric correlations or using full-information maximum likelihood techniques. Last, with the addition of Bayesian modeling via Markov chain Monte Carlo estimation in Mplus, users are able to examine dimensionality via posterior predictive model checking analyses (Gelman, Meng, & Stern, 1996) using likelihood ratio statistics.

TESTFACT

Nature of Analysis

The TESTFACT software package (Bock et al., 1999) can be used in either exploratory or confirmatory factor analysis. However, confirmatory factor analysis is limited to bifactor structures only.

Types of Data

TESTFACT supports analyses of polytomous and dichotomous data, permitting missingness, and proceeds by computing the tetrachoric correlation matrix. Because tetrachoric correlation matrices need not be positive definite, TESTFACT will produce a smoothed tetrachoric correlation matrix from which it extracts eigenvalues.

Technical Details

TESTFACT uses marginal maximum likelihood estimation procedure and supports Varimax and Promax rotations. The model supports the presence of lower asymptote parameters; however, they need to be supplied to the software, which requires that they be estimated outside the program (e.g., in BILOG; Mislevy & Bock, 1982).

Statistics for dimensionality assessment

In addition to eigenvalues, TESTFACT output yields a χ² statistic treated as a χ² variate with associated degrees of freedom. Exploratory dimensionality analysis can proceed in a model comparison framework by sequentially fitting models with additional latent variables and testing for the improvement in fit. Specifically, the test can be conducted by examining the difference in the χ² statistics from models with M and (M+ 1) latent variables and evaluating that difference in terms of (a) a central χ² distribution with degrees of freedom equal to the difference in the degrees of freedom for the two models or (b) twice the degrees of freedom (Haberman, 1977; Schilling & Bock, 2005). Similarly, a model comparison approach to determining the number of latent variables may proceed by fitting models with increasing number of latent variables and selecting the number of latent variables based on the model that yields the smallest value of likelihood-based information criteria (e.g., Akaike information criterion [AIC]; Akaike, 1987).

The Normal-Ogive Harmonic Analysis Robust Method (NOHARM)

Nature of Analysis

NOHARM, developed by McDonald (1962, 1967, 1981, 1997, 2000) and implemented in the program of the same name (Fraser & McDonald, 1988), uses a polynomial approximation to the normal-ogive MIRT model for either exploratory or confirmatory analysis.¹ When the exploratory option is selected, the user specifies the number of latent variables to be extracted (like in Mplus), and the pattern of coefficients is obtained. Unlike confirmatory analysis in TESTFACT, NOHARM is not limited to bifactor structures in its confirmatory analysis.

Types of Data

NOHARM supports dichotomous data only (i.e., not polytomous), and it does not allow for missing data. However, the user may input a product moment matrix, and therefore if missingness is present, an adjustment for missingness in estimating the product moment matrix may be used.

Technical Details

NOHARM uses least squares estimates based on the first- and second-order product moments, and, like TESTFACT and Mplus, it allows for Varimax and Promax rotations of the solution in exploratory approaches. Like TESTFACT, NOHARM allows for the input of lower asymptote parameters estimated elsewhere (e.g., BILOG).

Statistics for dimensionality assessment

Relevant NOHARM output for exploratory and confirmatory analyses of dimensionality includes the residual matrix for unique pairings of items, the sum of the squares of the residuals, RMSR, Tanaka’s goodness-of-fit index (GFI), as well as associated (rotated) factor loadings. In addition to the above-mentioned output, several statistics have been proposed that use the results of NOHARM, including $χ_{G / D}^{2}$ (Gessaroli & De Champlain, 1996), which uses the residual correlations, and may be calculated using the software CHIDIM (De Champlain & Tang, 1997) and an approximate likelihood ratio (ALR) statistic (Gessaroli, De Champlain, & Folske, 1997) based on likelihood ratio statistics for bivariate response patterns.

Several exploratory approaches to determine the optimal number of latent variables have been proposed. Finch and Habing (2005) suggested using tests of $χ_{G / D}^{2}$ (or ALR) in a sequential manner, fitting a series of models specifying between 1 and M latent variables and examining the difference in $χ_{G / D}^{2}$ (or ALR) for models specifying different numbers of latent variables. The resulting difference is referred to a χ² variate with degrees of freedom equal to the difference in the number of estimated parameters between the two models. Tate (2003) suggested a sequential model fitting approach until the change in RMSR does not exceed 10%.

To evaluate model fit in confirmatory analysis, Fraser and McDonald (1988) proposed that if the RMSR is of the order of four divided by the square root of sample size, then a test of significance would not reject the hypothesized model. In addition, to conduct a test of the fit of a NOHARM model, $χ_{G / D}^{2}$ and ALR may be used (Gessaroli & De Champlain, 1996) referring to a central χ² distribution with appropriate degrees of freedom (see, for example, Finch & Habing, 2005, 2007), although there is reason to doubt the adequacy of the assumed χ² distribution in small sample sizes (Maydeu-Olivares, 2001).

Dimensionality Evaluation to Enumerate Contributing Traits (DETECT)

Nature of Analysis

The DETECT procedure and software (H. R. Kim, 1994; Stout et al., 1996; Zhang, 2007; Zhang & Stout, 1999b) is primarily an exploratory technique that seeks to identify dimensionally homogeneous clusters of items, thereby characterizing the amount of multidimensional approximate simple structure present in the data.² Conceptually, DETECT searches for a partitioning of the items into clusters where within-cluster conditional covariances are positive and between-cluster conditional covariances are negative.

Although mostly applied in its exploratory mode, a confirmatory DETECT analysis can be run by specifying the desired partition rather than searching for the optimal partition; characteristics and capabilities of exploratory DETECT resemble those when DETECT is used in a confirmatory mode.

Types of Data

DETECT can only handle dichotomously scored data and does not support missing data (see Zhang, 2007, on theory for supporting missingness). The procedure has been extended theoretically, and software has been produced to handle polytomous data (F. Yu & Nandakumar, 2001; Zhang, 2007) and missingness (Zhang, 2007). However, the software packages supporting these analyses are not available to the public.

Technical Details

DETECT does not fit a model per se; rather, as a nonparametric procedure, DETECT conditions on a number-correct-score-based estimate of the dimension of best measurement for the total test (Zhang & Stout, 1999a, 1999b; see Zhang, 2007, for the use of other conditioning variables supporting missing data) and searches for clusters of homogeneous items.

Statistics for dimensionality assessment

DETECT provides several relevant pieces of information for exploring multidimensionality. First, DETECT provides an estimate of the amount of multidimensionality that may be operationalized via an analysis of all the data (yielding ${\hat{D}}_{max}$ ) or via a cross-validation approach (yielding ${\hat{D}}_{ref}$ ). See Zhang and Stout (1999b), H. R. Kim (1994), and Roussos and Ozbek (2006) for recommendations on interpreting the magnitudes of these values.

If multidimensionality is deemed present, the tenability of the assumption of approximate simple structure may be evaluated by considering the percentage of the signs of the conditional covariances that achieve the goal of having all within-cluster conditional covariances positive and all between-cluster signs be negative (approximate simple structure index, reported in DETECT as the IDN index). Alternatively, the ratio (R) of ${\hat{D}}_{max}$ to the theoretical maximum of the DETECT index may be considered. In both cases, a value equal to 1 indicates that approximate simple structure holds formally; in practice, values of R greater than or equal to .8 are interpreted as indicative of approximate simple structure (Jang & Roussos, 2007).

If the hypothesis of approximate simple structure is supported, the solution may be interpreted in terms of the number of homogeneous item clusters as the number of dominant latent variables and the assignment of items to those clusters. To the extent that there are clusters with few items or if approximate simple structure does not hold, inferring the number of dominant latent variables should be done with caution (Jang & Roussos, 2007).

DIMTEST

Nature of Analysis

The DIMTEST program (Stout, Douglas, Junker, & Roussos, 1993) implements Stout’s (1987, 1990) test for essential unidimensionality: a confirmatory approach to hypothesis testing of a model of one latent variable.

Types of Data

DIMTEST, like DETECT, can only handle dichotomously scored data and does not allow for missingness. Extensions to polytomous data have been implemented in software (Li, Habing, & Roussos, 2010, 2011; Nandakumar, Yu, Li, & Stout, 1998); however, the software is not currently commercially available.

Technical Details

Although DIMTEST was explicitly built for testing assumptions of unidimensionality, Stout et al. (1996) proposed a straightforward approach to assess assumed multidimensional simple structure by using the assumed groupings of items to define the partitioning test (PT) and the assessment test (AT; see Stout et al., 1996, for full example and application). DIMTEST allows for the inputting of a single estimate of a lower asymptote parameter applied to all items.

Statistics for dimensionality assessment

As a nonparametric procedure, DIMTEST develops DIMTEST statistic by aggregating the conditional covariances among a set of suspect items, AT, conditional on the remaining items, PT. The items constituting the AT may be declared by the user or may be chosen by an exploratory process in DIMTEST.

Under the null hypothesis of essential unidimensionality, the resulting statistic is approximately distributed as a standard normal variable. In finite tests, the statistic is biased; current versions of DIMTEST correct for this bias by generating unidimensional data using a nonparametric model based on the original data (Stout, Froelich, & Gao, 2001). Currently, a more sophisticated version of DIMTEST is being developed, which could improve the subset selection for dichotomous and polytomous versions of DIMTEST (Li, 2011; Li et al., 2010, 2011).

Create-Your-Own Software

There are a number of other promising dimensionality assessment procedures that have not yet been implemented in widely available software; researchers wishing to use them are left to do their own programming. For example, parallel analysis (Horn, 1965) has been shown to be one of the most effective tools for conducting exploratory dimensionality analysis in factor analysis of continuous data (e.g., Buja & Eyuboglu, 1992; Glorfeld, 1995; Velicer, Eaton, & Fava, 2000; Zwick & Velicer, 1986). To date, only a few studies have examined the performance of parallel analysis in discrete data (e.g., Cheng & Weng, 2005; Tran & Formann, 2009). Importantly, software for conducting parallel analysis based on the results of exploratory factor analysis (e.g., O’Connor, 2000; Watkins, 2006) has focused on continuous data. Researchers wishing to implement parallel analysis for discrete data may need to write their own software or program the desired computations in a general statistical computing environment.

As another example, a researcher may consider the body of work surrounding local dependence indices, such as Yen’s (1984) Q₃, Reckase’s (1997) model-based covariance, odds ratios and their logarithmic transformations and standardized values (Chen & Thissen, 1997), and χ² and G² statistics drawn from contingency table analyses (Agresti, 2002); see Ip (2001), Chen and Thissen (1997), and Levy, Mislevy, and Sinharay (2009) for descriptions and studies of these and related indices. The use of these indices for evaluating dimensionality trades on the connections between the assumptions of local independence and dimensionality (Ip, 2001; Levy & Svetina, 2011; Nandakumar & Ackerman, 2004). They have been used for confirmatory approaches to dimensionality assessment, almost exclusively in assessing unidimensionality (Chen & Thissen, 1997; Levy, 2011; Levy et al., 2009). Levy and Svetina (2011) examined several of these in the context of multidimensional models, and building off of them they developed a generalized dimensionality discrepancy measure for use in confirmatory dimensionality assessment when fitting multidimensional models. These indices and approaches hold promise in that they appear flexible enough to handle dichotomous, polytomous, and continuous data; missingness; complex as well as (approximate) simple structures; and models with lower asymptote parameters for dichotomously scored items.

Despite this promise, and in some cases a long-standing popularity of these indices (e.g., Q₃), they are not included in any widely available software. Rather, analysts are left to do the programming on their own via stand-alone software targeting specific applications (Chen, 1998; S.-H. Kim, Cohen, & Lin, 2006) or by writing their own code in a general statistical computing environment (Levy & Svetina, 2011), none of which is yet widely available to the mainstream research community. An additional complication is the necessary integration with additional software that conducts model estimation. This is not the case with Mplus, TESTFACT, and NOHARM, which do estimation within the software, and DETECT and DIMTEST that do not require estimation but use an observed score approximation to a single latent dimension.

Summary

The purpose of this article was to describe popular and available software and procedures for the assessment of multidimensionality that may be useful to researchers and practitioners across disciplines. These procedures have been successfully applied, and some simulation studies have been conducted that focus on certain aspects or situations (e.g., Finch & Habing, 2005, 2007; Levy & Svetina, 2011; Svetina, 2011); however, more simulation studies comparing them are warranted, particularly given the increased use of multidimensional psychometric models and the relative lack of work evidencing the procedures’ relative strengths and weaknesses.

In addition, it is important to note that each of the programs has advantages and disadvantages associated with it. In some programs or procedures determining the number of dimensions may seem as a rather straightforward process. For example, in DETECT, a researcher may infer the number of dimensions as the number of nonoverlapping clusters outputted by DETECT. In other programs, such as Mplus, TESTFACT, or NOHARM, the user must specify the number of factors in fitting the model and make further investigation to determine the optimal solution. In other words, there is no direct count of dimensions of the response data at hand. Rather, a researcher must first determine the number of factors to fit in an exploratory factor analysis. This could mean that a researcher would fit a single-, two-, and three-factor exploratory models in NOHARM. Then, using some criterion (e.g., $χ_{G / D}^{2}$ ), a researcher would determine via sequential model fitting which factor solution is optimal (e.g., Finch & Habing, 2007).

Finally, although the purpose of this article is not to make specific recommendations about which method or software should be used in particular situations (see Jasper, 2010; Levy & Svetina, 2010; Nandakumar & Ackerman, 2004), the authors emphasize that dimensionality assessment involves a heavy involvement on the part of the researcher regardless of the procedures used, including those that output the count of optimal number of dimensions. In evaluating the dimensionality of the item responses at hand, researchers should carefully examine any solution provided directly or indirectly from a procedure.

Footnotes

Acknowledgements

The authors would like to thank the editor and two anonymous reviewers for their useful comments and suggestions regarding earlier drafts of the manuscript.

Authors’ Note

The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work of the second author was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305D100021 to Arizona State University.

Notes

References

Ackerman

(1996). Graphical representation of multidimensional item response theory model. Applied Psychological Measurement, 20, 311-329.

Adams

R. J.

Wilson

Wang

(1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.

Agresti

(2002). Categorical data analysis (2nd ed.). New York, NY: Wiley.

Akaike

(1987). Factor analysis and AIC. Psychometrika, 3, 317-332.

Béguin

A. A.

Glas

C. A. W.

(2001). MCMC estimation and some model-fit analysis of multidimensional IRT models. Psychometrika, 66, 541-562.

Bock

R. D.

Gibbons

Schilling

S. G.

Muraki

Wilson

D. T.

Wood

(1999). TESTFACT 3: Test scoring, items statistics, and full-information item factor analysis. Chicago, IL: Scientific Software International.

Bollen

K. A.

(1989). Structural equations with latent variables. New York, NY: Wiley.

Bolt

D. M.

Lall

V. F.

(2003). Estimation of compensatory and noncompensatory multidimensional item response models using Markov chain Monte Carlo. Applied Psychological Measurement, 27, 395-414.

Buja

Eyuboglu

(1992). Remarks on parallel analysis. Multivariate Behavioral Research, 27, 509-540.

10.

Chen

(1998). IRTNEW: A computer program for the detection of LID [Computer software]. Chapel Hill: Thurstone Laboratory, University of North Carolina at Chapel Hill.

11.

Chen

Thissen

(1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.

12.

Cheng

C.-P.

Weng

L.-J.

(2005). Parallel analysis with unidimensional binary data. Educational and Psychological Measurement, 65, 697-716.

13.

Crawford

A. V.

Green

S. B.

Levy

W. J.

Scott

Svetina

Thompson

M. S.

(2010). Evaluation of parallel analysis methods for determining the number of factors. Educational and Psychological Measurement, 70, 885-901.

14.

De Champlain

A. F.

Tang

K. L.

(1997). CHIDIM: A FORTRAN program for assessing the dimensionality of binary item responses based on McDonald’s nonlinear factor analytic model. Educational and Psychological Measurement, 57, 174-178.

15.

Embretson

S. E.

(1997). Multicomponent response models. In van der Linden

Hambleton

(Eds.), Handbook of modern item response theory (pp. 305-321). New York, NY: Springer-Verlag.

16.

Finch

Habing

(2005). Comparison of NOHARM and DETECT in item cluster recovery: Counting dimensions and allocating items. Journal of Educational Measurement, 42, 149-169.

17.

Finch

Habing

(2007). Performance of DIMTEST- and NOHARM-based statistics for testing unidimensionality. Applied Psychological Measurement, 31, 292-307.

18.

Finney

S. J.

DiStefano

(2006). Nonnormal and categorical data in structural equation models. In Hancock

G. R.

Mueller

R. O.

(Eds.), A second course in structural equation modeling (pp. 269-314). Greenwich, CT: Information Age.

19.

Fraser

McDonald

R. P.

(1988). NOHARM: Least squares item factor analysis. Multivariate Behavioral Research, 23, 267-269.

20.

Gelman

Meng

X. L.

Stern

(1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6, 733-807.

21.

Gessaroli

M. E.

De Champlain

A. F.

(1996). Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157-192.

22.

Gessaroli

M. E.

De Champlain

A. F.

Folske (1997, March). Assessing dimensionality using a likelihood-ratio chi-square test based on a non-linear factor analysis of item response data. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

23.

Glorfeld

L. W.

(1995). An improvement on Horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55, 377-393.

24.

Haberman

S. J.

(1977). Log-linear models and frequency tables with small expected cell counts. Annals of Statistics, 5, 1148-1169.

25.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185.

26.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

27.

E. H.

(2001). Testing for local dependency in dichotomous and polytomous item response models. Psychometrika, 66, 109-132.

28.

Jang

E. E.

Roussos

L. A.

(2007). An investigation into the dimensionality of TOEFL using conditional covariance-based nonparametric approach. Journal of Educational Measurement, 44, 1-22.

29.

Jasper

(2010). Applied dimensionality and test structure assessment with the START-M mathematics test. International Journal of Educational and Psychological Assessment, 6(1), 104-125.

30.

Kim

H. R.

(1994). New techniques for the dimensionality assessment of standardized test data (Doctoral dissertation, University of Illinois at Urbana-Champaign). Dissertation Abstracts International: Section, B, 5598, 12-55.

31.

Kim

S.-H.

Cohen

A. S.

Lin

Y.-H.

(2006). LDIP: A computer program for local dependence indices for polytomous items. Applied Psychological Measurement, 30, 509-510.

32.

Levy

(2011). Posterior predictive model checking for conjunctive multidimensionality in item response theory. Journal of Educational and Behavioral Statistics, 36, 672-694.

33.

Levy

Mislevy

R. J.

Sinharay

(2009). Posterior predictive model checking for multidimensionality in item response theory. Applied Psychological Measurement, 33, 519-537.

34.

Levy

Svetina

(2010, May). A Framework for characterizing dimensionality assessment and overview of current approaches. Paper presented at the annual meeting of the National Council on Measurement in Education, Denver, CO.

35.

Levy

Svetina

(2011). A generalized dimensionality discrepancy measure for dimensionality assessment in multidimensional item response theory. British Journal of Mathematical and Statistical Psychology, 64, 208-232.

36.

(2011). Assessing the dimensionality of polytomous item exams (Doctoral dissertation, University of South Carolina). Retrieved from: http://search.proquest.com/pqdt/docview/914419153/138065FE5B4614FD5CE/1?accountid=11620

37.

Habing

Roussos

L. A.

(2010, May). Conditional covariance-based subset selection for polytomous item DIMTEST. Paper presented at the annual meeting of the National Council on Measurement in Education, Denver, CO.

38.

Habing

Roussos

L. A.

(2011, April). Improved conditional covariance-based subtest selection for polytomous item DIMTEST. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

39.

Maydeu-Olivares

(2001). Multidimensional item response theory modeling of binary data: Large sample properties of NOHARM estimates. Journal of Educational and Behavioral Statistics, 26, 49-69.

40.

McDonald

R. P.

(1962). A general approach to nonlinear factor analysis. Psychometrika, 27, 398-415.

41.

McDonald

R. P.

(1967). Nonlinear factor analysis [Psychometric Monographs, No. 15]. Chicago: University of Chicago Press.

42.

McDonald

R. P.

(1981). The dimensionality of tests and items. British Journal of Mathematical and Statistical Psychology, 34, 100-117.

43.

McDonald

R. P.

(1997). Normal-ogive multidimensional model. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 257-269). New York, NY: Springer-Verlag.

44.

McDonald

R. P.

(2000). A basis for multidimensional item response theory. Applied Psychological Measurement, 24, 99-114.

45.

Mislevy

R. J.

Bock

R. D.

(1982). BILOG: Item analysis and test scoring with binary logistic models. Mooresville, IN: Scientific Software.

46.

Muthén

L. K.

Muthén

B. O.

(1998-2010). Mplus User’s Guide (6th ed.). Los Angeles, CA: Author.

47.

Nandakumar

Ackerman

T. A.

(2004). Test modeling. In Kaplan

(Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 93-105). Thousand Oaks, CA: Sage.

48.

Nandakumar

H.-H.

Stout (1998). Assessing unidimensionality of polytomous data. Applied Psychological Measurement, 22, 99-115.

49.

O’Connor

B. P.

(2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instrumentation, & Computers, 32, 396-402.

50.

Reckase

M. D.

(1997). A linear logistic multidimensional model. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 271-286). New York, NY: Springer-Verlag.

51.

Roussos

L. A.

Ozbek

O. Y.

(2006). Formulation of the DETECT population parameter and evaluation of DETECT estimator bias. Journal of Educational Measurement, 43, 215-243.

52.

Satorra

Bentler

P. M.

(1994). Corrections to test statistics and standard errors in covariance structure analysis. In von Eye

Clogg

C. C.

(Eds.), Latent variables analysis: Applications for developmental research (pp. 399-419). Thousand Oaks, CA: SAGE.

53.

Schilling

Bock

R. D.

(2005). High-dimensional maximum marginal likelihood item factor analysis by adaptive quadrature. Psychometrika, 70, 533-555.

54.

Stout

(1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.

55.

Stout

(1990). A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation. Psychometrika, 55, 293-325.

56.

Stout

Douglas

Junker

Roussos

L. A.

(1993). DIMTEST manual. Urbana-Champaign: University of Illinois at Urbana-Champaign, Department of Statistics.

57.

Stout

Froelich

A. G.

Gao

(2001). Using resampling methods to produce an improved DIMTEST procedure. In Boomsma

van Duijn

M. A. J.

Snijders

T. A. B.

(Eds.), Essays on item response theory (pp. 357-375). New York, NY: Springer-Verlag.

58.

Stout

Habing

Douglas

Kim

H. R.

Roussos

L. A.

Zhang

(1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20, 331-354.

59.

Svetina

(2011). Assessing dimensionality in complex data structures: A performance comparison of DETECT and NOHARM procedures (Doctoral dissertation, Arizona State University). Retrieved from http://repository.asu.edu/attachments/56763/content

60.

Tate

(2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203.

61.

Tran

U. S.

Formann

A. K.

(2009). Performance of parallel analysis in retrieving unidimensionality in the presence of binary data. Educational and Psychological Measurement, 69, 50-61.

62.

Velicer

W. F.

Eaton

C. A.

Fava

J. L.

(2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In Goffin

R. D.

Helmes

(Eds.), Problems and solutions in human assessment: Honoring Douglas Jackson at seventy (pp. 41-71). Boston, MA: Kluwer.

63.

Walker

C. M.

Beretvas

S. N.

(2003). Comparing multidimensional and unidimensional proficiency classifications: Multidimensional IRT as a diagnostic aid. Journal of Educational Measurement, 40, 255-275.

64.

Watkins

M. W.

(2006). Determining parallel analysis criteria. Journal of Modern Applied Statistical Methods, 5, 344-346.

65.

Yao

Boughton

K. A.

(2007). A multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Applied Psychological Measurement, 31, 83-105.

66.

Yen

(1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 21, 93-111.

67.

Muthén

B. O.

(2002, April). Evaluation of model fit indices for latent variable models with categorical and continuous outcomes. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

68.

Nandakumar

(2001). Poly-DETECT for quantifying the degree of multidimensionality of item response data. Journal of Educational Measurement, 38, 99-120.

69.

Zhang

(2007). Conditional covariance theory and DETECT for polytomous items. Psychometrika, 72, 69-91.

70.

Zhang

Stout

(1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129-152.

71.

Zhang

Stout

(1999b). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.

72.

Zwick

W. R.

Velicer

W. F.

(1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442.