Bias of Exploratory and Cross-Validated DETECT Index Under Unidimensionality

Abstract

DETECT is a nonparametric ``full'' dimensionality assessment procedure that clusters dichotomously scored items into dimensions and provides a DETECT index of magnitude of multidimensionality. Four factors (test length, sample size, item response theory [IRT] model, and DETECT index) were manipulated in a Monte Carlo study of bias, standard error, and root mean square error (RMSE) under the condition of unidimensionality. Bias, standard error, and RMSE of both DETECT indices increased as test length and sample size decreased. Results suggest that the cross-validated index should always be preferred over the exploratory index, even for 100 examinees and five items. Bias, standard error, and RMSE may be problematic for both indices under certain conditions of small samples or short tests. A Monte Carlo procedure could be built into DETECT to estimate and adjust for potential bias.

Keywords

Index terms: dimensionality DETECT DIMTEST Monte Carlo bias.

Get full access to this article

View all access options for this article.

References

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 395-479). Menlo Park, CA : Addison-Wesley.

Douglas, J. , Kim, H.-R. , Roussos, L. , Stout, W. , & Zhang, J. (1999). LSAT dimensionality analysis for the December 1991, June 1992, and October 1992 administrations (Statistical Rep. No. 95-05) . Iowa City, IA: Law School Admission Council.

duToit, M. (Ed.). (2003). IRT from SSI: BILOG-MG, MULTILOG, PARSCALE, TESTFACT. Lincolnwood, IL: Scientific Software International.

Finch, H. , & Habing, B. (2005). Comparison of NOHARM and DETECT in item cluster recovery: Counting dimensions and allocating items. Journal of Educational Measurement, 42(2), 149-169.

Kim, H.R. (1994). New techniques for the dimensionality assessment of standardized test data. (Doctoral dissertation, University of Illinois at Urbana-Champaign). Dissertation Abstracts International , 55-12B, 5598.

Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

Mroch, A.A. , & Bolt, D.M. (2003, April). A Monte Carlo comparison of parametric and nonparametric dimensionality detection procedures. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.

Nandakumar, R. (1991). Traditional dimensionality versus essential dimensionality . Journal of Educational Measurement, 28(2), 99-117.

Nandakumar, R. , & Stout, W. (1993). Refinements of Stout's procedure for assessing latent trait dimensionality. Journal of Educational Statistics , 18(1), 41-68.

10.

Roussos, L.A. , & Ozbek, O. (in press). Formulation of the DETECT population parameter and evaluation of DETECT estimator bias. Journal of Educational Measurement .

11.

Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52(4), 589-617.

12.

Stout, W. , Froelich, A.G. , & Gao, F. (2001). Using resampling methods to produce an improved DIMTEST procedure. In A. Boomsma , M. A. J. van Duijn , & T. A. B. Snijders (Eds.), Essays on item response theory (pp. 357-375). New York: Springer-Verlag.

13.

Stout, W. , Habing, B. , Douglas, J. , Kim, H.R. , Roussos, L. , & Zhang, J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20(4), 331-354.

14.

Stout, W.F. (1990). A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation. Psychometrika, 55, 293-325.

15.

Zhang, J. (1996, April). A theoretical index of dimensionality and its estimation. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.

16.

Zhang, J. , & Stout, W. (1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64(2), 129-152.

17.

Zhang, J. , & Stout, W. (1999b). The theoretical detect index of dimensionality and its application to approximate simple structure. Psychometrika , 64(2), 213-249.

18.

Zhang, Y.O. , Yu, F. , & Nandakumar, R. (2003, April). The impact of conditional scores on the performance of DETECT. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.