Robustness of Hierarchical Modeling of Skill Association in Cognitive Diagnosis Models

Abstract

Several types of parameterizations of attribute correlations in cognitive diagnosis models use the reduced reparameterized unified model. The general approach presumes an unconstrained correlation matrix with K(K−1)/2 parameters, whereas the higher order approach postulates K parameters, imposing a unidimensional structure on the correlation matrix between the latent skills. This article investigates the differences in performance between the correlational structure parameterizations (a general structure, a higher order single-factor structure, and a baseline uniform distributional approach constraining the attributes to be independent) across a wide variety of simulated multidimensional attribute spaces. Results suggest that the correlational approaches perform equally well with respect to classification and item parameter estimation accuracy, regardless of the violations of the assumptions of the higher order model. Findings suggest the general robustness of the higher order model and the associated estimation procedure. The three approaches are also used to analyze a real-world test; results suggest that such tests can be analyzed effectively by the higher order algorithm.

Keywords

cognitive diagnosis correlational structure robustness classification-based assessment

Get full access to this article

View all access options for this article.

References

Barnard, J. , McCulloch, R. , & Meng, X.-L. (2000). Modeling covariance matrices in terms of standard deviations and correlations, with applications to shrinkage. Statistica Sinica, 10, 1281-1311.

Christoffersson, A. (1975). Factor analysis of dichotomized variables. Psychometrika, 40, 5-32.

de la Torre, J. , & Douglas, J.A. (2004). Higher order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.

DiBello, L.V. , Stout, W.F. , & Roussos, L.A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In P. D. Nichols , S. F. Chipman , & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361-392). Mahwah, NJ: Lawrence Erlbaum.

Educational Testing Service. (2004). Arpeggio: Release 3.1 [Computer software]. Princeton, NJ : Educational Testing Service.

Gelman, A. , & Rubin, D.B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457-511.

Haertel, E.H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.

Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign .

Henson, R. , & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29, 262-277.

10.

Junker, B.W. , & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.

11.

Macready, G.B. , & Dayton, C.M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99-120.

12.

Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 197-212.

13.

Patz, R.J. , & Junker, B.W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146-178.

14.

Tatsuoka, K.K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen , R. Glaser , A. Lesgold , & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 453-488). Mahwah, NJ: Lawrence Erlbaum.

15.

Templin, J. (2004). Generalized linear mixed proficiency models for cognitive diagnosis. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.

16.

Templin, J. , & Henson, R. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.