This article reports a study that investigated the effects of Q-matrix misspecifications on parameter estimates and misclassification rates for the deterministic-input, noisy ``and'' gate (DINA) model, which is a restricted latent class model for multiple classifications of respondents that can be useful for cognitively motivated diagnostic assessment. In this study, a Q-matrix for an assessment mapping all 15 possible attribute patterns based on four independent attributes was misspecified by changing one ``0'' or ``1'' for each item. This was done in a way that ensured that certain attribute combinations were completely deleted from the Q-matrix, and certain incorrect dependency relationships between attributes were represented. Results showed clear effects that included an itemspecific overestimation of slipping parameters when attributes were deleted from the Q-matrix, an item-specific overestimation of guessing parameters when attributes were added to the Q-matrix, and high misclassification rates for attribute classes that contained attribute combinations that were deleted from the Q-matrix.
Bock, R.D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Applications of an E-M algorithm. Psychometrika , 46, 443-459.
2.
Bolt, D., & Fu, J. (2004, April). A polytomous extension of the fusion model and its Bayesian parameter estimation. Paper presented at the annual meeting of the National Council on Measurement in Education , San Diego, CA.
3.
Borsboom, D., Mellenbergh, G.J., & van Heerden, J. (2004). The concept of validity . Psychological Review,111, 1061-1071.
4.
Christofferson, A. (1975). Factor analysis of dichotomized variables. Psychometrika, 40, 5-32.
5.
de la Torre, J., & Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.
6.
Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
7.
Gierl, M.J., Leighton, J.P., & Hunka, S.M. (in press). Using the attribute hierarchy method to make diagnostic inferences about examinees' cognitive skills. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and practice . Cambridge, UK: Cambridge University Press.
8.
Haertel, E.H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.
9.
Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, Department of Statistics, University of Illinois, Urbana-Champaign.
10.
Henson, R.A., & Templin, J.L. (2006). Implications of Q-matrix misspecification in cognitive diagnosis. Manuscript submitted for publication.
11.
Junker, B.W. (1999). Some statistical models and computational methods that may be useful for cognitively-relevant assessment. Retrieved March 26, 2007, from www.stat.cmu.edu/∼brian/nrc/cfa
12.
Junker, B.W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
13.
Macready, G.B., & Dayton, C.M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99-120.
14.
Maris, E. (1995). Psychometric latent response models. Psychometrika, 60, 523-547.
15.
Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.
16.
Messick, S. (1989). Validity. In R. Linn (Ed.), Educational measurement (pp. 13-103). New York: Macmillan.
17.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
18.
Muthén, L.K., & Muthén, B.O. (2006). Mplus Version 4.1 [Computer software]. Los Angeles: Author.
19.
Nichols, P. D., Chipman, S. F., & Brennan, R. L. (Eds.). (1995). Cognitively diagnostic assessment. Hillsdale, NJ: Lawrence Erlbaum.
20.
Rupp, A.A. (in press). The answer is in the question: A guide for describing and investigating the conceptual foundations and statistical properties of cognitive psychometric models. International Journal of Testing.
21.
Rupp, A.A., & Mislevy, R.J. (in press). Cognitive psychology as it applies to diagnostic assessment . In J. Leighton (Ed.), Cognitive diagnostic assessment in education: Theory and practice. Cambridge, UK: Cambridge University Press.
22.
Tatsuoka, K.K. (1983). Rule-space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement , 20, 345-354.
23.
Tatsuoka, K.K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 327-360). Hillsdale, NJ: Lawrence Erlbaum.
24.
Templin, J.L. (2006). Generalized linear mixed proficiency models for cognitive diagnosis. Manuscript submitted for publication.
25.
Templin, J.L., & Henson, R.A. (2005). The random effects reparametrized unified model: A model for joint estimation of discrete skills and continuous ability. Manuscript submitted for publication.
26.
Templin, J.L., & Henson, R.A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.
27.
Templin, J.L., Roussos, L., & Stout, W. (2003). An extension of the current fusion model to treat polytomous attributes. Unpublished Technical Report for the ETS External Diagnostic Group. Urbana-Champaign: University of Illinois at Urbana-Champaign.
28.
van der Linden, W.J., & Hambleton, R.K. (1997). Handbook of modern item response theory. New York: Springer-Verlag .
29.
Williamson, D.M., Almond, R.G., Mislevy, R.J., & Levy, R. (2006). An application of Bayesian networks in automated scoring of computerized simulation tasks. In D. M. Williamson, R. J. Mislevy, & I. I. Bejar (Eds.), Automated scoring of complex tasks in computer-based testing (pp. 201-258). Mahwah, NJ: Lawrence Erlbaum.
30.
Xu, X., & von Davier, M. (2006). Cognitive diagnosis for NAEP proficiency data (Research Report No. RR-06-08). Princeton, NJ: Educational Testing Service.
