The purpose of the present study was to examine the effect of response format—open-ended (OE) versus multiple-choice (Me)—on the diagnosis of examinee misconceptions in a procedural task. A test in fraction addition arithmetic was administered to 285 eighth- grade students, 148 of whom responded to the OE ver sion of the test and 137 to the MC version. The two datasets were compared with respect to the underlying structure of the test, the number of different error types, and the diagnosed sources of misconception (bugs) reflected in the response patterns. The overall results indicated considerable differences between the two formats, with more favorable results for the OE format.
Get full access to this article
View all access options for this article.
References
1.
Baillie, R., & Tatsuoka, K.K. (1983). SPBUG: A computer program for diagnosing bugs and analyzing responses. Urbana IL : University of Illinois, Computer-Based Education Research Laboratory.
2.
Bender, T.A. (1980, April). Processing multiple choice and recall test questions. Paper presented at the annual meeting of the American Educational Research Association, Boston . (ERIC ED189160)
3.
Birenbaum, M., & Shaw, D.J. (1985). Task specification chart—a key to a better understanding of test results. Journal of Educational Measurement , 22, 219-230.
4.
Birenbaum, M., & Tatsuoka, K.K. (1982). On the dimensionality of achievement test data. Journal of Educational Measurement , 19, 259-266.
5.
Birenbaum, M., & Tatsuoka, K.K. (1983). The effect of scoring based on algorithms underlying the students' response patterns on the dimensionality of achievement test data of the problem solving type. Journal of Educational Measurement, 20, 17-26.
6.
Brown, J.S., & Burton, R.R. (1978). Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science, 2, 155-192.
7.
Burton, R.R. (1981). Diagnostic bugs in a simple procedural skill. Palo Alto CA: Xerox Palo Alto Research Center.
8.
Cook, D.L. (1955). An investigation of three aspects of free response and choice type tests at the college level. Dissertation Abstracts International, 15, 1351 (University Microfilms No. A55-1791).
9.
Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
10.
Duchastel, P.C., & Nungester, R. (1982). Testing effects measured with alternate forms . Journal of Educational Research, 75, 309-314.
11.
Estes, W.K., & DaPolito, F.J. (1967). Independent variation of information storage and retrieval processes in paired-associate learning . Journal of Experimental Psychology, 75, 18-26.
12.
Fisher, K.M., & Lipson, J.H. (1985). Information processing interpretation of errors in college science learning. Instructional Science, 14, 49-74.
13.
Freund, R.D., Brelsford, J.W., Jr., & Atkinson, R.C. (1969). Recognition vs. recall: Storage or retrieval differences? Quarterly Journal of Experimental Psychology, 21, 214-224.
14.
Heim, A.W., & Watts, K.P. (1967). An experiment on multiple-choice versus open-ended answering in a vocabulary test. British Journal of Educational Psychology, 37, 339-346.
15.
Klein, M., Birenbaum, M., Standiford, S.N., & Tatsuoka, K.K. (1981). On the construction of an error-diagnosing test in fraction arithmetic (Technical Report 81-6-NIE) . Urbana IL: University of Illinois, Computer-Based Education Research Laboratory.
16.
Kumar, V.K., Rabinsky, L., & Pandey, T.N. (1979). Test mode, test instructions, and retention. Contemporary Educational Psychology, 4, 211-218.
17.
Lingoes, J.C. (1972). The Guttman Lingoes nonmetric program series. Ann Arbor MI: Matesis Press.
18.
Loftus, G.R. (1971). Comparison of recognition and recall in a continuous memory task. Journal of Experimental Psychology, 91, 220-226.
19.
Loftus, G.R., & Loftus, E.G. (1976). Human memory. The processing of information. Hillsdale NJ: Erlbaum.
20.
Marshall, S.P. (1980). Procedural networks and production systems in adaptive diagnosis. Instructional Science, 9, 129-143.
21.
Matz, M. (1980). Towards a computational theory of algebraic competence . Journal of Mathematical Behavior, 3, 93-166.
22.
Merwin, J.C., & Womer, F.B. (1969). Evaluation in assessing the progress of education to provide bases of public understanding and public policy. In R. W. Tyler (Ed.), Educational evaluation: New roles, new means—The sixty-eighth yearbook of the National Society for the Study of Education (Part II, pp. 305-334). Chicago: University of Chicago Press.
23.
Nie, N.H., Hull, C.H., Jenkins, J.C., Steinbrenner, K.S., & Brent, D.H. (1975). Statistical package for the social sciences (SPSS). New York: McGraw-Hill.
24.
Schlesinger, I.M., & Guttman, L. (1969). Smallest space analysis of intelligence and achievement tests. Psychological Bulletin,71, 95-100.
25.
Tatsuoka, K.K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement , 20, 345-354.
26.
Tatsuoka, K. K. (Ed.) (1984a). Analysis of errors in fraction addition and subtraction problems (Final Report for Grant No. NIE-G-81-0002). Urbana IL: University of Illinois, Computer-Based Education Research Laboratory.
27.
Tatsuoka, K.K. (1984b). Caution indices based on item response theory . Psychometrika, 49, 95-110.
28.
Tatsuoka, K.K. (1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational Statistics, 10, 55-73.
29.
Tatsuoka, K.K. (1986a). Diagnosing cognitive errors: Statistical pattern classification based on item response theory. Behaviormetrika , 19, 73-86.
30.
Tatsuoka, K.K. (1986b, July). Toward an integration of item response theory and cognitive diagnosis. Paper presented at the ONR Conference on Diagnostic Monitoring of Skill and Knowledge Acquisition , Princeton NJ.
31.
Tatsuoka, K.K., & Chevalaz, M.C. (1984). A map representation of misconceptions in the rule space: Fraction addition arithmetic (Research Report 84-2-NIE). Urbana IL: University of Illinois, Computer-Based Education Research Laboratory.
32.
Tatsuoka, K.K., & Tatsuoka, M.M. (1987). Bug distribution and statistical pattern classification. Psychometrika, 52, 193-206.
33.
Traub, R.E., & Fisher, C.W. (1977). On the equivalence of constructed-response and multiple-choice tests. Applied Psychological Measurement, 1, 355-369.
34.
Tversky, B. (1973). Encoding processes in recognition and recall. Cognitive Psychology, 5, 275-287.
35.
Van Lehn, K. (1981). Bugs are not enough: Empirical studies of bugs, impasses and repairs in procedural skills (Technical Report CIS-11). Palo Alto CA: Xerox, Palo Alto Research Center.
36.
Ward, W.C. (1982). A comparison of free-response and multiple-choice forms of verbal aptitude tests. Applied Psychological Measurement , 6, 1-11.
37.
Ward, W.C., Frederiksen, N., & Carlson, S.B. (1980). Construct validity of free response and machine-scorable forms of a test. Journal of Educational Measurement, 17, 11-29.
38.
White, K.R., & Carcelli, L. (1982, March). The effect of item format on students' standardized mathematics achievement test scores. Paper presented at the annual meeting of the American Educational Research Association, New York. (ERIC ED219425)
