The purpose of this study was to explore whether the results of Tree Based Regression (TBR) analyses, informed by a validated cognitive model, would enhance the interpretation of item difficulties in terms of the cognitive processes involved in answering the reading items included in two forms of the Michigan English Language Assessment Battery (MELAB). A cognitive model was first generated to explain the performance of the MELAB reading items, and then validated by expert judgment and student verbal protocols. Next, the validated model was used in the TBR analyses to obtain the final trees for each form. Finally, the cognitive processes (i.e., reading processes and testing strategies) measured by each item were traced back for each item in the terminal nodes of each tree. The results revealed that TBR, informed by a supportable cognitive theory, appears to be a promising addition to statistical item analysis that can be effectively used to enhance the interpretation of item analyses results.
Alderson JC ( 2005) The challenge of diagnostic testing: Do we know what we are measuring?Plenary presented at the annual meeting of the Language Testing Research Colloquium (LTRC), Ottawa, Canada.
4.
Alderson JC, Lukmani Y. ( 1989) Cognition and reading: Cognitive levels as embodied in test questions. Reading in a Foreign Language, 5: 253-270.
5.
Anderson N., Bachman L., Perkins K., and Cohen A. ( 1991) An exploratory study into the construct validity of a reading comprehension test: Triangulation of data resources. Language Testing, 8: 41-66.
6.
Bachman LF ( 1990) Fundamental considerations in language testing. Oxford: Oxford University Press.
7.
Bachman LF ( 2002) Some reflections on task-based language performance assessment . Language Testing, 19: 453-476.
8.
Bachman LF, Davidson F., and Milanovic M. ( 1996) The use of test method characteristics in the content analysis and design of EFL proficiency tests. Language Testing , 13: 125-150.
9.
Bachman LF, Davidson F., Ryan K., and Choi I. ( 1995) An investigation into the comparability of two tests of English as a foreign language. Cambridge, UK: Cambridge University Press.
10.
Bernhardt E. ( 2003) Challenges to reading research from a multilingual world . Reading Research Quarterly, 38: 112-117.
11.
Breiman L., Friedman JH, Olshen RA, and Stone CJ ( 1984) Classification and regression trees. Belmont, CA: Wadsworth International.
12.
Brennan RL ( 2001) Generalizability theory. New York: Springer.
13.
Carr NT ( 2006) The factor structure of test task characteristics and examinee performance. Language Testing, 23(3): 269-289.
14.
Clapham C. ( 1996) The development of IELTS: A study of the effect of background knowledge on reading comprehension. Cambridge, UK: University of Cambridge Local Examinations Syndicate.
15.
Cohen AD, Upton TA ( 2006) Strategies in responding to the new TOEFL reading tasks . TOEFL Monograph Report No. MS-33.
16.
Drum PA, Calfee RC, and Cook LK ( 1981) The effects of surface structure variables on performance in reading comprehension tests. Reading Research Quarterly , 16: 486-514.
17.
Efron B., Tibshirani R. ( 1986) Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science , 1: 54-77.
18.
Embretson SE ( 1993) Psychometric models for learning and cognitive processes . In N. Frederiksen, R. J. Mislevy, and I. I. Bejar (Eds.), Test theory for a new generation of tests (pp. 125- 150). Hillsdale, NJ: Lawrence Erlbaum.
19.
Embretson SE, Gorin JS ( 2001) Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38: 343-368.
20.
English Language Institute. (2003) Michigan English Language Assessment Battery Technical Manual. MELAB Program, University of Michigan, Ann Arbor, MI.
21.
Enright MK, Grabe W., Koda D., Mosenthal P., Mulcahy-Ernt P., and Schedl M. ( 2000) TOEFL 2000 reading framework: A working paper ( TOEFL Monograph SeriesMS-17). Princeton, NJ: Educational Testing Service.
22.
Enright MK, Morley M., and Sheehan KM ( 2002) Items by design: The impact of systematic feature variation on item statistical characteristics. Applied Measurement in Education, 15: 49-74.
23.
Ericsson KA, Simon HA ( 1993) Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
24.
Ewing M., Huff K. ( 2004, April) Using item difficulty modeling to evaluate skill categories. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.
25.
Freedle R., Kostin I. ( 1993) The prediction of TOEFL reading comprehension item difficulty: Implications for construct validity. Language Testing, 10: 131-170.
26.
Gao L. ( 2002) Passage dependence of the reading comprehension component of the College English Test. Unpublished master’s thesis, Queen’s University, Kingston, Ontario, Canada.
27.
Gorin JS ( 2002) Cognitive and psychometric modeling of text-based reading comprehension GRE-V items. Unpublished PhD dissertation, University of Kansas, Department of Psychology, Kansas, USA.
28.
Gorin JSEmbretson EE ( 2006) Item difficulty modeling of paragraph comprehension items . Applied Psychological Measurement, 30(5): 394-411.
29.
Gorin JS, Rupp A. ( 2008) An HLM analysis of reading comprehension item difficulty factors. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
30.
Grabe W. ( 1991) Current development in second-language reading research . TESOL Quarterly, 25: 375-406.
31.
Grabe W. ( 2004) Research on teaching reading. Annual Review of Applied Linguistics, 24: 44-69.
32.
Hartz SM ( 2002) A Bayesian guide for the Unified Model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illonois, Urbana-Champaign .
33.
Hudson T. ( 1996) Assessing second language academic reading from a communicative competence perspective: Relevance for TOEFL 2000 (TOEFL Monograph SeriesMS-4). Princeton, NJ: Educational Testing Service.
34.
Huff K. ( 2003) An item modeling approach to providing descriptive score reports. Unpublished doctoral dissertation, University of MassachusettsAmherst.
35.
Jamieson J., Jones S., Kirsch I., Mosenthal P., and Taylor C. ( 2000) TOEFL 2000 framework: A working paper. TOEFL Monograph SeriesMS-16. Princeton, NJ: Educational Testing Service.
36.
Jang EE(2009) Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion Model application to LanguEdge assessment. Language Testing, 26: 31-73.
37.
Johnston P. ( 1983) Reading comprehension assessment: A cognitive basis. Newark, DE: International Reading Association.
38.
Kintsch W. ( 1998) Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.
39.
Kirsch IS, Mosenthal PB ( 1990) Exploring document literacy: Variables underlying the performance of young adults. Reading Research Quarterly, 25: 5-30.
40.
Koda K. ( 2005) Insights into second language reading: A cross-linguistic approach. New York: Cambridge University Press.
41.
Leighton JP ( 2004) Avoiding misconception, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing. Educational Measurement: Issues and Practice, 23(4): 6-15.
42.
Leighton JP, Gierl M. ( 2005) Identifying models of cognition in educational measurement . Paper presented at the annual meeting of the Canadian Society for the Study of Education, London, Ontario, Canada.
43.
Leighton JP, Gierl M. ( 2007) Cognitive diagnostic assessment for education: Theory and applications. Cambridge, UK: Cambridge University Press.
44.
Leighton JP, Gierl MJ, and Hunka SM ( 2004) The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41: 205-237.
45.
Lewis C., Sheehan KM ( 1990) Using Bayesian decision theory to design a computerized mastery test. Applied Psychological Measurement, 14: 367-386.
46.
Lord FM ( 1980) Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
47.
Messick S. ( 1995) Validity of psychological assessment: Validation of inferences from person’s responses and performances as scientific inquiry into score meaning. American Psychologist, 50: 741-749.
48.
Mislevy RJ ( 1994) Evidence and inference in educational assessment. Psychometrika, 59: 439-483.
49.
Mislevy RJ, Almond RG, Yan D., and Steinberg LS ( 1999) Bayes nets in educational assessment: Where do the numbers come from? In K. B. Laskey and H. Prade (Eds.), Proceedings of the Fifteenth Conference on uncertainty in artificial intelligence (pp. 437-446). San Francisco, CA: Morgan Kaufmann.
50.
National Research Council. (2001) Knowing what students know: The science and design of educational assessment . Washington, DC: National Academy Press.
51.
Phakiti A. ( 2003) A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading achievement test performance. Language Testing, 20: 26-56.
52.
Phakiti A. ( 2008) Construct validation of Bachman and Palmer’s (1996) strategic competence model over time in EFL reading tests. Language Testing, 25: 237-272.
53.
Phillips LM ( 1988) Young readers’ inference strategies in reading comprehension . Cognition and Instruction, 5: 193-222.
54.
Powers DE, Leung SW ( 1995) Answering the new SAT reading comprehension questions without the passages. Journal of Educational Measurement, 32: 105-129.
55.
Purpura JE ( 1999) Learner strategy use and performance on language tests: a structural equation modeling approach. Cambridge, UK: Cambridge University Press.
56.
Qian DD ( 1999) Assessing the roles of depth and breadth of vocabulary knowledge in reading comprehension. The Canadian Modern Language Review, 56: 282-307.
57.
Rogers WT, Bateson DJ ( 1991a) The influence of test-wiseness upon the performance of high school seniors on school leaving examinations. Applied Measurement in Education, 4: 159-183.
58.
Rogers WT, Bateson DJ ( 1991b) Verification of a model of test-taking behavior of high school seniors. Journal of Experimental Education, 59: 331-350.
59.
Rogers WT, Yang P. ( 1996) Test-wiseness: Its nature and application. European Journal of Psychological Assessment, 12: 247-259.
60.
Rupp AA, Ferne T., and Choi H. ( 2006) How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23: 441-474.
61.
Rupp AA, Garcia P., and Jamieson J. ( 2001) Combining multiple regression and CART to understand difficulty in second language reading and listening comprehension tests. International Journal of Testing, 1: 185-216.
62.
Sheehan K. ( 1997) A tree-based approach to proficiency scaling and diagnostic assessment. Journal of Educational Measurement, 34: 333-352.
63.
Sheehan K., and Ginther A. ( 2001) What do multiple choice verbal reasoning items really measure? An analysis of the cognitive skills underlying performance on a standardized test of reading comprehension skill. Paper presented at the annual meeting of the National Council on Measurement in Education , Seattle, WA.
64.
Shiotsu T., Weir CJ ( 2007) The relative significance of syntactic knowledge and vocabulary breadth in the prediction of reading comprehension test performance. Language Testing, 24(1): 99-128.
65.
Skehan P. ( 1998) A cognitive approach to language learning. Oxford: Oxford University Press.
66.
Skehan P., Foster P. ( 2001) Cognition and tasks. In P. Robinson (Eds.), Cognition and second language instruction (pp. 183-205) New York: Cambridge University Press.
67.
Spearman C. ( 1904) The proof and measurement of association between two things . American Journal of Psychology, 15: 72-101.
68.
Spss (2005) SPSS Version 13.0 [Computer Software]. Chicago, IL: SPSS.
Stanovich K. ( 2000) Progress in understanding reading: Scientific foundations and new frontiers. New York: Guilford Press.
71.
Tatsuoka KK ( 1995) Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman , and R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 327-359). Hillsdale, NJ: Lawrence Erlbaum.
72.
Urquhart AH, Weir CJ ( 1998) Reading in a second language: Process, product and practice . New York: Addison Wesley Longman .
73.
Wang X., Bradlow ET, and Wainer, H. ( 2002) A general Bayesian model for testlets: Theory and applications . Applied Psychological Measurement, 26: 109-128.
74.
Wang X., Bradlow ET, and Wainer H. ( 2004) User’s guide for SCORIGHT (version 3.0): A computer program for scoring tests built of testlets including a module for covariate analysis. Princeton, NJ: Educational Testing Service; Philadelphia, PA: National Board of Medical Examiners.
75.
Yohannes Y., Hoddinot J. ( 1999) Classification and regression trees: An introduction. Washington, DC: International Food Policy Research Institute.
