The current study evaluated the effectiveness of two-stage testing on English and French versions of a science achievement test administered to a national sample in Canada in 1996 and 1999. The tests were administered and scored with the implicit assumption that the two language forms were equivalent. Analysis of the first-stage test revealed that 3 out of 12 items displayed differential item functioning (DIF) in both administrations. However, substantive reviews suggested that translation errors were not the cause of DIF. Analysis of the second-stage test revealed that the test was not comparable between ability groups but was comparable for English and French examinees within each ability group in both administrations. This study illustrates how test developers can monitor their adaptation and administration process when alternative testing procedures are used with multiple language groups. The results are also relevant to cross-cultural researchers who compare examinees from different ethnic and cultural backgrounds.
Allalouf, A., Hambleton, R., & Sireci, S. (1999). Identifying the causes of translation DIF on verbal items. Journal of Educational Measurement, 36, 185-198.
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
3.
Angoff, W. H., & Cook, L. L. (1988). Equating the scores of the Prueba de Aptitud Academica and the Scholastic Aptitude Test(College Board Report No. 88-2). New York: College Entrance Examination Board.
4.
Bock, R. D., & Zimowski, M. F. (1989). Duplex design: Giving students a stake in educational assessment. Chicago: National Opinion Research Center.
5.
Bock, R. D., & Zimowski, M. F. (1998). Feasibility studies of two-stage testing in large-scale educational assessment: Implications for NAEP. Washington, DC: American Institutes for Research.
6.
Council of Ministers of Education, Canada. (2000). Report on science assessment, School Achievement Indicators Program, 1999. Toronto: Council of Ministers of Education, Canada.
7.
Ercikan, K.Gierl, M. J., McCreith, T., Puhan, G., & Koh, K. (2004). Comparability of bilingual versions of assessments: Sources of incomparability of English and French versions of Canada's national achievement tests. Applied Measurement in Education, 17(3), 301-321.
8.
Gierl, M. J., & Khaliq, S. N. (2001). Identifying sources of differential item and bundle functioning on translated achievement tests. Journal of Educational Measurement, 38, 164-187.
9.
Gierl, M. J., Rogers, W. T., & Klinger, D. (1999, April). Consistency between statistical procedures and content reviews for identifying translation DIF. Paper presented at the annual meeting of the National Council on Measurement in Education, Montréal, Canada.
10.
Grisay, A. (2002). Translation and cultural appropriateness of the test and survey material. In R. Adams & M. Wu (Eds.), PISA 2000 technical report(pp. 57-70). Paris: Organization for Economic Cooperation and Development.
11.
Hambleton, R. K. (1994). Guidelines for adapting educational and psychological tests: A progress report. European Journal of Psychological Assessment, 10, 229-224.
12.
Hambleton, R. K., & Patsula, L. (1998). Adapting tests for use in multiple languages and cultures. Social Indicators Research, 45, 153-171.
13.
Hambleton, R. K., & Patsula, L. (1999). Increasing the validity of adapted tests: Myths to be avoided and guidelines for improving test adaptation practices. Journal of Applied Testing Technology, 1, 1-30.
14.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
15.
Jeanrie, C., & Bertrand, R. (1999). Translating tests with the International Test Commission's guidelines: Keeping validity in mind. European Journal of Psychological Assessment, 15, 277-283.
16.
Lord, F. M. (1971). A theoretical study of two-stage testing. Psychometrika, 36, 227-242.
17.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
18.
Ndalichako, J. L., & Rogers, W. T. (1997). Comparison of finite state score theory, classical test theory, and item response theory in scoring multiple-choice items. Educational and Psychological Measurement, 57(4), 580-589.
19.
Puhan, G. (2003). Evaluating the effectiveness of two-stage testing for English and French examinees on the SAIP science 1996 and 1999 tests. Unpublished doctoral thesis, University of Alberta, Edmonton, Canada.
20.
Reckase, M. D., & Kunce, C. (1999, April). Translation accuracy of a technical credentialing examination. Paper presented at the annual meeting of the National Council on Measurement in Education, Montréal, Canada.
21.
Roussos, L. A., & Stout, W. F. (1996). Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenszel Type I error performance. Journal of Educational Measurement, 33, 215-230.
22.
Shealy, R., & Stout, W. F. (1993). A model-based standardization approach that separates true bias/DIF from group differences and detects test bias/DIF as well as item bias/DIF. Psychometrika, 58, 159-194.
23.
Sireci, S. G. (1997). Problems and issues in linking tests across languages. Educational Measurement: Issues and Practice, 16, 12-19.
24.
van de Vijver, F., & Leung, K. (1997). Methods and data analysis for cross-cultural research. Thousand Oaks, CA: Sage.
25.
Zieky, M. (1993). Practical questions in the use of DIF statistics in test development. In P. W. Holland & H. Wainer (Eds.), Differential item functioning(pp. 337-347). Hillsdale, NJ: Lawrence Erlbaum.
26.
Zimowski, M. F., Muraki, E., Mislevy, R. J., & Bock, R. D. (1996). BILOG-MG: Multiple-group IRT analysis and test maintenance for binary items. Chicago: Scientific Software International.
