Recent research examining racial differences on standardized cognitive tests has focused on the impact of test item difficulty. Studies using data from the SAT and GRE have reported a correlation between item difficulty and differential item functioning (DIF) such that minority test takers are less likely than majority test takers to respond correctly to easy test items. The statistical techniques used and the effect sizes reported in these studies have been heavily criticized. This study addresses these criticisms by examining the relationship between item difficulty and DIF by using alternative statistical techniques based on item response theory and a different standardized test. The results replicate previous research and provide support for the generalizability of the findings.
Baker, F.B. (1997). EQUATE 2.1: Computer program for equating two metrics in item response theory [Computer software]. Madison : University of Wisconsin, Laboratory of Experimental Design.
2.
Bock, R.D., Gibbons, R., & Muraki, E. (1988). Full-information item factor analysis. Applied Psychological Measurement, 12, 261-280.
3.
Burton, E., & Burton, N. (1993). The effect of item screening on test scores and test characteristics. In P. Holland & H. Wainer (Eds.), Differential item functioning (pp. 321-335). Hillsdale, NJ: Lawrence Erlbaum.
4.
Camara, W., & Sathy, V. (2004, April). College Board response to Harvard Educational Review article by Freedle. Retrieved on September 1, 2005, from http://www.collegeboard.com/research/pdf/051425Harvard_050406.pdf
5.
Camilli, G., & Shepard, L. (1994). Methods for identifying biased test items. Thousand Oaks, CA: Sage.
6.
DeMars, C. (2003, April). Missing data and IRT item parameter estimation . Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL .
Dorans, N., & Kulick, E. (1986). Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the Scholastic Aptitude Test. Journal of Educational Measurement, 23, 355-368.
9.
Dorans, N.J., & Zeller, K. (2003). Examining Freedle's claims about bias and his proposed solution: Dated data, inappropriate measurement, and incorrect and unfair scoring (ETS RR-04-26). Princeton, NJ: Educational Testing Service.
10.
Drasgow, F., Levine, M.V., Tsien, S., Williams, B., & Mead, A. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement , 19, 143-165.
11.
Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
12.
Freedle, R., & Kostin, I. (1988). Relationship between item characteristics and an index of differential item functioning (DIF) for the four GRE verbal item types (ETS RR-88-29). Princeton, NJ: Educational Testing Service.
13.
Freedle, R., & Kostin, I. (1990). Item difficulty of four verbal item types and an index of differential functioning for Black and White examinees. Journal of Educational Measurement, 27, 329-343.
14.
Freedle, R., & Kostin, I. (1992). The prediction of GRE reading comprehension item difficulty for expository prose passages for each of three item types: Main ideas, inferences and explicit statements (GRE Board Professional Report 87-10P; ETS RR-91-59). Princeton, NJ: Educational Testing Service.
15.
Freedle, R., & Kostin, I. (1997). Predicting Black and White differential item functioning in verbal analogy performance. Intelligence, 24, 417-444.
16.
Freedle, R., Kostin, I., & Schwartz, L. (1987). A comparison of strategies used by Black and White students in solving SAT verbal analogies using a thinking aloud method and a match percentage-correct design (ETS RR-87-48). Princeton, NJ: Educational Testing Service.
17.
Freedle, R.O. (2003). Correcting the SAT's ethnic and social-class bias: A method for reestimating SAT scores. Harvard Educational Review, 73, 1-44.
18.
Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
19.
Hattrup, K., Schmitt, N., & Landis, R.S. (1992). Equivalence of constructs measured by job-specific and commercially available aptitude tests. Journal of Applied Psychology, 77, 298-308.
20.
Holland, P., & Wainer, H. (Eds.). (1993). Differential item functioning . Hillsdale, NJ: Lawrence Erlbaum .
21.
Hough, L.M., Oswald, F.L., & Ployhart, R.E. (2001). Determinants, detection and amelioration of adverse impact in personnel selection procedures: Issues, evidence and lessons learned. International Journal of Selection and Assessment, 9, 152-194.
22.
International Association for the Evaluation of Educational Achievement (IEA). (1999). IEA CIVIC EDUCATION STUDY, 1999 [UNITED STATES] (ICPSR version) [Electronic file]. Washington, DC: U.S. Dept. of Education, National Center for Education Statistics [Producer], 1999; Ann Arbor, MI: Inter-University Consortium for Political and Social Research [Distributor], 2004.
23.
Johanson, G.A. (1992). IRTDATA: An interactive or batch Pascal program for generating logistic item response data. Applied Psychological Measurement, 16, 52.
24.
Kulick, E., & Hu, P.G. (1989). Examining the relationship between differential item functioning and item difficulty (College Board Report No. 89-5; ETS RR-89-18) . New York: College Entrance Examination Board.
O'Neill, K., & McPeek, W. (1993). Item and test characteristics that are associated with differential item functioning. In P. Holland & H. Wainer (Eds.), Differential item functioning (pp. 255-276). Mahwah, NJ: Lawrence Erlbaum.
27.
Raju, N.S. (1998). DFIT4: A Fortran program for calculating DIF/DTF [Computer software]. Chicago: Illinois Institute of Technology.
28.
Raju, N.S., van der Linden, W., & Fleer, P.F. (1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353-368.
29.
Roth, P.L., Bevier, C.A., Bobko, P., Switzer, F.S.I., & Tyler, P. (2001). Ethnic group differences in cognitive ability in employment and educational settings: A meta-analysis. Personnel Psychology, 54, 297-330.
30.
Sackett, P., Schmitt, N., Ellingson, J., & Kabin, M. (2001). High-stakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative-action world. American Psychologist, 56, 302-318.
31.
Shepard, L., Camilli, G., & Williams, D. (1984). Accounting for statistical artifacts in item bias research. Journal of Educational Statistics, 9, 93-128.
32.
Stocking, M.L., & Lord, F.M. (1983). Developing a common metric in item response theory . Applied Psychological Measurement, 7, 201-210.
Wainer, H., & Skorupski, W.P. (2005). Was it ethnic and social-class bias or statistical artifact? Logical and empirical evidence against Freedle's method for reestimating SAT scores. Chance, 18, 17-24.
35.
Wilson, D.T., Wood, R., & Gibbons, R. (1991). TESTFACT: Test scoring, item statistics, and item factor analysis [Computer software]. Chicago: Scientific Software.
