A Comparison of Approaches for Improving the Reliability of Objective Level Scores

Abstract

This study is an evaluation of the psychometric issues associated with estimating objective level scores, often referred to as ‘‘subscores.’’ The article begins by introducing the concepts of reliability and validity for subscores from statewide achievement tests. These issues are discussed with reference to popular scaling techniques, classical test theory, and item response theory. Methods for increasing the reliability of subscore estimates that have been suggested in literature are then reviewed. Based on this review, an empirical study comparing some of the more promising procedures was conducted. Test score data from a large statewide testing program were analyzed in this study. The comparison of subscore augmentation approaches found that generally all methods were very successful in dramatically increasing the reliability of subscore estimates. However, this increase was accompanied by near-perfect correlations among the subscore estimates. This finding called into question the validity of the resultant subscores, and therefore the usefulness of the subscore augmentation process. Implications for practice are discussed.

Keywords

objective level scores subscore augmentation reliability

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, National Council on Measurement in Education. ( 1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association .

Crocker, L. , & Algina, J. ( 1986). Introduction to classical and modern test theory. New York CBS College.

Edwards, M.C. , & Vevea, J.L. ( 2006). An empirical Bayes approach to subscore augmentation: How much strength can we borrow? Journal of Educational and Behavioral Statistics, 31, 241-259.

Goodman, D.P. , & Hambleton, R.K. (2004). Student test score reports and interpretive guides: Review of current practices and suggestions for future research. Applied Measurement in Education, 17, 145-220.

Green, D.R. , Yen, W.M. , & Burket, G.R. ( 1989). Experiences in the application of item response theory in test construction. Applied Measurement in Education, 2, 297-312.

Hambleton, R.K. , & Swaminathan, H. (1985). Item response theory principles and applications. Boston: Kluwer-Nijhoff .

Kelley, T.L. ( 1927). The interpretation of educational measurements. New York: World Book.

Kelley, T.L. ( 1947). Fundamentals of statistics. Cambridge, MA: Harvard University Press.

Lord, F.M. ( 1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

10.

Luecht, R.M. ( 2003, April). Applications of multidimensional diagnostic scoring for certification and licensure tests. Paper presented at the annual meeting of the National Council on Measurement in Education , Chicago, IL.

11.

Messick, S. ( 1989). Validity. In R. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Washington, DC: American Council on Education.

12.

Skorupski, W.P. ( 2008). SUBSKOR [Computer Software]. Lawrence : University of Kansas.

13.

U.S. Department of Education. (2002). No Child Left Behind Act of 2001 (Pub. L. No. 107-110, §1111, 115 Stat. 1449-1452) (2002). Washington, DC: Author.

14.

Wainer, H. , Sheehan, K.M. , & Wang, X. ( 2000). Some paths towards making Praxis scores more useful. Journal of Educational Measurement, 37, 113-140.

15.

Wainer, H. , Vevea, J.L. , Camacho, F. , Reeve, B.B. , Rosa, K. , Nelson, L. , et al. (2001). Augmented score-‘‘borrowing strength’’ to compute scores based on small numbers of items. In D. Thissen & H. Wainer (Eds.), Test scoring (pp. 343-387). Mahwah, NJ: Lawrence Erlbaum.

16.

Yen, W.M. ( 1987, June). A Bayesian/IRT index of objective performance. Paper presented at the meeting of the Psychometric Society, Montreal, CA.

17.

Zimowski, M.F. , Muraki, E. , Mislevy, R.J. , & Bock, R.D. ( 2002). BILOG-MG[Computer software]. Chicago : Scientific Software International.