Standard-setting methods are widely used to determine cut scores on a test that examinees must meet for a certain performance standard. Because standard setting is a measurement procedure, it is important to evaluate variability of cut scores resulting from the standard-setting process. Generalizability theory is used in this study to estimate standard errors of cut scores resulting from two standard-setting methods: item rating (Angoff-based) and mapmark (bookmark-based) methods. In this study, two different generalizability (G) study designs and four different decision (D) study designs were examined, and the impact of varying different aspects of the study design and universes of generalization was examined. Results suggest that cut scores were generally consistent for both methods. The first round standard setting contributed the most to the overall variability for the mapmark method. Also, it is clear that there is no one standard error associated with a certain cut score.
Angoff, W.H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.
2.
Berk, R.A. (1986). A consumer's guide to setting performance standards on criterion-referenced test. Review of Educational Research , 56, 137-172.
3.
Bock, R.D. (1997). A brief history of item response theory. Educational Measurement: Issues and Practice, 16(4), 21-33.
4.
Brennan, R.L. (1995). Standard setting from the perspective of generalizability theory. In Proceedings of the joint conference on standard setting for large-scale assessments (Vol. 2). Washington, DC: National Center for Educational Statistics and National Assessment Governing Board.
5.
Brennan, R.L. (2001a). Generalizability theory. New York: Springer-Verlag.
6.
Brennan, R.L. (2001b). Manual for urGENOVA. Iowa City: University of Iowa, Iowa Testing Programs.
7.
Brennan, R.L., & Lockwood, R.E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219-240.
8.
Cizek, G. J. (Eds.). (2001). Setting performance standards. Mahwah, NJ: Lawrence Erlbaum.
9.
Council of Chief State School Officers (2001 ). State student assessment programs annual survey. Data Volume II. Washington, DC: Author.
10.
Jaeger, R.M. (1989). Certification of student competence. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 485-514). Phoenix, AZ : Oryx Press.
11.
Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
12.
Kane, M., & Wilson, J. (1984). Errors of measurement and standard setting in mastery testing. Applied Psychological Measurement, 8, 425-461.
13.
Hambleton, R.K., & Eignor, D.R. (1980). Competency test development, validation and standard setting. In R. M. Jaeger & C. K. Tittle (Eds.), Minimum competency achievement testing: Motives, models, measures and consequences (pp. 367-396). Berkeley, CA: McCutchan.
14.
Hambleton, R., & Swaminathan, H. (1985). Item response theory: Principles and applications (pp. 61-69). Norwell, MA: Kluwer Academic Publishers.
15.
Lee, G., & Lewis, D.M. (2001, April). A generalizability theory approach toward estimating standard errors of cutscores set using the bookmark standard setting procedure. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
16.
Lewis, D.M., Mitzel, H.C., & Green, D.R. (1996, June). Standard setting: A bookmark approach. Paper presented at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.
17.
Loomis, S.C., & Bourque, M.L. (2001). From tradition to innovation: Standard setting on the National Assessment of Educational Progress. In G. J. Cizek (Ed.), Setting performance standards (pp. 249-281). Mahwah, NJ: Lawrence Erlbaum.
18.
Loomis, S.C., & Hanick, P.L. (2000). Setting achievement levels on the 1998 National Assessment of Educational Progress in civics: Pilot study final report. Iowa City, IA: ACT, Inc.
19.
Masters, G.N., Adams, R., & Lokan, J. (1994). Mapping student achievement. International Journal of Educational Research, 21, 595-609.
20.
Mitzel, H.C., Lewis, D.M., Patz, R.J., & Green, D.R. (2001). The bookmark procedure: Psychological perspectives . In G. J. Cizek (Ed.), Setting performance standards (pp. 249-281). Mahwah, NJ: Lawrence Erlbaum.
21.
Reckase, M.D. (2000). The evolution of the NAEP achievement levels setting process: A summary of the research and development efforts conducted byACT. Iowa City, IA: ACT, Inc .
22.
Schulz, E.M., Kolen, M., & Nicewander, W.A. (1999). A rationale for defining achievement levels using IRT-estimated domain scores. Applied Psychological Measurement, 23, 347-362.
23.
Schulz, E.M., Lee, W., & Mullen, K. (2005). A domain-level approach to describing growth in achievement. Journal of Educational Measurement, 42, 1-26.
24.
Schulz, E.M., & Mitzel, H. (2005, April). The mapmark standard setting method. Paper presented at the annual meeting of the National Council on Measurement and Education, Montreal, Canada.
25.
Shepard, L. (1980). Standard setting, issues, and methods. Applied Psychological Measurement, 4, 447-467.
26.
Surowiecki, J. (2004). The wisdom of crowds. New York : Doubleday.
27.
Williams, N.J., & Schulz, E.M. (2005, April). An investigation of response probability (RP) values used in standard setting. Paper presented at the annual meeting of the National Council on Measurement in Education , Montreal, Canada.
28.
Yin, P., & Schulz, E.M. (2005, April). A comparison of cut scores and cut score variability from Angoff-based and bookmark-based procedures in standard setting . Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal , Canada.
