The assessment of students' writing skills through essays is a common practice in educational institutions. Scoring of essays requires considerable judgment on the part of those who rate the response. When raters assign different scores to an essay, testing practitioners must resolve the discrepancy before computing an operational score to report to the examinee. This study investigated five forms of score resolution that were reported in a national survey of state department of education-testing agencies. The study examined the effect that each form of resolution has on the reliability of the resulting operational scores. It is shown that some methods of resolution can be associated with higher interrater reliability than can others. It is also shown that the choice of resolution can affect the magnitude of the reported score as well as the final passing rate of an assessment.
Get full access to this article
View all access options for this article.
References
1.
Breland, H.
(1983). The direct assessment of writing skill: A measurement review (Technical Report No. 83-6). Princeton, NJ: College Entrance Examination Board.
2.
Brennan, R.
(1996). Generalizability of performance assessments. In G. Phillips (Ed.), Technical issues in large-scale performance assessment (pp. 19-58). Washington, DC: National Center for Education Statistics.
3.
Brown, J.
, Hilgers, T., & Marsella, J. (1991). Essay prompts and topics: Minimizing the effect of mean differences. Written Communication, 8, 533-556.
4.
Cherry, R.
, & Meyer, P. (1993). Reliability issues in holistic assessment. In M. Williamson & B. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 109-141). Cresskill, NJ: Hampton.
5.
Clauser, B.
, Clyman, S., & Swanson, D. (1999). Components of rater error in a complex performance assessment. Journal of Educational Measurement, 36, 29-45.
6.
Coffman, W.
(1971). Essay examinations. In R. Thorndike (Ed.), Educational measurement (2nd ed., pp. 271-302). Washington, DC: American Council on Education.
7.
Cronbach, L.
, Linn, R., Brennan, R., & Haertel, E. (1995). Generalizability analysis for educational assessments. Los Angeles: University of California at Los Angeles, Center for the Study of Evaluation, Standards, and Student Testing.
8.
Cronbach, L.
, Linn, R., Brennan, R., & Haertel, E. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399.
9.
Georgia Department of Education
. (1993). Development of writing. Atlanta, GA: Author.
10.
Hayes, J. R.
, & Hatch, J. (1999). Issues in measuring reliability: Correlation versus percentage of agreement. Written Communication, 16, 354-367.
11.
Hayes, J. R.
, Hatch, J., & Silk, C. (2000). Does holistic assessment predict writing performance? Estimating the consistency of student performance on holistically scored writing assignments. Written Communication, 17, 3-26.
12.
Herman, J.
, Aschbacher, P., & Winters, L. (1992). A practical guide to alternative assessment. Alexandria, VA: Association for Supervision and Curriculum Development.
13.
Hieronymous, A.
, Hoover, H., Cantor, N., & Oberley, K. (1987). Handbook for focused holistic scoring. Chicago: Riverside.
14.
Huot, B.
(1990). The literature of direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60, 237-263.
15.
Janopoulos, M.
(1989). Reader comprehension and holistic assessment of second language writing proficiency. Written Communication, 6, 218-237.
16.
Johnson, R.
, McDaniel, F., & Willeke, M. (2000). Using portfolios in program evaluation: An investigation of interrater reliability. The American Journal of Evaluation, 21, 65-80.
17.
Johnson, R.
, Penny, J., & Gordon, B. (2000). The relationship between score resolution methods and score reliability: An empirical study of an analytic scoring rubric. Applied Measurement in Education, 13, 121-138.
18.
Johnson, R.
, Penny, J., & Johnson, C. (2000, April). A conceptual framework for score resolution in the rating of performance assessments: The union of validity and reliability. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
19.
Johnson, R.
, Willeke, M., Bergman, T., & Steiner, D. (1997). Family literacy portfolios: Development and implementation. Window on the World of Family Literacy, 2(2), 10-17.
20.
Kane, M.
, Crooks, T., & Cohen, A. (1999). Validating measures of performance. Educational Measurement: Issues and Practice, 18(2), 5-17.
21.
Mehrens, W.
, & Lehmann, I. (1991). Measurement and evaluation in education and psychology (4th ed.). Fort Worth, TX: Harcourt Brace College.
22.
Moss, P.
, Schutz, A., & Collins, K. (1997, April). Developing coherence between assessment and reform: An integrative approach to portfolio evaluation for teacher licensure. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
23.
National Education Goals Panel
. (1996). Profile of 1994-1995 state assessment systems and reported results. Washington, DC: Author.
24.
Olson, J.
, Bond, L., & Andrews, C. (1999). Annual survey of state student assessment programs: A summary report. Washington, DC: Council of Chief State School Officers.
25.
Shavelson, R.
, & Webb, N. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
26.
Smith, W.
(1993). Assessing the reliability and adequacy of using holistic scoring of essays as a college composition placement technique. In M. Williamson & B. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 142-205). Cresskill, NJ: Hampton.
27.
Veal, L.
, & Hudson, S. (1983). Direct and indirect measures for large-scale evaluation of writing. Research in the Teaching of English, 17, 290-296.
28.
Welch, C.
, & Martinovich-Barhite, D. (1997, April). Reliability issues and possible solutions. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
29.
White, E.
(1993). Holistic scoring: Past triumphs, future challenges. In M. Williamson & B. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 79-107). Cresskill, NJ: Hampton.
30.
Wolcott, W.
(1998). An overview of writing assessment: Theory, research, and practice. Urbana, IL: National Council of Teachers of English.
