Using Generalizability Theory to Estimate the Reliability of Writing Scores Derived from Holistic and Analytical Scoring Methods

Abstract

Issues surrounding the psychometric properties of writing assessments have received ongoing attention. However, the reliability estimates of scores derived from various holistic and analytical scoring strategies reported in the literature have relied on classical test theory (CT), which accounts for only a single source of variance within a given analysis. Generalizability theory (GT) is a more powerful and flexible strategy that allows for the simultaneous estimation of multiple sources of error variance to estimate the reliability of test scores. Using GT, two studies were conducted to investigate the impact of the number of raters and the type of decision (relative vs. absolute) on the reliability of writing scores. The results of both studies indicated that the reliability coefficients for writing scores decline as (a) the number of raters is reduced and (b) when absolute decisions rather than relative decisions are made.

Get full access to this article

View all access options for this article.

References

Cronbach, L. J. , Gleser, G. C. , Nanda, H. , & Rajaratnum, N. (1972). The dependability of behavioral measurements: Theory of generalizability of scores and profiles. New York: John Wiley.

Crowley, S. L. , Thompson, B. , & Worchel, F. (1994). The children’s depression inventory: A comparison of generalizability and classical test theory analysis. Educational and Psychological Measurement, 54(3), 705-713.

Educational Testing Service . (1992). Exploring new methods for collecting students’ school-based writing: NAEP’s 1990 portfolio study. Washington, DC: Government Printing Office.

Hammill, D. D. , & Larson, S. C. (1988). Test of Written Language-2. Austin, TX: PRO-ED.

Huot, B. (1990). The literature of direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60, 237-263.

Jaeger, R. (1991). Foreword. In R. J. Shavelson & N. M. Webb (Eds.), Generalizability theory: A primer (pp. ix-x). Newbury Park, CA: Sage.

Nunnally, J. C. (1967). Psychometric theory. New York: McGraw-Hill.

Roid, G. H. (1994). Patterns of writing skills derived from cluster analysis of direct writing assessments. Applied Measurement in Education, 7(2), 159-170.

Shavelson, R. J. , & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.

10.

Shavelson, R. J. , Webb, N. M. , & Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44(6), 922-932.