Effects of Using a Scoring Guide on Essay Scores: Generalizability Theory

This study was conducted to test the effect of task level and item consistency when two conditions, with and without the assistance of a scoring guide, were used to score essays. The use of generalization theory was proposed as a framework for examining the effect of task variability and use of the scoring guide on achievement measures. Participants were 21 students in Grade 9 enrolled in regular Turkish language and literature classes. Of these students 11 were men and 10 were women. Ten teachers from the city were raters. In the past, raters of essays have given varied judgements of writing quality. Utilizing decision and generalizability theories, variation in scores was evaluated using a three-way (person × rater × task) analysis of variance design. The scoring guide was beneficial in reducing variability of evaluating grammar and reading comprehension but not as helpful when assessing knowledge of concepts.