Think-aloud protocols (TAPs) are frequently used in research on essay rating processes. However, there are very few empirical studies of the completeness of TAP data and the effects of this technique on rater performance (i.e., rating processes and outcomes). This study aims to start to address this research gap. As part of a larger study on rater decision-making behaviors, 11 novice and 14 experienced raters rated, both analytically and holistically, a sample of ESL essays silently and while thinking aloud. The raters were then interviewed about their perceptions of thinking aloud and its effects. Essay scores were submitted to FACETS analyses, while TAP and interview data were analyzed qualitatively. Score and qualitative data analyses provided evidence and explanations concerning the veridicality and reactivity of TAPs across rater groups (novice vs. experienced) and rating scales (holistic vs. analytic). The paper concludes with several theoretical and methodological implications and questions for future studies using TAPs to build models of and compare essay rating processes across individuals, groups and contexts.
Barkaoui K. ( 2008) Effects of scoring method and rater experience on ESL essay rating processes and outcomes. Unpublished Ph.D. Thesis, University of Toronto.
2.
Blaikie N. ( 2000) Designing social research. Molden, MA: Polity Press and Blackwell.
3.
Chi MTH ( 1997) Quantifying qualitative analysis of verbal data: A practical guide. The Journal of the Learning Science, 6, 271-315.
4.
Cohen AD ( 1998) Strategies in learning and using a second language . London: Longman.
5.
Connor U. and Carrell PL ( 1993) The interpretation of the tasks by writers and readers in holistically rated direct assessment of writing. In JG Carson and I Leki (eds), Reading in the composition classroom: Second language perspectives (pp. 141-160). Boston, MA: Heine and Heine.
6.
Connor-Linton J. ( 1995) Looking behind the curtain: What do L2 composition ratings really mean?TESOL Quarterly, 29, 762-765.
7.
Critchfield TS and Epting LK ( 1998) The trouble with babies and the value of bathwater: Complexities in the use of verbal reports as data. The Analysis of Verbal Behavior, 15, 65-74.
8.
Cumming A. ( 1990) Expertise in evaluating second language compositions . Language Testing, 7, 31-51.
9.
Cumming A., Kantor R. and Powers D. ( 2001) Scoring TOEFL essays and TOEFL 2000 prototype writing tasks: An investigation into raters’ decision making and development of a preliminary analytic framework. (TOEFL Monograph Series N 22). Princeton, NJ: Educational Testing Service.
10.
Cumming A., Kantor R. and Powers D. ( 2002) Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86, 67-96.
11.
Delaruelle S. ( 1997) Text type and rater decision-making in the writing module . In G Brindley and G Wigglesworth (eds), Access: Issues in English language test design and delivery (pp. 215-242). Sydney: Macquarie University.
12.
DeRemer M. ( 1998) Writing assessment: Raters’ elaboration of the rating task. Assessing Writing, 5, 7-29.
13.
Duckett P. ( 2002) New insights: Eye fixations and the reading process . Talking Points, 13(2), 16-21.
14.
Erdosy MU ( 2004) Exploring variability in judging writing ability in a second language: A study of four experienced raters of ESL compositions. (TOEFL Research Report RR-03-17). Princeton, NJ: Educational Testing Service.
15.
Ericsson KA and Simon HA (1984/93 ) Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
16.
Ericsson KA and Simon HA ( 1987) Verbal reports on thinking. In C Faerch and G Kasper (eds), Introspection in second language research (pp. 24-53). Clevedon : Multilingual Matters.
17.
Faerch C. and Kasper G. ( 1987)From product to process: Introspective methods in second language research. In C Faerch and G Kasper (Eds), Introspection in second language research (pp. 5- 23). Clevedon: Multilingual Matters.
18.
Green A. ( 1998) Verbal protocol analysis in language testing research: A handbook. Cambridge: Cambridge University Press.
19.
Greene JC, Caracelli VJ and Graham WF ( 1989) Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis , 11, 255-274.
20.
Hamp-Lyons L. ( 1991) Scoring procedures for ESL contexts. In L Hamp-Lyons (ed.), Assessing second language writing in academic contexts (pp. 241-276). Norwood, NJ: Ablex.
21.
Huot BA ( 1993). The influence of holistic scoring procedures on reading and rating student essays. In MM Williamson and BA Huot (eds), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 206-236). Creskill, NJ: Hampton Press.
22.
Hyona J., Lorch RF Jr. and Kaakinen J. ( 2002) Individual differences in reading to summarize expository text: Evidence from eye fixation patterns. Journal of Educational Psychology, 94, 44-55.
Kormos J. ( 1998) The use of verbal reports in L2 research: Verbal reports in L2 speech production research. TESOL Quarterly , 32, 353-358.
25.
Linacre JM ( 2007) A user’s guide to FACETS Rasch model computer program . Available on line at: www.winsteps.com.
26.
Lumley T. ( 2002) Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19, 246-76.
27.
Lumley T. ( 2005) Assessing second language writing: The rater’s perspective . New York: Peter Lang .
28.
Mathison S. ( 1988) Why triangulate?Educational Researcher , 17, 13-17.
29.
Milanovic M., Saville N. and Shuhong S. ( 1996) A study of the decision-making behavior of composition markers. In M Milanovic and N Saville (eds), Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Colloquium (pp. 92-114). Cambridge: Cambridge University Press.
30.
Patton MQ ( 2002) Qualitative evaluation and research methods (3rd ed.). Newbury Park, CA: Sage.
31.
Russo, JE, Johnson EJ and Stephens DL ( 1989) The validity of verbal protocols. Memory and Cognition, 17, 759-769.
32.
Sakyi A. ( 2000) Validation of holistic scoring for ESL writing assessment: A study of how raters evaluate ESL compositions on a holistic scale. In AJ Kunnan (ed.), Fairness and validation in language assessment (pp. 130-153). Cambridge: Cambridge University Press.
33.
Sasaki T. ( 2003) Recipient orientation in verbal report protocols: Methodological issues in concurrent think-aloud. Second Language Studies , 22, 1-54.
34.
Smagorinsky P. ( 1994) Think-aloud protocol analysis: Beyond the black box. In P Smagorinsky (ed.), Speaking about writing: Reflections on research methodology (pp. 3-19). Thousand Oaks, CA: Sage.
35.
Smagorinsky P. ( 2001) Rethinking protocol analysis from a cultural perspective . Annual Review of Applied Linguistics, 21, 233-245.
36.
Smith D. ( 2000) Rater judgments in the direct assessment of competency-based second language writing ability. In G Brindley (ed.) Studies in immigrant English language assessment , Volume 1 (pp. 159-189). Sydney: Macquarie University.
37.
Sternglass MS ( 1988) The presence of thought: Introspective accounts of reading and writing. Norwood, NJ: Ablex .
38.
Stratman JF and Hamp-Lyons L. ( 1994) Reactivity in concurrent think-aloud protocols: Issues for research. In P Smagorinsky (ed.), Speaking about writing: Reflections on research methodology (pp. 89-111). Thousand Oaks, CA: Sage.
39.
Vaughan C. ( 1991) Holistic assessment: What goes on in the rater’s mind? In L Hamp-Lyons (ed.), Assessing second language writing in academic contexts (pp. 111-125). Norwood, NJ: Ablex.
40.
Weigle SC ( 1994) Effects of training on raters of ESL compositions . Language Testing, 11, 197-223.
41.
Weigle SC ( 1999) Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing , 6, 145-178.
42.
Wolfe EW ( 1997) The relationship between essay reading style and scoring proficiency in a psychometric scoring system. Assessing Writing , 4, 83-106.
43.
Wolfe EW, Kao C. and Ranney M. ( 1998) Cognitive differences in proficient and non-proficient essay scorers. Written Communication, 15, 465-492.
