A multi-method analysis of evaluation criteria used to assess the speaking proficiency of graduate student instructors

Abstract

The study reported here examined the evaluation criteria used to assess the proficiency and effectiveness of the language produced in an oral performance test of English conducted in an American university context. Empirical methods were used to analyze qualitatively and quantitatively transcriptions of the Oral English Tests (OET) of 44 prospective Graduate Student Instructors (GSI). The language required to complete the tasks on the test was conceptualized from the functional perspective of transactional and interactional language use as defined by Brown and Yule (1989). Listening comprehension and pronunciation were also analyzed and scored. Stepwise logistic regression was used to determine the extent to which these linguistic features contributed to final ratings. These quantitative findings were then compared to ‘real-time’ written comments made by evaluators during the tests. Intuitive methods were then used to further explore those features of candidate performance attended to by evaluators: interviews were conducted with experienced evaluators to determine the features they judged necessary for communicating effectively in instructional settings. Results indicate that the three data sources converge on two main features — pronunciation and listening comprehension — that are important in describing and evaluating the proficiency of prospective GSIs.

Keywords

evaluation criteria for speaking tests oral performance testing rating scale. rating scale validation second language speaking assessment second language speaking proficiency teaching assistants

Get full access to this article

View all access options for this article.

References

Beebe, L. & Waring, H.Z. ( 2004). The linguistic encoding of pragmatic tone: Adverbials as words that work. In Boxer, D. & Cohen, A. (Eds.), Studying speaking to inform second language learning (pp. 228-249). Clevedon , UK: Multilingual Matters .

Blum-Kulka, S. , House, J. , & Kasper, G. (Eds.) (1989). Cross-cultural pragmatics: Requests and apologies . Norwood, NJ: Ablex.

Briggs, S.L. (Developer). (1987, 1999, 2003). Graduate Student Instructor Oral English Test. English Language Institute, University of Michigan. Ann Arbor, MI: English Language Institute .

Briggs, S.L. ( 1994). Using performance assessment methods to screen ITAs. In Madden, C. G. & C. L. Myers (Eds.), Discourse and performance of international teaching assistants. Alexandria: TESOL.

Brown, A. ( 2003). Interviewer variation and the co-construction of speaking proficiency . Language Testing, 20, 1-25.

Brown, G. & Yule, G. ( 1989). Discourse analysis. Cambridge: Cambridge University Press.

Bygate, M. ( 1999). Quality of language and purpose of task: Patterns of learners’ language on two oral communication tasks . Language Testing Research, 3, 185-214.

Chalhoub-Deville, M. ( 1995). Deriving oral assessment scales across different tasks and rater groups . Language Testing, 12, 16-33.

Chalhoub-Deville, M. ( 2003). Second language interaction: Current perspectives and future trends . Language Testing, 20, 369-383.

10.

Chapelle, C. 1998: Construct definition and validity inquiry in SLA research . In L. F. Bachman & A. D. Cohen (Eds.), Second language acquisition and language testing interface (pp. 32-70). Cambridge: Cambridge University Press.

11.

Chapelle, C.A. ( 1999). Validity in language assessment . Annual Review of Applied Linguistics, 19, 254-272.

12.

Douglas, D. ( 2000). Assessing languages for specific purposes. Cambridge: Cambridge University Press.

13.

Foster, P. , Tonkyn, A. , & Wigglesworth, G. (2000). Measuring spoken language: A unit for all reasons . Applied Linguistics, 21, 354-375.

14.

Fulcher, G. ( 1996). Does thick description lead to smart tests? A data-based approach to rating scale construction . Language Testing, 13, 208-238.

15.

Fulcher, G. ( 2003). Testing second language speaking. London: Pearson.

16.

Fulcher, G. , & Márquez Reiter, R. (2003). Task difficulty in speaking tests . Language Testing, 20, 321-344.

17.

Halliday, M.A.K. , & Hasan, R. ( 1976). Cohesion in English. London: Longman.

18.

Halliday, M.A.K. , & Hasan, R. ( 1989). Language, context, and text: A social semiotic perspective . Oxford: Oxford University Press .

19.

Huang, Li-Shih. ( 2005). Fine-tuning the craft of teaching by discussion . Business Communication Quarterly, 68, 492-500.

20.

Iwashita, N. , Brown, A. , McNamara, T. , & O’ Hagan, S. (2008). Assessed levels of second language speaking proficiency: How distinct ? Applied Linguistics , 29, 24-49.

21.

Lantolf, J.P. , & G. Appel. (Eds.). (1994). Theoretical framework: An introduction to Vygotskian perspectives on second language research. Westport, CT: Ablex.

22.

Levelt, W.J.M. ( 1981). The speaker’s linearisation problem . Philosophical Transactions of the Royal Society of London, 295, 305-315.

23.

Lumley, T. , & O’ Sullivan, B. (2005). The effect of test-taker gender, audience, and topic on task performance in tape-mediated assessment of speaking . Language Testing, 22, 415-437.

24.

Madden, C. G. , & C. L. Myers (Eds.) (1994). Discourse and performance of international teaching assistants. Alexandria: TESOL.

25.

Myers, C.L. ( 1994). Question-based discourse in science labs: Issues for ITAs. In Madden, C. G. , & C. L. Myers (Eds.), Discourse and performance of international teaching assistants. Alexandria: TESOL.

26.

Nakatsuhara, F. ( 2008). Inter-interviewer variation in oral interview tests . ELT Journal, 62, 266-275.

27.

O’Loughlin, K. ( 2002). The impact of gender in oral proficiency testing . Language Testing, 19, 169-192.

28.

Plakans, B.S. ( 1997). Undergraduates’ experiences with and attitudes toward international teaching assistants . TESOL Quarterly , 31, 95-119.

29.

Rounds, P.L. ( 1990). Student questions: What do we know about them? Paper presented at the Preconference Symposium on ITA Training, 24th Annual TESOL Convention, San Francisco, CA.

30.

Skehan, P. ( 2001). Tasks and language performance assessment. In Bygate, M. , P. Skehan , & M. Swain (Eds.). Researching pedagogic tasks second language learning, teaching, and testing. London: Pearson.

31.

Swain, M. ( 2001). Examining dialogue: Another approach to content specification and to validating inferences drawn from test scores . Language Testing, 18, 275-302.

32.

Taguchi, N. ( 2007). Task difficulty in oral speech act production . Applied Linguistics, 28, 113-135.

33.

van Dijk, T.A. ( 1977). Text and context. London: Longman.

34.

Upshur, J. , & Turner, C. ( 1995). Constructing rating scales for second language tests . English Language Teaching Journal, 49, 3-12.

35.

Upshur, J. , & Turner, C. ( 1999). Systematic effects in the rating of second-language speaking ability: test method and learner discourse . Language Testing , 16, 82-111.