The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational I settings

Abstract

The increasing demand for performance assessment of speaking skills in second languages has led to logistic complications, for example, the delivery of tests in overseas locations. One solution to the problem has been to train native speaker interlocutors to carry out a series of oral interactions with the candidate, with assessment from audiorecordings of the test session postponed and conducted cen trally by a small team of trained raters. This technique is currently used in two large-scale occupationally related ESP tests administered internationally on behalf of the Australian government. But these procedures raise questions about the effect of such facets of the assessment situation as interlocutor variables and the quality of the audiotape recording. Recent developments in multifaceted Rasch measure ment have significantly broadened the possibilities for investigation of these issues. The research presented in this article investigates potential problems associated with the above approach to the offshore testing of speaking skills. Data from audiotape-based assessments of 70 offshore candidates from two administrations of the Occupational English Test, an advanced-level ESP test for health pro fessionals, are considered. In addition to multiple ratings of candidate perform ance, each recording is rated for perceptions of the competence of the interlocutor, the rapport established between the candidate and the interlocutor, and the audi bility of the interaction. These aspects of the assessment situation are treated as facets in a series of multifaceted Rasch analyses of the data.

The results of the analysis reveal the effects of interlocutor variability and audio tape quality on ratings. The article concludes with an evaluation of the overall feasibility of the procedure, and implications for test administration are considered. The study is also a further demonstration of the application of multifaceted Rasch measurement in performance assessment settings.

Get full access to this article

View all access options for this article.

References

Clark, J.L.D. 1978: Interview testing research at Educational Testing Service . In Clark, J.L.D. , editor, Direct testing of speaking proficiency: theory and application, Princeton, NJ: Educational Testing Service, 211-28.

Filipi, A. 1994: Interaction in an Italian oral test: the role of some expansion sequences. Australian Review of Applied Linguistics Series S, 11, 119-36.

Hamp-Lyons, L. and Mathias, S.P. 1994: Examining expert judgements of task difficulty on essay tests. Journal of Second Language Writing 3, 49-68.

Ingram, D.E. 1984: Report on the formal trialling of the Australian Second Language Proficiency Ratings (ASLPR). Canberra: Australian Government Publishing Service.

Lazaraton, A. 1991: A conversation analysis of structure and interaction in the language interview. Unpublished PhD dissertation, University of California, Los Angeles.

Linacre, J.M. 1989: Many-facet Rasch measurement. Chicago, IL: MESA Press.

1994 : Constructing measurement with a many-facet Rasch model. In Wilson, M. , editor, Objective measurement: theory into practice. Vol. 2, Norwood, NJ: Ablex, 129-44.

Linacre, J.M. and Wright, B. 1992: FACETS: Rasch measurement computer program, version 2.6 Chicago, IL: MESA Press.

Lowe, P. 1978: Third rating of FSI interviews. In Clark, J.L.D. , editor, Direct testing of speaking proficiency: theory and application, Princeton, NJ: Educational Testing Service, 159-69.

10.

Lumley, T. and Brown, A. 1996: Specific purpose language performance tests: task and interaction. In G. Wigglesworth and C. Elder, editors, The language testing cycle: from inception to washback. Australian Review of Applied Linguistics Series S, 13, 105-36.

11.

McNamara, T.F. 1988: The development of an English as a second language speaking test for health professionals. Part one of a report to the Council on Overseas Professional Qualifications on a consultancy to develop the Occupational English Test. Parkville, Victoria: University of Melbourne, Department of Russian and Language Studies.

12.

— 1990: Assessing the second language proficiency of health professionals . Unpublished PhD thesis, University of Melbourne.

13.

— 1996: Measuring second language performance. London and New York: Longman .

14.

Morton, J. and Wigglesworth, G. 1994: Evaluating interviewer input in oral interaction tests. Paper presented at the Second Language Research Forum, Montréal, 6-9 October.

15.

Morton, J. , Wigglesworth, G. and Williams, D. 1995: Evaluating interviewer performance in oral interaction tests. Paper presented at the Applied Linguistics Association of Australia Congress, Canberra, September. Forthcoming in Wigglesworth, G. and Brindley, G., editors, access: issues in English language test design and delivery, Sydney : National Centre for English Language Teaching and Research, Macquarie University.

16.

O'Loughlin, K. 1997: The equivalence of two versions of an oral proficiency test. PhD thesis, University of Melbourne.

17.

Porter, D. 1991: Affective factors in language testing. In Alderson, J.C. and North, B. , editors, Language testing in the 1990s, London: Modern English Publications and The British Council, 32-40.

18.

Ross, S. 1992: Accommodative questions in oral proficiency interviews . Language Testing 9, 173-86.

19.

Ross, S. and Berwick, R. 1990: The discourse of accommodation in oral proficiency interviews . Studies in Second Language Acquisition 14, 159-76.

20.

Young, R. 1995: Conversational styles in language proficiency interviews . Language Learning 45, 3-42.