Speech act theory-based, second language pragmatics testing (SLPT) poses problems for validation due to a lack of correspondence with empirical conversational data. Since conversation analysis (CA) provides a richer and more accurate account of language behavior, it may be preferred as a basis for SLPT development. However, applying CA methodology in turn poses epistemological and practical challenges to psychometrics-driven language testing. The present pilot study, attempting to resolve this seeming conflict, focuses on the behavior of two CA-trained raters applying a holistic rubric to responses on a test of ESL oral pragmatic competence. Results showed that CA-informed testing (CAIT) could be practical, though statistical reliability was not achieved. However, post-rating hermeneutic dialogues between the raters provided evidence that valid inferences of examinee oral ESL pragmatic ability could be made through iterative, rater recourse to empirical data in a conversation-analytic mode. A model for iterative, CA-informed SLPT development is offered.
Get full access to this article
View all access options for this article.
References
1.
Austin, J.L.1962: How to do things with words. Cambridge, MA: Harvard University Press.
2.
Bachman, L.F.1990: Fundamental considerations in language testing. Oxford: Oxford University Press.
3.
Bachman, L.F. and Palmer, A.S.1996: Language testing in practice. Oxford : Oxford University Press.
4.
Bachman, L.F. and Savignon, S.J.1986: The evaluation of communicative language proficiency: A critique of the ACTFL Oral Interview. The Modern Language Journal70: 380—90.
5.
Bardovi-Harlig, K.2001: Evaluating the empirical evidence: Grounds for instruction in pragmatics? In Rose, K.R. and Kasper, G., editors, Pragmatics in language teaching. Cambridge: Cambridge University Press, 13—32.
6.
Barraja-Rohan, A.-M. and Pritchard, C.R.1997: Beyond talk: A course in communication and conversation for intermediate adult learners of English. Footscray, Victoria: Western Melbourne Institute of TAFE Publishing Service .
7.
Beebe, L., Takahashi, T. and Uliss-Weltz, R.1990: Pragmatic transfer in ESL refusals. In Scarcella, R.C., Anderson, E. and Krashen, S.C., editors, Developing communicative competence in a second language. New York: Newbury House, 55—73.
8.
Blum-Kulka, S.1982: Learning how to say what you mean in a second language . Applied Linguistics3: 29—59.
9.
Blum-Kulka, S., House, J. , and Kasper, G., editors 1989: Cross-cultural pragmatics: Requests and apologies. Norwood, NJ: Ablex.
10.
Blum-Kulka, S. and Olshtain, E.1986: Too many words: Length of utterance and pragmatic failure . Studies in Second Language Acquisition8: 47—61.
11.
Brown, J.D.2001: Pragmatics tests: Different purposes, different tests . In Rose, K.R. and Kasper, G., editors, Pragmatics in language teaching. Cambridge: Cambridge University Press, 301—25.
12.
Chapelle, C.1999: Validity in language assessment. Annual Review of Applied Linguistics19: 254—72.
13.
Cohen, A. and Olshtain, E.1981: Developing a measure of sociocultural competence: The case of apology. Language Learning31: 113—34.
14.
Faerch, C. and Kasper, G.1989: Internal and external modification in interlanguage request realization. In Blum-Kulka, S., House, J. and Kasper, G. , editors, Cross-cultural pragmatics: Requests and apologies . Norwood, NJ: Ablex, 221—47.
15.
Fulcher, G.N.1993: The construction and validation of rating scales for oral tests of English as a foreign language. Unpublished dissertation, Lancaster University. Cited in Pollitt et al., 1996.
16.
Gamaroff, R.2000: Rater reliability in language assessment: The bug of all bears. System28: 31—53.
17.
Golato, A.2003: Studying compliment responses: A comparison of DCTs and recordings of naturally occurring talk. Applied Linguistics1: 1—54.
18.
Green, G.M.1996: Pragmatics and natural language understanding. Mahwah, NJ: Lawrence Erlbaum.
19.
Hancher, M.1979: The classification of cooperative illocutionary acts. Language in Society8: 1—14.
20.
Heritage, J.1984: Conversation analysis. In Heritage, J., editor, Garfinkel and ethnomethodology. Cambridge: Polity Press, 233—92.
21.
House, J. and Kasper, G.1987: Interlanguage pragmatics: Requesting in a foreign language . In Lorscher, W. and Schulze, R., editors, Perspectives on language in performance, Vol. 2. Tubingen : Narr, 1250—88.
22.
Hudson, T., Detmer, E., and Brown, J.D.1995: Developing prototypic measures of cross-cultural pragmatics . Honolulu: University of Hawai'i Press.
23.
Johnson, M. and Tyler, A.1998: Re-analyzing the OPI: How much does it look like natural conversation? In Young, R. and He, A.W., editors, Talking and testing: discourse approaches to the assessment of oral proficiency. Amsterdam: John Benjamins, 27—51.
24.
Kasper, G.1981: Pragmatische Aspekte in der InterimspracheTubingen: Narr.
25.
—— 1989: Variation in interlanguage speech act realization. In Gass, S., Madden, C., Preston, D. and Selinker, L., editors, Variation in second language acquisition, Vol. 2: Discourse and Pragmatics. Clevedon, CH: Multilingual Matters, 37—58.
26.
—— 1992: Pragmatic transfer. Second Language Research8: 203—31.
27.
—— 2005: Speech acts in interaction: Towards discursive pragmatics . Plenary talk given at the 16th International Conference on Pragmatics and Language Learning. Bloomington, IN, 15 April 2005.
28.
Kasper, G. and Dahl, M.1991: Research methods in interlanguage pragmatics. Studies in Second Language Acquisition13: 215—47.
29.
Kasper, G. and Rose, K.R.2002: Pragmatic development in a second language. Oxford: Blackwell.
30.
Levinson, S.C.1983: Pragmatics. Cambridge: Cambridge University Press.
31.
Linn, R.L., Baker, E.L., and Dunbar, S.B.1991: Complex, performance-based assessment: Expectations and validation criteria. Educational Researcher15—21.
32.
Mey, J.L.2001: Pragmatics: An introduction. Malden, MA: Blackwell.
33.
Messick, S.1989: Validity. In Linn, R.L. , editor, Educational measurement, third edition. London: Collier Macmillan , 13—103.
34.
Moss, P.A.1994: Can there be validity without reliability?Educational Researcher23, 2: 5—12.
35.
—— 1996: Enlarging the dialogue in educational measurement: Voices from the interpretive traditions. Educational Researcher25: 20—28.
36.
Olshtain, E.1989: Apologies across languages. In Blum-Kulka, S., House, J. and Kasper, G., editors, Cross-cultural pragmatics: Requests and apologies. Norwood, NJ: Ablex, 155—73.
37.
Pollitt, A. and Murray, N.L.1996: What raters really pay attention to. In Milanovic, M. and Saville, N., editors, Studies in language testing 3: Performance, testing, cognition, and assessment. Cambridge: Cambridge University Press.
38.
Pomerantz, A.1978: Compliment responses: Notes on the co-operation of multiple constraints. In Schenkein, J., editor, Studies in the organization of conversational interaction. New York: Academic Press, 57—101.
39.
—— 1984: Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes. In. Atkinson, J.M and Heritage, J., editors, Structures of social action: Studies in conversational analysis . New York: Academic Press, 79—112.
40.
Richards, J.C. and Schmidt, R.W.1983: Conversational analysis. In Richards, J.C. and Schmidt, R.W., editors, Language and communication. London: Longman, 117—54.
41.
Rose, K.1992: Speech acts and questionnaires: The effect of hearer response. Journal ofPragmatics17: 49—62.
42.
Sacks, H., Schegloff, E., and Jefferson, G.1974: A simplest schematics for the organization of turn-taking for conversation. Language50: 696—735.
43.
Schegloff, E.A.1993: Reflections on quantification in the study of conversation . Research on Language and Social Interaction26: 99—128.
Schwandt, T. and Jang, E.2002: On the significance of questioning psychometric discourse in language testing. Paper presented at the International Conference on World Englishes, October 2002, University of Illinois at Urbana-Champaign.
46.
Searle, J.R.1969: Speech acts: An essay in the philosophy of language. Cambridge: Cambridge University Press.
47.
—— 1975: Indirect speech acts. In Cole, P. and J. Morgan, editors, Syntax and semantics, Vol. 3: Speech acts. New York: Academic Press, 59—82.
48.
Tanaka, N.1988: Politeness: Some problems for Japanese speakers of English . JALT Journal9: 81—102.
49.
Walters, J.1979: The perception of deference in English and Spanish. In Yorio, C.A., Perkins, K. and Schacter, J., editors, OnTESOL279. Washington, DC: TESOL, 288—96.
50.
Yamashita, S.2001: Using pictures for research in pragmatics: Eliciting pragmatic strategies by Picture Response Tests. In Hudson, T. and Brown, J. D., editors, A focus on language test development. Honolulu: University of Hawai'i Press.
