The difficulty of speaking tasks has only recently become a topic of investigation in
language testing. This has been prompted by work on discourse variability in second
language acquisition (SLA) research, new classificatory systems for describing
tasks, and the advent of statistical techniques that enable the prediction of task
difficulty. This article reviews assumptions underlying approaches to research into
speaking task difficulty and questions the view that test scores always vary with
task conditions or discourse variation. A new approach to defining task difficulty
in terms of the interaction between pragmatic task features and first language (L1)
cultural background is offered, and the results of a study to investigate the impact
of these variables on test scores are presented. The relevance for the
generalizability of score meaning and the definition of constructs in speaking tests
is discussed.
Get full access to this article
View all access options for this article.
References
1.
Bachman, L. F.1990: Fundamental considerations in language testing.
Oxford: Oxford University Press
.
2.
Bachman, L. F.
,
Davidson, F.
,
Ryan, K.
and
Choi, I.-C.1995: An investigation into the comparability of two tests of
English as a foreign language. The Cambridge-TOEFL comparability study.
Cambridge: Cambridge University Press
.
3.
Bachman, L. F.
,
Kunan, A.
,
Vanniarajan, S.
and
Lynch, B.1988:
Task and ability analysis as a basis for examining content and construct
comparability in two EFL proficiency test batteries
. Language Testing5,
160-186
.
4.
Bachman, L. F.
,
Lynch, B.
and
Mason, M.1995:
Investigating variability in tasks and rater judgements in a performance
test of foreign language speaking
. Language Testing12,
238-257
.
5.
Bachman, L. F.
and
Palmer, A.1996: Language testing in practice.
Oxford: Oxford University Press
.
6.
Bachman, L. F.
and
Savignon, S. J.1986:
The evaluation of communicative language proficiency: a critique of the
ACTFL Oral Interview
. Modern Language Journal70,
380-390
.
7.
Brown, A.
and
Hill, K.1998: Interviewer style and candidate performance in the IELTS
oral interview. In
Woods, S.
, editor, Research Reports 1997, 1, 173-191.
Sydney: ELICOS (English Language Intensive
Courses for Overseas Students)
.
8.
Brown, J. D.
,
Hudson, T.
and
Norris, J.
1999:
Validation of test-dependent and task-independent ratings of performance assessment
. Paper presented at the 21st Language Testing Research
Colloquium, Tsukuba, Japan, July.
9.
Chalhoub-Deville, M.1995:
Deriving oral assessment scales across different tests and rater groups
. Language Testing12,
16-33
.
10.
Chapelle, C.1988: Construct definition and validity inquiry in SLA
research. In
Bachman, L. F.
and
Cohen, A. D.
, editors, Interfaces between second language acquisition and language
testing research.
Cambridge: Cambridge University Press
, 32-70.
11.
Chapelle, C.1999: From reading theory to testing practice. In
Chalhoub-Deville, M.
, editor, Issues in computer-adaptive testing of reading.
Cambridge: Cambridge University Press
, 150-166.
12.
Clapham, C.2000:
Assessment for academic purposes: where next?System28,
511-521
.
13.
Cohen, J.1988: Statistical power analysis for the behavioural sciences.
Hillsdale, NJ: Erlbaum
.
14.
Cohen, J.1992:
A Power Primer
. Psychological Bulletin112,
155-159
.
15.
Crookes, G.
and
Gass, S.1993: Introduction. In
Crookes, G.
and
Gass, S.
, editors, Tasks and language learning: integrating theory and
practice.
Clevedon: Multilingual Matters
, 1-7.
16.
Douglas, D.1998: Language for Specific Purposes. In
Clapham, C.
and
Corson, D.
, editors, Language testing and assessment. Vol. 7 of the Encyclopedia of
language and education.
Dordrecht: Kluwer Academic Publishers
, 111-119.
17.
Douglas, D.2000: Assessing languages for specific purposes.
Cambridge: Cambridge University Press
.
18.
Douglas, D.
and
Selinker, L.1985:
Principles for language tests within the ‘discourse
domains’ theory of interlanguage: research, test construction and interpretation
. Language Testing2,
205-226
.
19.
Douglas, D.
and
Smith, J.1997: Theoretical underpinnings of the Test of Spoken English
revision project. TOEFL Monograph Series 9.
Princeton, NJ: Educational Testing Service
.
20.
Ellis, R.1985:
A variable competence model of second language acquisition.’IRAL23,
47-59
.
21.
Foster, P.
and
Skehan, P.1996:
The influence of planning on performance in task-based learning
. Studies in Second Language Acquisition18,
299-324
.
22.
Fulcher, G. 1993: The construction and validation of rating scales for oral
tests in English as a foreign language. University of Lancaster, UK: Unpublished PhD thesis.
23.
Foster, P.
and
Skehan, P.1995:
Variable competence in second language acquisition: a problem for research methodology?System23,
25-33
.
24.
Foster, P.
and
Skehan, P.1996a:
Testing tasks: issues in task design and the group oral
. Language Testing13,
23-51
.
25.
Foster, P.
and
Skehan, P.1996b:
Does thick description lead to smart tests? A data-based approach to rating
scale construction
. Language Testing13,
208-238
.
26.
Foster, P.
and
Skehan, P.1999:
Assessment in English for academic purposes: putting content validity in
its place
. Applied Linguistics20,
221-236
.
27.
Iwashita, N.
,
McNamara, T.
and
Elder, C.2001:
Can we predict task difficulty in an oral proficiency test? Exploring the
potential of an information processing approach to task design
. Language Learning51,
401-436
.
28.
Kirk, R. E.1996:
Practical significance: a concept whose time has come
. Educational and Psychological Measurement56,
746-759
.
29.
Labov, W.
and
Fanshel, D.1977: Therapeutic discourse.
New York: Academic Press
.
30.
Leech, G.1983: Principles of pragmatics.
London: Longman
.
31.
Linacre, J. M.
and
Wright, B. D.1990: Facets - many faceted rasch analysis.
Chicago, IL: Messa Press
.
32.
Annals of the New York Academy of Sciences379,
259-278
.
33.
Lynch, B.
and
McNamara, T.1998:
Using G-theory and many-faceted Rasch measurement in the development of
performance assessments of the ESL speaking skills of immigrants
. Language Testing15,
158-180
.
34.
Madsen, H. S.
and
Jones, R. L.1981: Classification of Oral Proficiency Tests. In
Palmer, A. S.
,
Groot, P. J. M.
and
Trosper, G. A.
, editors, The construct validation of tests of communicative
competence.
Washington DC: TESOL Publications
, 15-30.
35.
Márquez Reiter, R.1997: Sensitising Spanish learners of English to cultural
differences: the case of politeness. In
Pütz, M.
, editor, The cultural context in foreign language teaching.
Frankfurt: Peter Lang
, 143-155.
36.
Pica, T.
,
Kanagy, R.
and
Falodun, J.1993: Choosing and using communication tasks for second language
instruction and research. In
Crookes, G.
and
Gass, S.
, editors, Tasks and language learning: integrating theory and
practice.
Clevedon: Multilingual Matters
, 9-34.
37.
Pollitt, A.1991: Giving students a sporting chance: assessment by counting
and judging. In
Alderson, J. C.
and
North, B.
, editors, Language testing in the 1990s.
Modern English Publications in Association with the British Council
, 46-59.
38.
Shohamy, E.1983:
The stability of oral proficiency in the oral interview procedure
. Language Learning33,
527-540
.
39.
Shohamy, E.1988:
A proposed framework for testing the oral language of second/foreign
language learners
. Studies in Second Language Acquisition10,
165-179
.
40.
Shohamy, E.1990:
Language testing priorities: a different perspective
. Foreign Language Annals23,
365-394
.
41.
Shohamy, E.
,
Reves, T.
and
Bejerano, Y.1986:
Introducing a new comprehensive test of oral proficiency
. English Language Teaching Journal40,
212-220
.
42.
Skehan, P.1998a: A cognitive approach to language learning.
Oxford: Oxford University Press
.
43.
Skehan, P.1998b:
Processing perspectives to second language development, instruction,
performance and assessment
. Thames Valley Working Papers in Applied Linguistics4,
70-88
.
44.
Skehan, P.
and
Foster, P.1997:
The influence of planning and post-task activities on accuracy and
complexity in task based learning
. Language Teaching Research1,
185-211
.
45.
Swain, M.2001:
Examining dialogue: another approach to content specification and to
validating inferences drawn from test scores
. Language Testing18,
275-302
.
46.
Tarone, E.1983:
On the variability of interlanguage systems
. Applied Linguistics4,
142-163
.
47.
Tarone, E.1985:
Variability in interlanguage use: a study of style-shifting in morphology
and syntax
. Language Learning35,
373-403
.
48.
Tarone, E.1987: Methodologies for studying variability in second language
acquisition. In
Ellis, R.
, editor, Second language acquisition in context.
Hemel Hempstead: Prentice Hall International
, 35-40.
49.
Tarone, E.1988: Variation in interlanguage.
London: Edward Arnold
.
50.
Tarone, E.1998: Research on interlanguage variation: implications for
language testing. In
Bachman, L. F.
and
Cohen, A. D.
, editors, Interfaces between second language acquisition and language
testing research.
Cambridge: Cambridge University Press
, 71-89.
51.
Underhill, N.1987: Testing spoken language. A handbook of oral testing
techniques.
Cambridge: Cambridge University Press
.
52.
Upshur, J. A.
and
Turner, C. E.1999:
Systematic effects in the rating of second-language speaking ability: test
method and learner discourse
. Language Testing16,
82-111
.
53.
Valette, R.1977: Modern language testing. Second edition.
San Diego, CA: Harcourt Brace Jovanovich
.
54.
Weir, C.1988: Communicative language testing.
Exeter: Exeter University Press
.
55.
Weir, C.1993: Understanding and developing language tests.
Hemel Hempstead: Prentice Hall International
.
56.
Wilkinson, L.
and the
Task Force on Statistical Inference, American Psychological Organization,
Science Directorate1999:
Statistical methods in psychology journals: guidelines and explanations
. American Psychologist54,
594-604
.
57.
Yamashita, S. O.1996: Six measures of JSL pragmatics.
Hawaii: University of Hawaii Press
.
