Most modern language testers believe that the writing of a successful instrument
begins with specifications, and that instruments constructed devoid of
specifications are likely to go astray. The purpose of this study was to explore the
relative impact of the test-developer on the performance of test-takers using
multiple-choice reading comprehension tests that had no specifications. Traditional
reading comprehension tests often consist of short prose passages followed by sets
of multiple-choicecomprehension questions. The test constructor formulates the stem
and the choices on the basis of his or her own renditions of the passage. The
test-takers provide evidence of their comprehension in terms of their written
responses to the items designed by the test constructor. Since the characteristics
of the test method limit the responses the testees can provide, the expected
response becomes part of the test method. Accordingly, it seems reasonable to
suggest that there may be a facet associated with the test-developer.Six passages
each with 3 different sets of multiple-choice items constructed by 3 (groups of)
individuals were trialled on 335 Iranian EFL (English as a foreign language)
learners. The results revealed differential performance on almost all sets,
suggesting a test constructor effect.
Get full access to this article
View all access options for this article.
References
1.
Af‘ erbach, P.1986: The influence of prior knowledge on expert readers’
main idea construction processes.
Newark, DE: International Reading Association
(ED 284 193)
.
2.
Alderson, J. C.1983: The cloze procedure and proficiency in English as a foreign
language. In
Oller, John W., Jr.
, editor, Issues in language testing research.
Rowley, MA: Newbury House
, 205-217.
3.
Alderson, J. C.2000: Assessing reading.
Cambridge: Cambridge University Press
.
4.
Alderson, J. C.
,
Clapham, C.
and
Wall, D.1995: Language test construction and evaluation.
Cambridge: Cambridge University Press
.
5.
Alderson, J. C.
and
Urquhart, A. H.1985:
The effect of students’ academic discipline on their performance
on ESP reading tests
. Language Testing2,
192-204
.
6.
Anderson, N. J.
,
Bachman, L.
,
Perkins, K.
and
Cohen, A.1991:
An exploratory study into the construct validity of a reading comprehension
test: triangulation of data sources
. Language Testing8,
41-66
.
7.
Bachman, L. F.1985:
Performance on the cloze tests with fixed-ratio and rational deletions
. TESOL Quarterly19,
535-556
.
8.
Bachman, L. F.1990: Fundamental considerations in language testing.
Oxford: Oxford University Press
.
9.
Bachman, L. F.2000:
Modern language testing at the turn of the century: assuring that what we
count counts
. Language Testing17,
1-42
.
10.
Bachman, L. F.
,
Davidson, F.
and
Milanovic, M.1996:
The use of test method characteristics in the content analysis and design
of EFL proficiency tests
. Language Testing13,
125-150
.
11.
Bachman, L. F.
and
Palmer, A. S.1996: Language testing in practice: designing and developing useful
language tests.
Oxford: Oxford University Press
.
12.
Bensoussan, M.
,
Goldenblatt, L.
and
Kreindler, I.1984:
Changing the difficulty level of multiple-choice EFL reading comprehension questions
. Language Testing1,
105-109
.
13.
Bernhardt, E. B.1985: Reconstructions of literary texts by learners of
German. In
Heid, M.
, editor, New Yorker Werkstattgesprach 1984: Literarische Texte im
Frendsprachenuntericht.
München: Kemmler and Hoch
, 254-289.
14.
Bormuth, J. R.1967:
Comparable cloze and multiple-choice test scores
. Journal of Reading10,
291-299
.
15.
Bügel, K.
and
Buunk, B. P.1996:
Sex differences in foreign language text comprehension: the role of
interests and prior knowledge
. Modern Language Journal80,
15-31
.
16.
Carrell, P. L.
and
Eisterhold, J. C.1983:
Schema theory and ESL reading pedagogy
. TESOL Quarterly17,
553-573
.
17.
Chihara, T.
,
Sakurai, T.
and
Oller, J. W., Jr.1989:
Background and culture as factors in EFL reading comprehension
. Language Testing6,
143-151
.
18.
Clapham, Caroline1996: Studies in language testing 4: the development of IELTS: a
study of the effect of background knowledge on reading comprehension.
Cambridge: Cambridge University Press
.
19.
Clarke, M. A.
and
Silberstein, S.1977:
Toward a realization of psycholinguistic principles in the ESL reading class
. Language Learning27,
135-154
.
20.
Davey, B.1988:
Factors affecting the difficulty of reading comprehension items for
successful and unsuccessful readers
. Experimental Education56,
67-76
.
21.
Davey, B.
and
Lasasso, C.1984:
The interaction of reader and task factors in the assessment of reading comprehension
. Experimental Education52,
199-206
.
22.
Davies, A.1990: Principles of language testing.
Oxford: Blackwell
.
23.
Drum, P. A.
,
Calfee, R. C.
and
Cook, L. K.1981:
The effects of surface structure variables on performance in reading
comprehension tests
. Reading Research Quarterly16,
486-514
.
24.
Educational Testing Service1995: TOEFL practice tests.
Princeton, NJ: Educational Testing Service
.
25.
Farhady, H.
,
Jafarpur, A.
and
Birjandi, P.1994: Testing language skills: from theory to practice.
Tehran: SAMT Publications
.
26.
Fortus, R.
,
Corriat, R.
and
Fund, S.1998: Prediction of item difficulty in the English subtest of
Israel’s inter-university psychometric entrance test. In
Kunnan, A. J.
, editor, Validation in language assessment.
Mahwah, NJ: Lawrence Erlbaum
, 61-87.
27.
Freedle, R.
and
Kostin, I.1993:
The prediction of TOEFL reading item difficulty: implications for construct validity
. Language Testing10,
133-170
.
28.
Freedle, R.
and
Kostin, I.1999:
Does the text matter in a multiple-choice test of comprehension? The case
for the construct validity of TOEFL’s minitalks
. Language Testing16,
2-32
.
29.
Gipps, C.
and
Murphy, P.1994: A fair test? Assessment, achievement and equity?
Buckingham: Open University Press
.
30.
Hammadou, J.1991:
Interrelationships among prior knowledge, inference and language
proficiency in foreign language reading
. Modern Language Journal75,
27-38
31.
Harris, D. P.1969: Testing English as a second language.
New York: McGraw-Hill
.
32.
Harris, D. P.
1976:
Testing reading comprehension in ESL: background and current state of the art
. In
Morely, J.
, editor. Papers in ESL: selected conference papers of the Association
of Teachers of English as a Second Language. Washington, DC,
NAFSA,
25-30
.
33.
Harrison, A.1983: A language testing handbook.
Bethleham, PA: ELTS
.
34.
Hudson, T.1982:
The effects of induced schemata on the ‘short circuit’
in L2 reading: non-decoding factors in L2 reading performance
. Language Learning32,
2-31
.
35.
Hughes, A.1989: Testing for language teachers.
Cambridge: Cambridge University Press
.
36.
Johnson, P.1982:
Effects of reading comprehension on building background knowledge
. TESOL Quarterly16,
503-516
.
37.
Katz, S.
,
Lautenschlager, G.
,
Blackburn, A.
and
Harris, F.1990:
Answering reading comprehension items without passages on the SAT
. Psychological Science1,
122-127
.
38.
Kintsch, W.
and
Yarbrough, J. C.1982:
Role of rhetorical structure in text comprehension
. Educational Psychology74,
828-834
.
39.
Klein-Braley, C.1983: A cloze is a cloze is a question. In
Oller, J. W., Jr.
, editor, Issues in language testing research.
Rowley, MA: Newbury House
, 218-228.
40.
Lee, J. F.1986:
Background knowledge in L2 reading
. Modern Language Journal70,
350-354
.
41.
Lee, J. F.1987:
Comprehending the Spanish subjunctive: an information processing perspective
. Modern Language Journal71,
50-57
.
42.
Lutje Spelberg, H.
,
De Boer, P.
and
Van den Bos, K. P.2000:
Item type comparisons of language comprehension tests
. Language Testing17,
311-322
.
43.
Lynch, B.1992: Evaluating a program inside and out. In
Alderson, J. C.
and
Beretta, A.
, editors, Evaluating second language education.
Cambridge: Cambridge University Press
, 61-99 and 350-366.
44.
Pearson, P. D.
and
Johnson, D. D.1978: Teaching reading comprehension.
New York: Holt, Rinehart and Winston
.
45.
Perkins, K.1992:
The effect of passage topical structure types on ESL reading comprehension difficulty
. Language Testing9,
163-172
46.
Perkins, K.
and
Brutten, S. R.1993: A model of ESL reading comprehension difficulty.
In
Huhta, A.
,
Sajavaara, K.
and
Takala, S.
, editors, Language testing: new openings.
Jyväskylä: University of Jyväskylä
, 205-218.
47.
Perkins, K.
,
Gupta, L.
and
Tammana, R.1995:
Predicting item difficulty in a reading comprehension test with an
artificial neutral network
. Language Testing12,
34-53
.
48.
Riley, G. L.
and
James, F. L.1996:
A comparison of recall and summary protocols as measures of second language
reading comprehension
. Language Testing13,
173-189
.
49.
Royer, J.1990: The sentence verification technique: a new direction in the
assessment of reading comprehension. In
Legg, S.
and
Algina, J.
, editors. Cognitive assessment of language and math outcomes.
Norwood, NJ: Ablex
, 144-191.
50.
Sasaki, M.2000:
Effects of cultural schemata on students’ test-taking processes
for cloze tests: a multiple data source approach
. Language Testing17,
85-114
.
51.
Shohamy, E.1984:
Does the testing method make a difference? The case of reading comprehension
. Language Testing1,
147-170
.
52.
Spolsky, B.1978: Introduction: linguists and language testers. In
Spolsky, B.
, editor, Advances in language testing series: 2: approaches to language
testing.
Arlington, VA: Center for Applied Linguistics
, pp. v-x.
53.
Tan, S. H.1990: The role of prior knowledge and language proficiency as
predictors of reading comprehension among undergraduates. In
de Jong, J. H. A. L.
and
Stevenson, D. K.
, editors. Individualizing the assessment of language abilities.
Clevedon: Multilingual Matters
, 214-244.
54.
Valencia, S. W.
,
Stallman, A. C.
,
Commeryas, M.
and
Hartman, D. K.1991:
Four measures of topical knowledge: a study of construct validity
. Reading Research Quarterly16,
204-233
.
55.
Weir, C.1988: Communicative language testing.
Exeter: University of Exeter
.
56.
Wolf, D.1993:
A comparison of assessment tasks used to measure foreign language reading comprehension
. Modern Language Journal77,
473-489
.
