The important yet under-researched role of item writers in the selection and adaptation of texts for high-stakes reading tests is investigated through a case study involving a group of trained item writers working on the International English Language Testing System (IELTS). In the first phase of the study, participants were invited to reflect in writing, and then audio-recorded in a semantic-differential-based joint discussion, on the processes they employed to generate test material. The group were next observed at a simulated item writers’ editing meeting to refine their texts and items for an IELTS reading test module. The participants’ written descriptions and recorded discussions provided rich data on how source texts were perceived, selected and adapted for the Test. The study reports findings from textual analyses using indices of readability and lexical density from the original material sourced by the item writers and their adapted versions for the test. Results from qualitative and quantitive analyses are discussed in terms of the implications for the IELTS reading module of editing actions such as: reducing redundancy and technical language, changing styles, deciding on potentially sensitive issues and relationships between texts and test items. The important issue of text authenticity in tests such as IELTS is also broached.
AldersonJ. C.ClaphamC.WallD. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press.
2.
ALTE – The Association of Language Testers in Europe (2001). ALTE minimum standards for establishing quality profiles in language assessment. Retrieved from www.alte.org, 11August2010.
3.
BormuthJ. R. (1970). On the theory of achievement test items. Chicago: University of Chicago Press.
4.
ChapelleC.DuffP. (2003). Some guidelines for conducting quantitative and qualitative research in TESOL. TESOL Quarterly, 37(1), 157–178.
5.
CharmazK. (2006). Constructing grounded theory: A practical guide through qualitative analysis. Thousand Oaks, CA: Sage Publications.
6.
ClaphamC. (1996). The academic modules: Reading. In ClaphamC.AldersonJ. C. (Eds.), IELTS Research Report 3: Constructing and trialling the IELTS test. London: The British Council, UCLES and IDP Education Australia.
7.
CoxheadA. (2000). An academic word list. TESOL Quarterly, 34(2), 213–238.
8.
DavidsonF.LynchB. K. (2002). Testcraft: A teacher’s guide to writing and using language test specifications. New Haven, CT: Yale University.
9.
DowningS. M. (2006). Twelve steps for effective test development. In DowningS. M.HaladynaT. M. (Eds.), Handbook of test development (pp. 3–25). Mahwah, NJ: Lawrence Erlbaum.
10.
DowningS. M.HaladynaT. M. (1997). Test item development: Validity evidence from quality assurance procedures. Applied Measurement in Education, 10(1), 61–82.
11.
EbelR. L. (1951). Writing the test item. In LindquistE. F. (Ed.), Educational measurement. Washington, DC: American Council on Education.
12.
EALTA – European Association for Language Testing and Assessment (2006). Guidelines for good practice in language testing and assessment. Adopted 20 May 2006. Retrieved from www.ealta.eu.org, 11 August 2010.
13.
EnrightM. K.GrabeW.KodaK.MosenthalP.Mulcahy-ErntP.SchedlM. (2000). TOEFL 2000 Reading Framework: A Working Paper. TOEFL Monograph Series 17. Princeton, NJ: Educational Testing Service.
14.
FulcherG.DavidsonF. (2007). Language testing and assessment. London: Routledge.
15.
GarlandR. (1990). A comparison of three forms of the semantic differential. Marketing Bulletin, 1(1), 19–24.
16.
GassS. M.MackeyA. (2000). Stimulated recall methodology in second language research. Mahwah, NJ: Lawrence Erlbaum.
17.
GreenA.JayD. (2005). Quality assurance and quality control: Reviewing and pretesting examination material at Cambridge ESOL. Research Notes, 21, 5–7.
18.
GreenA.ÜnaldiA.WeirC. J. (2010). Empiricism versus connoisseurship: Establishing the appropriacy of texts for testing reading for academic purposes. Language Testing, 27(3): 1–21.
19.
HaladynaT. M. (1999). Developing and validating multiple choice test items. Mahwah, NJ: Lawrence Erlbaum.
20.
HaladynaT. M.DowningS. M.RodriguezM. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–334.
21.
HeatonJ. B. (1990). Classroom testing. London: Longman.
22.
HorneJ. A. (2004). Is there a sleep debt?Sleep, 27, 1047–1049.
23.
HughesA. (2003). Testing for language teachers. Cambridge: Cambridge University Press.
IELTS (2007b). The IELTS Handbook 2007. Cambridge: University of Cambridge Local Examinations Syndicate.
26.
IELTS (Cambridge ESOL Examinations, British Council, IDP: IELTS Australia, IELTS International USA) (2009). IELTS: Guide for educational institutions, governments, professional bodies and commercial organisations). Retrieved 10 September 2010from www.ielts.org/pdf/IELTSGuideforStakeholdersMarch2009.pdf.
27.
InghamK. (2008). The Cambridge ESOL approach to Item Writer training: The case of ICFE Listening. Research Notes, 32, 5–9.
28.
International Language Testing Association (2007). Guidelines for practice. Adopted at the annual meeting of ILTA, held in Barcelona, June 2007. Retrieved 11 August 2010from www.iltaonline.com.
29.
IrvineS. H.KyllonenP. C. (2002). Item generation for test development. New York: Lawrence Erlbaum.
30.
KimJ.ChiY.HuenschA.JunH.LiH.RoullionV. (2010). A case study on an item writing process: Use of test specifications, nature of group dynamics, and individual item writers’ characteristics. Language Assessment Quarterly, 7(2), 160–174.
31.
KincaidJ. P.FishburneR. P.RogersR. L.ChissomB. S. (1975). Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Research Branch Report 8–75. Millington, TN: Naval Technical Training, US Naval Air Station, Memphis.
32.
LauferB. (1992). How much lexis is necessary for reading comprehension? In ArnaudP.BéjointH. (Eds.), Vocabulary and applied linguistics (pp. 126–132). London: Macmillan.
33.
NationP.WaringR. (1997). Vocabulary size, text coverage, and word lists. In SchmittN.McCarthyM. (Eds.), Vocabulary: Description, acquisition, pedagogy (pp. 6–19). Cambridge: Cambridge University Press.
34.
NorrisJ. M.BrownJ. D.HudsonT.YoshiokaJ. (1998). Designing second language performance assessments. Honolulu, HI: University of Hawaii.
35.
SalisburyK. (2005). The edge of expertise: Towards an understanding of listening test item writing as professional practice. Unpublished PhD thesis, Kings College London.
36.
SheehanK. M.KostinI.FutagiY.FlorM. (2010). Generating automated text complexity classifications that are aligned with targeted text complexity standards. Princeton, NJ: Educational Testing Service.
37.
WeirC. J. (1993). Understanding and developing language tests. London and New York: Prentice Hall.
38.
WengerE. (1999). Communities of practice: Learning, meaning and identity. Cambridge: Cambridge University Press.
39.
WesmanA. G. (1971). Writing the test item. In ThorndikeR. L. (Ed.), Educational measurement (1st ed., pp. 99–111). Washington, DC: American Council on Education.
