Advances in natural language processing (NLP) and automatic speech recognition and processing technologies offer new opportunities for language testing. Despite their potential uses on a range of language test item types, relatively little work has been done in this area, and it is therefore not well understood by test developers, researchers or users in language assessment. This paper introduces NLP for language assessment as an area of inquiry and practice by describing the historical roots coming from computational linguistics, statistical NLP, speech recognition and processing technologies, language assessment, and computer-assisted language learning. It outlines uses of NLP and speech recognition and processing technologies in language assessment through illustrations of current testing projects, and identifies areas in need of further development.
Alderson JC ( 1987). Innovation in language testing: Can the microcomputer help? Language Testing Update Special Report, No. 1. Lancaster : Centre for Research in Language Education, Lancaster University.
2.
Alderson JC, Percsich R., and Szabo G. ( 2000). Sequencing as an item type. Language Testing, 17(4), 423-447.
3.
Attali Y., Burnstein J. ( 2006). Automated essay scoring with e-rater® V.2.0. Journal of Technology, Learning, and Assessment, 4(3). Retrieved on June 22, 2010 from http://www.jtla.org.
4.
Bailin A (ed.) (1989). Computers and the humanities (Special Issue on Intelligent Computer Assisted Language Instruction), 23, 59-70.
5.
Bennett RE ( 1993). On the meanings of constructed responses. In RE Bennett and WC Ward (eds.), Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment (pp. 1-27). Hillsdale, NJ: Lawrence Erlbaum.
6.
Bennett RE, Bejar II ( 1998). Validity and automated scoring: It’s not only the scoring. Educational Measurement: Issues and Practice, 17, 9-17.
7.
Bernstein J. ( 1999). PhonePass TM testing: Structure and construct. Menlo Park, CA: Ordinate.
8.
Burstein J. ( 2003). The e-rater® scoring engine: Automated essay scoring with natural language processing. In MD Shermis , and JC Burstein (eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 113-121). Mahwah, NJ: Lawrence Erlbaum.
9.
Chanier T., Pengelly M., Twidale M., and Self J. ( 1992). Conceptual modelling in error analysis in computer-assisted language learning systems. In ML Swartz , and M Yazdani (eds.), Intelligent tutoring systems for foreign language learning (pp. 125-150). Berlin: Springer-Verlag.
10.
Chapelle CA ( 2003). English language learning and technology: Lectures on applied linguistics in the age of information and communication technology . Amsterdam: John Benjamins.
11.
Chapelle CA, Douglas D. ( 2006). Assessing language through computer technology. Cambridge: Cambridge University Press.
12.
Chapelle CA, Enright MK, and Jamieson JM ( 2008). Building a validity argument for the Test of English as a Foreign LanguageTM. New York/ London: Routledge.
13.
Clauser BE, Kane MT, and Swanson DB ( 2002). Validity issues for performance-based tests scored with computer-automated scoring systems. Applied Measurement in Education, 15(4), 413-432.
14.
Corbel C. ( 1993). Computer-enhanced language assessment. Sydney, Australia: National Centre for English Language Teaching and Research, Macquarie University.
15.
Cotos E., Pendar N. ( 2008). Automated diagnostic writing test: Why? How? In CA Chapelle, Y-R Chung , and J Xu (eds.), Towards adaptive CALL: Natural language processing for diagnostic language assessment (pp. 65-81). Ames, IA: Iowa State University.
16.
Cullingford RE ( 1986). Natural language processing: A knowledge-engineering approach. Totowa, NJ: Rowman & Littlefield.
17.
Deerwester S., Dumais S., Landauer T., Furnas G., and Harshman R. ( 1990). Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6), 391-407.
18.
Dikli S. ( 2006). An overview of automated scoring of essays. The Journal of Technology, Learning, and Assessment, 5(1), 3-35.
19.
Dobrin DN ( 1990). A limitation on the use of computers in composition. In DH. Holdstein and CL Selfe (eds.), Computers and writing: Theory, research, practice (pp. 40-57). New York: Modern Language Association of America.
20.
Dumais ST, Furnas GW, Landauer TK, Deerwester S., and Harshman R. ( 1988). Using Latent Semantic Analysis to improve access to textual information. In Human factors in computing systems, CHI’88 conference proceedings (Washington, D.C.), May (pp. 281-285). New York: ACM.
21.
Dunkel A. ( 1991). Research on the effectiveness of computer-assisted instruction and computer-assisted language learning. In P Dunkel (ed.), Computer-assisted language learning and testing (pp. 5-36). New York: Newbury House.
22.
Elliot S. ( 2003). IntelliMetricTM: From here to validity. In MD Shermis and JC Burstein (eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 71-86). Mahwah, NJ: Lawrence Erlbaum.
23.
Embretson SE ( 1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179-197.
24.
Enright M., Quianlan T. ( 2010). Complementing human judgment of essays written by English language learners with e-rater® scoringLanguage Testing , 27(3): 317-334.
25.
Felshin S. ( 1995). The Athena language learning project NLP system: A multilingual system for conversation-based language learning. In V. M. Holland, J. Kaplan, and M. Sams (eds.), Intelligent language tutors: Theory shaping technology (pp. 257-272). Hillsdale, NJ: Lawrence Erlbaum.
26.
Hart RS ( 1981). Language study and the PLATO system. Studies in Language Learning, 3, 1-24.
27.
Heift T., Schulze M. ( 2007). Errors and intelligence in computer-assisted language learning: Parsers and pedagogues. New York/ London: Routledge.
28.
Jurafsky D., Martin JH ( 2000). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Upper Saddle River, NJ: Prentice Hall.
29.
Kane MT ( 2006). Validation. In RL Brennan (ed.), Educational measurement (4th ed., pp. 18-64). Westport, CT: American Council on Education/Praeger.
30.
Kane MT, Crooks T., and Cohen A. ( 1999). Validating measures of performance. Educational Measurement: Issues and Practice, 18(2), 5-17.
31.
Keith TZ ( 2003). Validity of automated essay scoring systems. In MD Shermis, and JC Burstein (eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 147-207). Mahwah, NJ: Lawrence Erlbaum.
Landauer TK, Laham D., and Foltz PW ( 2002). The Intelligent Essay Assessor. IEEE Intelligent Systems, 15(5), 27-31.
34.
Landauer TK, Laham D., and Foltz PW ( 2003). Automated scoring and annotation of essays with the Intelligent Essay AssessorTM. In MD Shermis and JC Burstein (eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 87-112). Mahwah, NJ: Lawrence Erlbaum.
35.
Madsen HS ( 1991). Computer adaptive testing of listening and reading comprehension: The Brigham Young University approach. In P Dunkel (ed.), Computer assisted language learning and testing: Research issues and practice (pp. 237-257). New York: Newbury House.
36.
Marty F. ( 1981). Reflections on the use of computers in second-language acquisition. Studies in Language Learning, 3, 25-53.
37.
Mislevy R., Chapelle CA, Chung YR, and Xu J. ( 2008). Options for adaptivity in computer-assisted language learning and assessment. In CA Chapelle, Y-R Chung, and J Xu (eds.), Towards adaptive CALL: Natural language processing for diagnostic language assessment (pp. 9-24). Ames, IA: Iowa State University.
38.
Molholt G., Presler A. ( 1986). Correlation between human and machine ratings of Test of Spoken English reading passages. In C Stansfield (ed.), Technology and language testing (pp. 111-128). Washington, DC: TESOL Publications.
39.
Page EB ( 2003). Project Essay Grade: PEG. In MD Shermis, and JC Burstein (eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 43-54). Mahwah, NJ: Lawrence Erlbaum.
40.
Phillips SM ( 2007). Automated essay scoring: A literature review. Available from www.saee.ca/pdfs/036.pdf.
41.
Pusack JP ( 1983). Answer-processing and error correction in foreign language CAI. System, 11(1), 53-64.
42.
Quirk R., Greenbaum S., Leech S., and Svartik J. ( 1985). A comprehensive grammar of the English language. New York: Longman.
43.
Reid J. ( 1986). Using the Writer’s Workbench in composition teaching and testing. In CW Stansfield (ed.), Technology and language testing (pp. 167-186). Washington, DC : TESOL Publications.
44.
Rudnerz L., Garcia V., and Welch C. ( 2005). An evaluation of Intellimetric TM essay scoring system using responses to GMAT® AWA prompts (GMAC research report number RR-05-08) . Retrieved December 27, 2008, from www.vantagelearning.com/docs/intellimetric/IM_ReseachSummary_IntelliMetric_Accuracy_Across_Genre_and_Grade_Levels.pdf .
45.
Rudner LM, Liang T. ( 2002). Automated essay scoring using Bayes’ Theorem. The Journal of Technology, Learning, and Assessment, 1(2), 3-21.
46.
Stansfield C (ed.) (1986). Technology and language testing. Washington, DC: TESOL Publications.
47.
Tanimoto SL ( 1987). The elements of artificial intelligence. Rockville, MD: Computer Science Press.
48.
Tennant H. ( 1981). Natural language processing: An introduction to an emerging technology. New York: Petrocelli Books.
49.
Townshend B., Todic O. ( 1999). Comparison of PhonePassTM Testing with the Educational Testing Service Test® of Spoken EnglishTM (TSE®). Menlo Park, CA: Ordinate.
50.
Underwood J. ( 1984). Linguistics, computers, and the language teacher. Rowley, MA: Newbury House.
51.
Valenti S., Nitko A., and Cucchiarelli A. ( 2003). An overview of current research on automated essay grading . Journal of Information Technology Education, 2, 319-329.
52.
Vantage Learning (2006). Research summary: IntelliMetric TM scoring accuracy across genres and grade levels . Retrieved December 27, 2008 from www.vantagelearning.com/docs/intellimetric/IM_ReseachSummary_IntelliMetric_Accuracy_Across_Genre_and_Grade_Levels.pdf .
53.
Wresch W. ( 1993). The imminence of grading essays by computer - 25 years later. Computers and Composition, 10(2), 45-58.
54.
Xi X. ( 2008). What and how much evidence do we need? Critical considerations in validating an automated scoring system. In CA Chapelle, YR Chung, and J Xu (eds.), Towards adaptive CALL: Natural language processing for diagnostic language assessment (pp. 102-114). Ames, IA: Iowa State University.
55.
Xi X., Higgins D., Zechner K., and Williamson DM ( 2008). Automated scoring of spontaneous speech using SpeechRater v1.0 (ETS Research Rep. No. RR-08-62). Princeton, NJ: ETS.
56.
Zock M. ( 1996). Computational Linguistics and its use in real world: The case of computer assisted language learning. In Proceedings of the 16th International Conference on Computational Linguistics (COLING) (pp. 1002-1004).
