Language Testing

This construct validation study investigated the factor structure of the Test of English as a Foreign Language™ Internet-based test (TOEFL® iBT). An item-level confirmatory factor analysis was conducted for a test form completed by participants in a field study. A higher-order factor model was identified, with a higher-order general factor (ESL/EFL ability) and four first-order factors for reading, listening, speaking and writing. Integrated Speaking and Writing tasks, which require language processing in multiple modalities, defined the target modalities (speaking and writing). These results broadly supported the current practice of reporting a total score and four scores corresponding to the modalities for the test, as well as the test design that permits the integrated tasks to contribute only to the scores of the target modalities.

With recent statistical advances in cognitive diagnostic assessment (CDA), the CDA approach has been increasingly applied to non-diagnostic tests partly to meet accountability demands for student achievement. The study aimed to evaluate critically the validity of the CDA application to an existing non-diagnostic L2 reading comprehension test and to provide information about challenges and conditions for the CDA approach. Based on Jang's study (2005), this paper focuses on the dependability of the Fusion Model's skill profiling, the characteristics of resulting L2 skill profiles, and the diagnostic capacity of LanguEdge™ test items. In addition, the paper examines the validity arguments from the users' perspective by focusing on the usefulness of the diagnostic feedback. The results suggest that the CDA approach can provide more fine-grained diagnostic information about the level of competency in reading skills than traditional aggregated-test scoring can. While various empirical evidence supported the dependability of the skill profiling process, the results also raised some concerns about the application of the CDA approach to a test developed for non-diagnostic purposes, most significantly, a lack of diagnostic capacity of some of the test items with extremely easy or difficult levels. The results offer useful information about the potential challenges and conditions for future application of cognitive diagnostic assessment.

Multifaceted Rasch measurement was used in the present study with 91 student and 4 teacher raters to investigate how self- and peer-assessments work in comparison with teacher assessments in actual university writing classes.

The results indicated that many self-raters assessed their own writing lower than predicted. This was particularly true for high-achieving students. Peer-raters were the most lenient raters; however, they rated high-achieving writers lower and low-achieving writers higher. This tendency was independent of their own writing abilities and therefore offered no support for the hypothesis that high-achieving writers rated severely and low-achieving writers rated leniently. On the other hand, most peer-raters were internally consistent and produced fewer bias interactions than self- and teacher-raters. Each of the four teachers was internally consistent; however, each displayed a unique bias pattern. Self-, peer-, and teacher-raters assessed Grammar severely and Spelling leniently. The analysis also revealed that teacher-raters assessed Spelling, Format, and Punctuation differently from the other criteria.

It was concluded that self-assessment was somewhat idiosyncratic and therefore of limited utility as a part of formal assessment. Peer-assessors on the other hand were shown to be internally consistent and their rating patterns were not dependent on their own writing performance. They also produced relatively few bias interactions. These results suggest that in at least some contexts, peer-assessments can play a useful role in writing classes. By using multifaceted Rasch measurement, teachers can inform peer-raters of their bias patterns and help them develop better quality assessment criteria, two steps that might lead to better quality peer-assessment.

Gist identification or coherent situation model construction performance is an important criterion not only in L1 and L2 reading comprehension assessment but also in every aspect of discourse processing. Whereas most previous L2 research has investigated schematic knowledge of readers about relations within a text, more recent studies have used models of comprehension that also pay attention to external reference and world knowledge. The present study proposes a way to assess gist identification by L2 readers using Hoey's (1991) theory of networks of repetition of lexical items including morphologically similar words and paraphrases of keywords containing the gist of a text. Based on Hoey's tax-onomy, two models were tested. One, following Kintsch (e.g. 1998), assessed the ability to identify the gist of a text using minimal lookbacks. The other encouraged multiple lookbacks. The second model, which involved more reader involvement in finding referential relationships within a text, resulted in more detailed understanding of gist.

Just like buildings, tests are designed and built for specific purposes, people, and uses. However, both buildings and tests grow and change over time as the needs of their users change. Sometimes, they are also both used for purposes other than those intended in the original designs. This paper explores architecture as a metaphor for language test development. Firstly, it describes test purpose and use, and how this affects test design. Secondly, it describes and illustrates the layers of test architecture and design. Thirdly, it discusses the concept of test retrofit, which is the process of altering the test after it has been put into operational use. We argue that there are two types of test retrofit: an upgrade and a change. Each type of retrofit implies changes to layers of the test architecture which must be articulated for a validity argument to be constructed and evaluated. As is true in architecture, we argue that a failure to be explicit about retrofit seriously limits validity claims and clouds issues surrounding the intended effect of the test upon users.