Language Testing

This study describes the adaptation process of a vocabulary knowledge test for British Sign Language (BSL) into American Sign Language (ASL) and presents results from the first round of pilot testing with 20 deaf native ASL signers. The web-based test assesses the strength of deaf children’s vocabulary knowledge by means of different mappings of phonological form and meaning of signs. The adaptation from BSL to ASL involved nine stages, which included forming a panel of deaf/hearing experts, developing a set of new items and revising/replacing items considered ineffective, and piloting the new version. Results provide new evidence in support of the use of this methodology for assessing signed languages, making a useful contribution toward the availability of tests to assess deaf children’s signed language skills.

English language proficiency assessments (ELPA) are used in the United States to measure annually the English language progress and proficiency of English-language learners (ELLs), a subgroup of language minority students who receive language acquisition support mandated and largely funded by Title III (NCLB, 2001). ELPA proficient and non-proficient classifications are determined by applying decision rules to combine the sub-domains of listening, speaking, reading, and writing in a conjunctive, compensatory, mixed or complementary manner in order that an ELP performance standard can be set. Although the ELP performance standard is used to set accountability objectives for federal reporting, it also is used to reveal students’ readiness for exit from English language services. This study operationalizes and tests the ELP performance standard for student-level decision making by describing to what extent students are classified as non-proficient under different models and rules and the effect of these differences on their eligibility for redesignation. Test performances from one state’s ELPA were gathered from a statewide sample of ELL (n = 875) and randomly selected sample of native English speaker students (non-ELL, n = 92) in fifth grade. Findings indicate sizable differences in non-proficient classifications for ELLs, non-ELLs, and a constructed subgroup of academically high-performing students. There were also observed differences in redesignation eligibility in all groups suggesting that choice of model and decision rule can extend the length of time even high-performing students spend in English language services. Discussion includes implications for validation of high-stakes classification systems.

While the research literature on second language (L2) fluency is replete with descriptions of fluency and its influence with regard to English as an additional language, little is known about what fluency features influence judgments of fluency in L2 French. This study reports the results of an investigation that analyzed the relationship between utterance fluency measures and raters’ perceptions of L2 fluency in French using mixed-effects modeling. Participants were 40 adult learners of French at varying levels of proficiency, studying in a university immersion context. Speech performances were collected on three different types of narrative tasks. Four utterance fluency measures were extracted from each performance. Eleven untrained judges rated the speech performances and we examined which utterance fluency measures are the best predictors of the scores awarded by the raters. The mean length of runs and articulation rate proved to be the most influential factors in raters’ judgments, while the frequency of pauses played a less important role. The length of pauses was positively related to fluency scores, indicating a prominent cross-linguistic variation specific to French. The relative importance of the utterance measures in predicting fluency ratings, however, was found to vary across tasks.

Despite rapid growth in literacy-related programmes and evaluation in sub-Saharan Africa, little critical attention has been paid to the relevance of assumptions that underlie existing assessment methods. This study focuses on the issue of timing in the assessment of oral reading fluency, a critical component of successful reading (Chard, Vaughn, & Tyler, 2002; National Institute of Child Health and Human Development (NICHD), 2000; Pikulski & Chard, 2005). Within the context of the Primary Math and Reading Initiative, a randomized controlled trial of several instructional interventions in Kenya, timed and untimed Early Grade Reading Assessment (EGRA) oral reading fluency and reading comprehension tasks were administered to 4385 students in 95 government and 125 informal schools. Using the data from the EGRA – whose administration has expanded within sub-Saharan Africa recently – we found that students did not perform significantly better on the assessments when they had more time. This pattern largely held when we examined the effects disaggregated over student ability level. This suggests that timed assessments, which are faster to administer and logistically easier, are appropriate for use in Kenya.

A short training program for evaluating responses to an essay writing task consisted of scoring 20 training essays with immediate feedback about the correct score. The same scoring session also served as a certification test for trainees. Participants with little or no previous rating experience completed this session and 14 trainees who passed an accuracy threshold proceeded to score other essays. Performance of the newly-trained raters was compared to that of 16 expert raters with extensive experience in scoring responses to the writing task. Results showed that the scores from the newly-trained group of raters exhibited similar measurement properties (mean and variability of scores, reliability and various validity coefficients, and underlying factor structure) to those from the experienced group of raters. Implications for the place of initial training and screening of raters on rater performance are discussed.

Two factors were investigated that are thought to contribute to consistency in rater scoring judgments: rater training and experience in scoring. Also considered were the relative effects of scoring rubrics and exemplars on rater performance. Experienced teachers of English (N = 20) scored recorded responses from the TOEFL iBT speaking test prior to training and in three sessions following training (100 responses for each session). Scores were analyzed using multifaceted Rasch measurement and traditional measures of rater reliability and agreement, and the frequency with which exemplar responses were viewed was measured. Prior to training, rater severity and internal consistency were already of a standard typical for operational language performance tests, but training resulted in increased inter-rater correlation and agreement as well as improved agreement with established reference scores. Additional experience gained after training appeared to have little further effect on raters’ scoring consistency, although the level of agreement with reference scores continued to increase. The most accurate raters generally reviewed exemplar responses more often and took longer to make scoring decisions compared to the least accurate raters. These results raise questions regarding the relative contribution of scoring aids such as exemplars and scoring rubrics to desirable scoring patterns.