Abstract
Diagnostic language assessment (DLA) has emerged as a major topic of interest among language testers, as evidenced by recent publications of journal articles and special journal issues on this topic (Alderson, Brunfaut, & Harding, 2014; Lee & Sawaki, 2009). Two fundamental goals of DLA are to identify language learners’ weaknesses and deficiencies, as well as their strengths, in the targeted language domains and provide useful diagnostic feedback and guidance for remedial learning and instruction. In other words, DLA seeks to promote further learning designed to address the test-takers’ weaknesses and increase their overall growth potential. Thus, it is important to create meaningful linkages between outcomes of diagnosis and subsequent learning and instruction when designing the DLA system.
At the moment, DLA, as a subfield of language assessment, is at an important juncture on its course of development and evolution. In order to advance the field beyond where it stands, breakthroughs can be made on multiple fronts, which include, but are not limited to, refining frameworks and methodology for diagnosis, feedback, and guidance for remedial instruction. One of the urgent issues is to come up with workable frameworks of DLA (whether they are theoretical or practical) that can guide the whole process of designing, developing, implementing, and validating DLA. These require us not only to review previous work and ongoing developments in DLA but also to look into, and gain insights from, various related fields of inquiry, such as the following: (a) fields where diagnosis is frequently practiced; (b) dynamic language assessment; (c) cognitive diagnostic assessment models; (d) technological innovations in assessment and scoring; and (e) feedback research in second language acquisition and writing.
The primary goal of this special issue is to bring together expertise and insights from various fields related to DLA, with a view to providing a focused forum through which current thinking and ideas about DLA are actively shared among researchers and practitioners and thereby facilitating development of a shared understanding among language testers of where the whole field of DLA should be moving in the future. This issue consists of six articles, most of which are either revised versions or spin-off versions of the papers that were presented at an invited symposium at LTRC (Language Testing Research Colloquium) 2013 held in Seoul, Korea. The authors of the six papers explore various issues arising from diagnostically oriented language assessments and envision the future of DLA based on what we have learned from various research and development activities.
In the first, introductory article, which is entitled “Diagnosing diagnostic language assessment,” Yong-Won Lee reviews the current state of DLA and identifies promising areas of future research and development deserving language testers’ attention in conjunction with further advancing the field of DLA. He attempts to clarify the existing definitions and some essential characteristics of DLA and the relationships among its major components (diagnosis, feedback, and remedial learning) and identify challenging issues and difficulties associated with each of these major components. In addition, he urges language testers to explore a number of important topics for future research and development, which include the following: multi-staged diagnosis and multi-layered feedback systems; more active utilization of learners’ misconceptions and errors in refining diagnosis and feedback; and the expansion of the validity framework for DLA.
In the second article entitled, “Diagnostic assessment of reading and listening in a second or foreign language: Elaborating on diagnostic principles,” Luke Harding, Charles Alderson, and Tineke Brunfaut provide elaborations on, and a worked-out example of, how principle-based, diagnostic assessment procedures and instruments can be designed and used to identify the learners’ strengths and weaknesses in the context of second or foreign language (SFL) reading and listening assessment. Drawing on their previous work (Alderson et al., 2014), Harding and his colleagues attempt to apply and illustrate five important principles of DLA in SFL listening and reading assessment, which they had previously identified based on their review of diagnostic practices in various fields. In particular, they explore what types of tools and tests might usefully be developed for use within the diagnostic assessment process. The authors also propose a four-step diagnostic process for SFL listening assessment, which includes the listening/observation, initial assessment, hypothesis testing, and decision-making stages.
The third article, entitled “Computerized dynamic assessment (C-DA): Diagnosing L2 development according to learner responsiveness to mediation,” is co-authored by Matthew Poehner, Jie Zhang, and Xiaofei Lu. Building on the ideas of the zone of proximal development and mediation, they have developed online, diagnostic L2 Chinese reading and listening comprehension tests in multiple-choice format. In their tests, each test item is accompanied by a set of graduated prompts arranged from most implicit to most explicit, which are provided to the learners one after the other when they need it. The dynamic nature of the test format makes it possible to obtain not only the actual number-correct and mediated scores (the score they achieved with the help of external mediation) but also the learning potential score for the learners. The results of their study suggest that the set of automatically generated scores, together with learner performance profile data, can provide not only a fine-grained diagnosis of the learners’ L2 development but also detailed feedback for subsequent teaching and learning.
In the fourth article, entitled “Mediation of goal orientations and perceived ability on junior students’ responses to diagnostic feedback,” Eunice E. Jang, Margaret Dunlop, Gina Park, and Edith Vander Boom report the findings from the second phase of their large-scale research study. In this article, the authors examine the relationships between the goal orientations and perceived ability of Canadian elementary school students and their estimated cognitive reading skill profiles obtained from cognitive diagnosis modeling (CDM) analysis, and thereby illustrate how diagnostic feedback is perceived, interpreted, and processed by the students with different goal orientations and perceived abilities in relation to remedial action. The results of their analyses show that the learners’ goal orientations and perceived abilities, which seem to be influenced by their parents’ and teachers’ goals and expectations, do impact the learners’ perceptions, interpretations, and uses of diagnostic feedback. Based on this finding, they recommend that language testers need to consider a variety of psychological factors carefully when they design and evaluate feedback systems for DLA.
In the fifth article, entitled “Diagnostic assessment with automated writing evaluation: A look at validity arguments for new classroom assessments,” Carol Chapelle, Elena Cotos, and Jooyoung Lee outline a useful validity argument framework for an automated writing evaluation (AWE) system with a capability of individualized diagnostic feedback to the learners. One of their core principles is that a validity argument can be developed for the diagnostically oriented AWE by collecting theoretical rationales and empirical evidence to support the use of the assessments in particular learning or instructional contexts. In the paper, the authors demonstrate the usefulness of such a framework by creating interpretive arguments for two AWE-based diagnostic assessments and illustrating various types of data that can be collected and used as backing for the major assumptions supporting the interpretive framework in the process of developing validity arguments for the AWE assessment.
In the sixth article, entitled “Design in four diagnostic language assessments,” Alister Cumming reviews the four preceding articles in this volume from the perspective of design-based research to offer an analysis and synthesis of the articles. His analysis of the four studies uses Anderson and Shattuck’s (2012) six guiding principles of design-based research, which include real educational contexts, significant intervention, mixed research methods, multiple iterations, collaborative partnerships, and practical impact. Cumming observes that the four studies are different in many respects but complement one another by illustrating ways in which a systematic investigation can be conducted into DLA. He concludes that DLA in general tends to be oriented more towards specific-purpose assessments rather than general proficiency testing.
In this volume, the authors of the six articles collectively attempt to examine a range of issues arising from DLA and envision its future based on lessons learned from various research activities. I hope that this special issue will help interested readers to develop a general sense of where the whole field of DLA stands at the moment, what major challenges are ahead of us, and in what direction we should be moving to tackle them. There is one little issue worth mentioning here, though, in relation to the terminological unity across these six papers. One should note that, although they are all put together under a broad umbrella of DLA, the collection of these six articles represents quite an array of approaches to diagnostically oriented language assessment. Since these approaches are grounded in different theoretical and practical frameworks, it seems difficult, and more precisely premature, particularly in the current stage of integration among these DLA approaches, to impose common terminology and uniform language across the six articles. I envision that this special issue will create an important momentum, and a starting point, for the researchers (including the authors) to cross-fertilize ideas and develop standard terminology for DLA, which will eventually facilitate further integration among these related but still somewhat distinct perspectives on DLA into a coherent and unified framework. Finally, it is also my sincere wish that the collection of articles in this volume will not only challenge but also inspire many language testers to join in continued efforts to further develop, evaluate, revise, and refine both the theoretical frameworks and practical applications of DLA in the years to come.