Abstract
In the editorial to the first issue of Language Testing Journal, which appeared in June 1984, the editors, Arthur Hughes and Don Porter, state that “This new journal has come into being as a forum devoted exclusively to the issues that concern those involved in, or simply interested in, the assessment of language ability in one form or another.” In addition to encouraging a wide writer- and readership, they also assure that “the field covered will be a broad one”, encompassing diverse language teaching learning and testing situations (1984, p. i). This approach is indeed reflected in the range of topics covered in this very first issue: from validity and reliability concerns in language tests, to research on the test taker’s perspective and to testing English as a Second Language versus Mother Tongue and English as a Foreign Language. The contributions to the journal upon its inception mapped out at that point in time the basic areas of research in the language testing terrain, setting the stage for professional developments in the field that have been shaped to a great extent by the journal’s publications. It is thus apt that this current issue, which marks the 30th anniversary of the journal, attempts to examine via the lens of language assessment literacy the knowledge base that underlies and typifies language testing and assessment as a discipline of study.
Discussion on the nature of assessment literacy, a concept introduced with reference to the knowledge assessors need to possess (Stiggins, 1991), reflects an ongoing debate on the nature of the professional knowledge in the field of language testing. The debate has emerged more forcibly as of late due to an increased demand for and use of assessment data by a more diverse group of stakeholders than before, many of whom are novices in the field and have limited or partial knowledge in language testing and assessment and its concerns. The discussion is conducted on inter-related themes: the first focuses on assessment literacy in the language domain, that is, whether there is an agreed-upon theoretical, practical, and experiential knowledge base required for fulfilling assessment and testing functions in language-related situations; the second relates to whether language testing professionals should act as gatekeepers and allow in only those who are considered assessment literate, or whether the profession should reach out to impart differential language assessment knowledge according to needs, to wide circles of applied linguists, teachers, parents, bureaucrats, and politicians. Another issue that is currently being debated vis-à-vis the aforementioned deliberations is the feasibility of forming an agreed-upon hierarchy of knowledge with some topics declared more basic and essential than others, which is required by all stakeholders regardless of their assessment roles and the functions they need to fulfill. These dilemmas get at the core knowledge of the profession, at the identity of language testers, and at the ‘raison d’être’ of the field as a discipline in its own right.
This journal issue includes five research papers that address these dilemmas, each offering a glimpse into language assessment literacy from the perspective of different protagonists involved in language testing situations: from professional testers to teachers, to parliament members and university administrators. It concludes with a contribution by Lynda Taylor who reflects on some of the themes in the field of language assessment literacy as they emerge in the issue and at future directions in this area. The following is a brief description of each of the research studies.
The first study by Scarino focuses on assessment literacy for language teachers. The researcher notes the centrality of assessment in teachers’ professional practices in view of their dual instructional and evaluative roles, and the fact that they are expected to carry out in-house classroom-based assessment while simultaneously abiding by larger accountability paradigms. Scarino argues for the need to expand our understanding of language assessment literacy for teachers and include the assessment perceptions the instructors themselves bring to the teaching and assessment encounter. Evidence from professional teacher development projects is brought forth to illustrate how the assessment knowledge acquisition process is tied up with the teachers’ own professional beliefs, practices, and local context. The importance of allowing teachers to develop their understandings and awareness of their own inner-worlds is emphasized, as well as the implications of these notions to teacher education.
The next two studies by Malone and Jeong focus on language assessment courses, or rather on the perceptions of different populations with differential language testing expertise, on the components that such courses need to include. Analysis of the contents of instructional language testing and assessment frameworks can provide a useful tool for identifying the features of language assessment literacy at a given point in time, as the contents are indicative of current understandings of the scope of knowledge and skills language specialists should possess. The process is a recursive and dynamic one, as research on the course contents will contribute to an understanding of assessment literacy, which in turn will inform and feed into future instructional language testing and assessment endeavors.
In the first paper Malone reports on an initiative intended to support the acquisition of assessment literacy among foreign language teachers through an online tutorial. Both language testing experts and language teachers were asked to judge various facets of the course to enhance its development. Findings point at a gap between the reactions of the two groups to the course contents, with the testing experts stressing the need to expand and deepen knowledge on the theoretical underpinnings of the field, versus the teachers who make a case for increasing the volume of the ‘how to’ components in the tutorial. The gap between the views expressed by the two groups raises questions as to the nature of the assessment literacy required in terms of the balance between theory and practice, of core components versus others, and the mode of presentation. The differences in opinion expressed by the two parties lead the researcher to wonder ‘who actually makes the decisions with regard to the most important tenets of language assessment literacy’. This research also brings attention to the opportunities that online internet facilities offer for disseminating assessment knowledge and what such use entails.
While the previous research analyzed comments on course contents by the course participants and professional external observers, the next research by Jeong looks at course contents from the point of view of their designers – the course instructors. The study distinguishes between instructors whose main focus and expertise is in language testing and those who find themselves teaching these courses even though they are not considered language testing professionals, referred to in the research as ‘non-language testers’. Jeong raises the question of whether there is a certain required background in assessment literacy that entitles one to be the instructor of a course on language assessment. She examines how the contents and structure of language testing courses and the choice of the language testing textbooks used are influenced by the professional background of the instructors who teach them, whether they form part of the testing community or not. Her findings point at differences between the two groups with reference to some of the topic areas they deem important, such as the language testers focusing more on technical testing issues and the non-testers emphasizing classroom assessment practices. Similar trends emerged with regard to textbook preferences. The question posed is who qualifies as a language tester and what are the possible ramifications regarding the formation of language assessment literacy of the course participants. An interesting analogy is drawn between the native–non-native construct and the professional tester and non-tester paradigm.
The next two studies look into the language assessment literacy requirements of those stakeholders further removed from the traditional language testing circles, who are nonetheless involved in high-stakes decision-making and are thus in need of becoming assessment literate. The general question posed here, with reference to language assessment literacy, is what knowledge is needed in order to perform specific assessment tasks in particular contexts. The argument presented by the studies is that the quality and the depth of the assessment literacy knowledge the stakeholders possess may contribute to or undermine the quality of the decision-making process. This is particularly significant in situations where the decisions involved affect the lives of individuals participating in the language assessment process (e.g. acceptance to academic studies and employment opportunities).
The first study by O’Loughlin is situated in two Australian universities where admission decisions need to be made based on the scores of the IELTS English proficiency test. The research focuses on the assessment needs of the university administrators charged with carrying out the admission decisions, and the question asked is what knowledge in assessment do the administrators feel they have and what they would like to or feel they need to know more about in order to enhance their performance. This is considered in light of the ‘objective’ designated assessment literacy regarding the test specified by the test producers in accompanying documents and websites. Findings show that in order to make informed decisions administrators need to be more literate with regard to the test upon which the admission decisions are made, its components and the meanings that can be drawn from the scores. In particular the importance of realizing the fallibility of test scores is noted, even when dealing with a widely accredited test, and the significance of relating to additional variables in the admission process. Though the assessment literacy needs in this case are more restricted than those of other decision-makers like teachers, the researcher concludes that a more comprehensive knowledge base than the one presently provided is necessitated for further understanding and to create a more ethical decision-making assessment culture.
The last article by Pill and Harding offers a fascinating account of policy makers’ understanding, or rather what is seen as their lack of understanding, of assessment issues. The research evidence comprises transcripts of a parliamentary inquiry in Australia that looked into the English proficiency requirements for overseas-trained doctors as part of their registration processes. A discourse analysis of the transcripts demonstrates how lack of understanding of both language and testing issues and lack of familiarity with the tools used and with their intentions, can lead to meaningful misconceptions. Such severe misconceptions may result in misinformed and misguided decisions by the policy makers on crucial issues. The value of differential assessment literacy that is matched to particular needs is noted here as in the previous research, as well as the importance of enlightening the public on assessment issues.
How then do the research studies relate to the questions previously posed with regard to the nature of language assessment literacy?
What is apparent from the articles is that language assessment literacy is a multilayered entity and that defining it presents a major challenge. Clearly it is akin in many ways to educational measurement and affected by current paradigms in this area. It is not clear, however, which of the measurement issues are general and which are distinctive and unique and focus on the language learning and assessment field and its concerns, what is the relative balance amongst them, and what can be considered basic, agreed-upon themes that typify knowledge in this discipline and set it apart. The difficulty of reaching an agreed-upon definition reflects the lack of consensus within the professional testing community as to what constitutes the assessment knowledge that will be disseminated to future experts in the field. The following sections will relate to these aspects in terms of the breadth of the approach offered, the language component in this equation, and considerations as to who decides on what constitutes language assessment literacy and on the essentials within this knowledge base.
Breadth of the approach
What emerges from the studies is that the breadth of the approach taken and the ensuing scope and nature of the assessment literacy required, depend on the assessment circumstances and on the protagonists’ roles. Some of the quoted experts in the articles relate to the knowledge base mostly in terms of measurement characteristics and test design, as was evident in Jeong’s research, while others take a broader view emphasizing the social aspects of assessment and the relevance of the context or local culture within which it is practiced. This latter approach is inherent in the paradigm represented by Scarino, who looks at the individual teachers’ perceptions on assessment, and how they impact their understanding and implementation of classroom assessment practices.
A broad, conceptual, socially embedded orientation is also reflected in the O’Loughlin article, which advocates for an expanded paradigm that reaches beyond mere test scores and takes a critical perspective. Language assessment literacy according to O’Loughlin “potentially includes the acquisition of a range of skills related to test production, test score interpretation and use, and test evaluation in conjunction with the development of a critical understanding about the roles and functions of assessment within education and society” (2013, p. 363).
This view coincides with the expanded working definition of language assessment literacy offered recently by Fulcher (2012), based on a study designed to elicit the assessment training needs of language teachers. The definition provides a combination of measurement knowledge and assessment know-how with the social perspective.
The knowledge, skills and abilities required to design, develop, maintain or evaluate large-scale standardized and/or classroom based tests, familiarity with test processes, and awareness of principles and concepts that guide and underpin practice, including ethics and codes of practice. The ability to place knowledge, skills, processes, principles and concepts within wider historical, social, political and philosophical frameworks in order understand why practices have arisen as they have, and to evaluate the role and impact of testing on society, institutions, and individuals. (Fulcher, 2012, p. 125)
Pill and Harding (2013, p. 381), whose research considers the knowledge base of decision-makers removed from the testing community, offer an all-inclusive definition of assessment literacy in terms of the participants and their responsibilities. The definition includes all individuals who are engaged in assessment functions related to language use, either as knowledgeable users or creators: “language assessment literacy may be understood as indicating a repertoire of competences that enable an individual to understand, evaluate and, in some cases, create language tests and analyze test data”.
The language component
Not all the definitions relate to the ‘language’ component in language assessment literacy, to the unique traits of language learning, and hence the formation of compatible assessment strategies and tools. Interestingly and perhaps quite naturally, the research studies in this issue which see language teachers as their focal point, do emphasize the language elements as part of their understanding of the term. Malone’s conception of assessment literacy is more classroom-focused, and refers to language teaching: “familiarity with measurement practices and the application of this knowledge to classroom practices in general and specifically to issues of assessing language”. This matches the contents of the online tutorial, which includes both general assessment concepts and language-focused considerations. This matter is also taken up by Davison and Leung in their 2009 survey of teacher-based assessment (TBA). The knowledge-base framework advocated includes understanding of language-related competencies as they have been discussed in communicative models:
First, TBA, in its emphasis on language use in context, calls into action a multifaceted combination of linguistic, pragmatic, and cultural resources. Models of second language competence (e.g., Bachman, 1990; Canale & Swain, 1980) have set out components such as grammatical competence, sociocultural competence, and strategic competence. (Davidson & Leung, 2009, p. 406)
The absence of the ‘language’ trait from some of the definitions raises the question of whether just as ‘non-testing’ experts can instruct in language assessment literacy, so can ‘non-language’ experts whose expertise lies in issues of educational measurement but not necessarily in language-related spheres. This also implies that the tradition of providing courses in assessment specifically tailored to language specialists, such as language teachers, is not mandatory, as the topics are generic and cut across subject areas. Language teachers would thus partake as part of their training in general courses that offer the required know how and skills in assessment literacy along with teachers from other subject areas. To illustrate this predicament an analogy can be drawn to the seminal teacher knowledge framework introduced by Shulman in 1987. The model specifies seven teacher knowledge components. Among them are content knowledge, which is deemed specific per subject area, and pedagogical knowledge, a general trait required by all the practitioners regardless of their topic area. However, a third knowledge component is included, one which represents a merger between the content and pedagogy, referred to as pedagogical content knowledge (PCK). Acquiring this latter competency ensures that the practitioners will have specific expert knowledge in utilizing and implementing general pedagogical skills in their own context. A similar argument can be made for the need to merge general assessment knowledge with language-related expertise, hence creating a special knowledge base, a prerequisite for assessing language abilities.
Who will decide?
Not only is there divergence in defining the knowledge base as can be seen above, it is also unclear who amongst language testers, teachers, professionals or non-experts, is granted the authority to determine the content knowledge. That is whose point of view is the ‘correct’ one, and whether there is room for wider interpretations on what constitutes language assessment literacy. This comes across quite forcibly in the Jeong article, experts versus non-experts, but also in the other studies. Will it be the publishers of the ILTS test specifications or the users in the O’Loughlin study? Will it be the expert testers or practitioners asked to review and react to the online tutorial (Malone, 2013, p. 329) And who will draw the lines in the case of the knowledge legislators need so as to make valid decisions (Pill and Hardy, 2013, p. 381) This matter is accentuated when one considers the perception gaps between the different groups requested in each of the studies to decree and announce their ruling, assuming there needs to be one. The significant issue here is whether and to what extent the profession might be willing, as part of the reaching-out agenda, to listen to other voices, to re-examine set beliefs in view of current dynamic assessment needs. The non-testing background of the instructors, administrators, and others may hold great potential for creating a rich interface between different areas of expertise and the knowledge base of the language testing profession, similar to the interface notions explored in the Bachman and Cohen (1998) publication between testing and second language acquisition research. The diverse characteristics of this new language testers’ community can be viewed as a means to facilitate a dialogue between testing professionals and practitioners on the language assessment literacy components they would find most useful.
What are the essentials?
At last we are left with the question of whether it is possible to determine the essential basic components of language assessment literacy to be mastered by all users regardless of the language assessment context. What seems to motivate the studies is the existence of a multi-dimensional language assessment literacy continuum, which comprises differential levels of expertise in the relevant language assessment knowledge competencies: knowledge about assessment, knowledge about language, knowledge about the context, and the ability to design, collect, administer, or interpret assessment data in order to make sensible ethical decisions. What is thus offered is a cluster of continua for each knowledge component. The depth of knowledge in each category would vary and become more or less intense depending on the stakeholders’ needs. However, according to the paradigm expressed in the research studies presented, the threshold level needs to include a basic understanding of the situated approach towards language testing and awareness of the consequential validity of the assessment action taken. Though university administrators, for example, says O’Loughlin, cannot be expected to acquire and fathom the historical development of language testing “a better knowledge of the processes, principles and concepts of language testing would enable university staff to exercise their responsibilities as informed test users, and, hopefully, to become advocates of valid and ethical practice” (2013, p. 378).
In sum, the diversity noted in the first editorial of the journal in 1984 still reigns, and issues initially discussed then are far from being resolved almost three decades later. Many questions remain unanswered and more research is needed to facilitate and fuel this ongoing discussion, to provide some insights to the above-mentioned dilemmas, to substantiate and validate, as well as question, the current knowledge base in language testing and assessment, the generic and the unique.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
