Language assessment literacy as self-awareness: Understanding the role of interpretation in assessment and in teacher learning

Abstract

The increasing influence of sociocultural theories of learning on assessment practices in second language education necessitates an expansion of the knowledge base that teacher-assessors need to develop (what teachers need to know) and related changes in the processes of language teacher education (how they learn and develop it). Teacher assessors need to acquire concepts from diverse assessment paradigms; they need to learn to use these concepts in developing, using and analysing assessment procedures and results; they need to exercise critical perspectives on their own assessment practices for particular purposes in diverse contexts, especially in seeking to do justice to all in education. In this paper I argue that, to develop language assessment literacy with the dual goals of transforming teacher assessment practices and developing teacher understanding of the phenomenon of assessment itself and themselves as assessors, it is necessary to reconsider both the knowledge base and the complex processes of language teacher education. I draw on projects I have conducted on developing and investigating teacher understanding and practices in second language assessment, to discuss the need to work with the often tacit preconceptions, beliefs, understandings and world-views about assessment that teacher-assessors bring to teacher professional learning programs and that inform their conceptualizations, interpretations, judgments and decisions in assessment. I discuss the need in developing language assessment literacy for processes that develop teacher-assessors’ capability to explore and evaluate their own preconceptions so as to become aware of how they interpret their own assessment practices and their students’ second language learning. Through these processes they develop a deeper understanding of the interpretive nature of assessment and their own self-awareness as assessors.

Keywords

Alternative assessment language assessment literacy teacher knowledge teacher self-awareness

Introduction: Setting the scene

In the past decade in general education, there has been an increasing interest in developing the professional knowledge and understanding of teachers. This is because of the strong line of evidence that the teacher is the most important factor influencing student learning (Darling-Hammond, 2000; Rowe & Hill, 1998; Wright, Horn & Sanders, 1997). In the field of second language education, which arguably relies in a distinctive way on the assessment of student achievements, teacher knowledge, understanding and practices of assessment – that is, language assessment literacy on the part of teachers – are crucial.

At the same time, it is worth recognizing that assessment remains the aspect of the curriculum and teaching and learning practices that is the least amenable to change. This is undoubtedly because of traditional pressure in assessment towards objectivity, conformity, consistency and certainty. It is also that part of the educational process where teachers perform the dual role of both teacher and assessor or judge and where their personal accountability is at a premium. That pressure is compounded by major conceptual shifts that are taking place in the field of educational assessment and, by extension, assessment in language education, as it seeks to incorporate contemporary views of learning, notably sociocultural theories. The conceptual shifts have complexified the professional repertoire of teachers. Assessment operates between two contrasting paradigms, which McNamara (2003) has described as a “paradigm war”. Teachers need to work within these contrasting paradigms – often as a site of tension or struggle – as they fulfil the different purposes of assessment for which they are responsible. This tension provides an intellectual framework for understanding language assessment (see Shepard, 2000a) and by extension, language assessment literacy. These considerations, taken together, set the scene for the need for and expectations related to language assessment literacy on the part of teachers.

My discussion of language teacher assessment literacy in this paper relates to the professional learning of teachers of languages (called “foreign” or “modern” languages in some contexts) in years K–12 in Australia. First, I situate the development of language assessment literacy in the context of two considerations that render the process particularly complex: the institutional character of assessment that creates a culture of certainty and compliance; and contrasting theoretical paradigms in assessment that create, for teachers, uncertainty as to what is “permissible” or not in practice. Then I discuss the conceptualization of an expanded knowledge base that includes concepts from different and dynamic paradigms. I argue that, while defining the knowledge base for language teacher assessment literacy is necessary, it is not a sufficient basis for actually developing knowledge and understanding of assessment on the part of language teachers. Not only do teachers need to understand the conceptual bases of different approaches, they also need to relate such knowledge to their professional practice in their particular context. In teacher professional learning there are two interrelated goals: on the one hand, to transform teacher practices in assessment in order to benefit student learning, and on the other hand, to develop teachers’ own understanding or meta-awareness of the nature of assessment itself as phenomenon and their role and practices as teacher-assessors. Working simultaneously towards these two interrelated goals renders the process of developing assessment literacy particularly complex. It entails working with the often tacit preconceptions, beliefs, understandings and world-views about assessment that teacher-assessors bring to their professional learning and their practices of assessment. I illustrate the complexity involved with examples drawn from two projects carried out at the Research Centre for Languages and Cultures at the University of South Australia on developing and investigating teacher understanding and practices in second language education, including assessment. I conclude by arguing for the need for teachers to develop language assessment literacy in ways that enable them to explore and evaluate their own preconceptions, to understand the interpretive nature of the phenomenon of assessment and to become increasingly aware of their own dynamic framework of knowledge, understanding, practices and values, which shape their conceptualizations, interpretations, judgments and decisions in assessment and their students’ second language learning. Through these processes, they will gradually develop self-awareness as assessors, an integral part of their language assessment literacy.

Contextual considerations

In seeking to develop language assessment literacy, two contextual considerations are worth highlighting: the institutional character of assessment and contrasting paradigms that provide the theoretical context in assessment.

The institutional character of assessment

Assessment is always situated in distinctive institutional and policy contexts that confer on the assessment process particular characteristics and requirements. For example, in the context of learning languages in K–12 in Australia, language learning and assessment are defined by state curriculum and assessment frameworks. Each state of Australia has its own curriculum and assessment framework. These vary in nature and in the degree of specification and prescription. In most states, these frameworks for the languages learning area are generic, not language-specific (e.g., South Australia: Department of Education and Children’s Services, n.d.; Victorian Curriculum and Assessment Authority, n.d.). Nevertheless, they define the categories through which valued learning and outcomes or achievement standards are described and with which teachers need to work. As is typical of such frameworks, they also describe in a highly generalized way the nature and scope of learning and the expected levels of achievement. In this respect, they are not unlike frameworks such as the Common European Framework of Reference (Council of Europe, 2001). The definitional categories and generalized descriptions of achievement standards presented in these frameworks provide the frame of reference for conducting assessment.

Complexity arises when teachers of diverse languages use these common frameworks because, first, they have to “fit” a generic framework to the specific language that they teach, and second, they have to “fit” the state framework to their local context. A teacher of Chinese, for example, needs to adapt the generic framework that is intended to be used for all languages to the teaching of Chinese, making “adjustments” to take into account the fact that Chinese is an ideographic language (just to name one aspect of the specificity of Chinese). In addition, the teacher of Chinese then needs to adapt the framework to the specific context of the Chinese program in his or her particular school, which is likely to have to cater for both L2 learners of Chinese and learners with a home background in Chinese language whose achievements are likely to be different. In so doing, if they “adapt” rather than “adopt” the framework they may be seen to compromise the systemic assessment “requirements”; if they do not, they may compromise learners and learning (Brindley, 2001b). These institutional and policy requirements create a culture of certainty and compliance that is not easily challenged by teachers. They also contribute to shaping the preconceptions about assessment purposes, constructs, methods and judgments that teachers bring to the process of developing assessment literacy.

Contrasting paradigms

As indicated above, the field of educational assessment operates between two contrasting paradigms. This “operating between” is often a site of tension for teachers (see Gipps, 1999; James & Pedder, 2006; Moss, Girard & Greeno, 2008; Shepard, 2000b). The tension is between traditional assessment, which tends to be aligned with cognitive views of learning and psychometric testing, and alternative assessment, which tends to be aligned with sociocultural views of learning. Within sociocultural views, learning and its assessment at any point in time and over time is contextualized in the social interaction between learners and their environments (including conceptual tools, physical tools and people). Within this view of learning, assessment encompasses a range of practices that include, for example, performance assessment, classroom-based assessment and formative assessment (see, e.g., Birenbaum, 1996; Black & Wiliam, 1998; Fox, 2008; Gipps & Stobart, 2003; James, 2006; Moss, 2008). Assessment authorities, which are responsible for formal assessment, generally operate within the traditional paradigm, with a focus on testing the content of learning through objective procedures, normally as single events, and student performances are generally norm-referenced. In this psychometrically oriented perspective, the major interest is in the measurement of learning.

Summative assessments conducted by schools and their teachers, though less formal than the assessments conducted by assessment authorities, are also focused on end-of-unit or end-of-year assessment of learning, often reflecting the kinds of assessments performed by assessment authorities. Although assessment within an alternative paradigm is variously understood (Fox, 2008), it expands the traditional perspective. Its focus is on teachers finding ways of allowing students to demonstrate what it is that they know and have learnt. It includes assessment in the context of learners’ developing capabilities, described as “dynamic” assessment (Poehner, 2010). It allows for both formative (assessment for learning) and summative assessment, with the purpose of “forming” or shaping learning. It allows for diverse kinds of evidence, and diverse evidence for diverse learners. It tends to value both the product and process of learning (Birenbaum, 1996; Gipps, 1999; Teasdale & Leung, 2000). It also recognizes that learning is developmental and that assessment includes a collection of evidence of performances over time to provide evidence of growth and learning. Within this paradigm assessment is closely related to teachers’ day-to-day work of teaching and learning and assessment for learning. To address the range of assessment purposes, teachers of languages need to understand the assumptions of both paradigms and move between them. This creates a sense of uncertainty for them, as will be evident in the data I discuss below.

The difference between these paradigms is not just a matter of considering different purposes and methods of assessment, or different ways of referencing judgments of performance. As Moss, Girard and Hanniford (2006) explain, the difference is philosophical. At a fundamental level, the contrast is between naturalist and interpretive views of social science. From the former perspective, knowledge is seen as objective, that is, uninfluenced by the particular knower; social phenomena are studied in the same way that the natural sciences study natural phenomena, and the goal is to arrive at generalizable explanations. In the latter, interpretive view, social phenomena are studied with a view to understanding “what people mean and intend by what they say and do and to locate those understandings within the historical, cultural, institutional and immediate situational contexts that shape them” (Moss, Girard & Hanniford, 2006, p. 110); in other words, it becomes important to understand knowledge as held and influenced by the knower, recognizing that this is shaped by his or her personal history and context of experience. Seen from this perspective, learning and assessment are situated in the experiences of those who participate in them (see also Haertel, Moss, Pullin & Gee, 2008).

This philosophical distinction that underlies the contrasting paradigms in assessment and their alignment with distinctive learning theories is relevant in at least two ways to the development of language assessment literacy by teachers. First, understanding these distinctions and alignments is part of the knowledge base that teacher-assessors need to develop (what teachers need to know). Here, the increasing influence of sociocultural theories of learning on assessment practices has opened up discussion of alternative assessment paradigms and practices in the field, thereby expanding the knowledge base that teachers need to develop. A good understanding of the contrasting paradigms provides teachers with a basis for understanding the tensions they often experience in practice. Second, this distinction and alignment with learning theories is relevant to ways of working in language teacher education to develop teachers’ assessment literacy (how they learn and develop it). Here, sociocultural theory provides a basis for incorporating ways of working to develop assessment literacy that recognize both “objective”, theoretical knowledge of the field and interpretive views.

In summary, the two contextual considerations discussed above create, on the one hand, a tendency towards compliance and, on the other hand, a tension for teachers as they work within contrasting paradigms to meet the requirements of their roles as teachers and assessors. It is this force towards compliance and these tensions that render particularly complex the process of developing teachers’ language assessment literacy. More than just having a knowledge base, they need to understand and question the assumptions made and the possibilities and limitations of the different paradigms. This context has implications both for the knowledge base and processes for developing language assessment literacy in teacher education.

Conceptualizing the knowledge base and processes for developing language teacher assessment literacy

Developing language teacher assessment literacy requires a consideration of two aspects. First, it requires the identification of relevant domains that comprise the knowledge base and the relationship among these domains. The identification of disciplinary domains of knowledge in this context also gives rise to important philosophical questions about the nature of knowledge and the ways in which it is developed and used in practice. Specifically, in working with teachers, it needs to accommodate views about the distinctive nature of teacher knowledge (see Ball, Thames & Phelps, 2008; Shulman, 1987) and a conception of knowing as dynamic. Second, developing teacher assessment literacy requires a consideration of processes for its development that invite teachers to examine, in a critical way, their own conceptions of the assessment process itself and the conceptions of others.

Domains that comprise the knowledge base

The knowledge base comprises a number of intersecting domains. These include knowledge of language assessment, which encompasses not only diverse assessment paradigms, theories, purposes, and practices related to elicitation, judgment and validation in diverse contexts, but also learning theories and practices and evolving theories of language and culture. Furthermore, in school language education, assessment cannot be separated from its relationship with the curriculum, and processes of teaching and learning.

In a recent paper, Inbar-Lourie (2008) considers the requirements of a language assessment literacy knowledge base and builds on Brindley’s (2001a) earlier proposal for five modules for professional development programs in assessment for teachers. These included: the social context of assessment (as a core unit), defining and describing proficiency (as a core unit), constructing and evaluating language tests, assessment in the languages curriculum and putting assessment into practice. Inbar-Lourie concurs with him that one module needs to provide background to the social, educational and political aspects of assessment. She sees this as including “the social turn” in language assessment (McNamara & Roever, 2006) as well as critical views on the role of language tests in society (Shohamy, 2001). She notes, appropriately, that Brindley’s proposal does not specifically address the concepts of language (to which I would also add culture) and language assessment. She also concurs with Brindley’s inclusion of a module that considers the theoretical bases for assessment, specifically the concepts of validity and reliability, and a discussion of the prevailing models of language knowledge. She adds the importance of considering the norms of English. The notion of “norms” in this context refers to standards of “acceptable” use. In school language learning, which is necessarily developmental across the K–12 span of schooling, it is also important for teachers to have a sense of expected norms as levels of achievement at different phases along the continuum of learning. It is assumed that the now readily available frameworks of content and achievement standards, developed by educational authorities, provide a resource that addresses this aspect of language assessment practice and, by extension, language assessment literacy. In the languages learning area, these are normally available as generic descriptions, which present challenges when they are to be used by teachers making assessments in specific languages.

Inbar-Lourie (2008) highlights the importance of particular conceptual relationships in language learning that need to be considered in language assessment literacy. These include the relationship between language and pedagogies that pertain to the teaching of particular aspects of language learning, and the integration of language and content. To these, I would add the relationship between language and second language learning (see Seedhouse, Walsh & Jenks, 2010), as well as the relationship between language and culture (see Byram, 1997; Kramsch, 2006; Scarino, 2010).

With respect to the “how” dimension of language assessment literacy, Inbar-Lourie (2008, p. 392) questions Brindley’s (2001a) separation of “test-development and analysis” and “assessment in the language curriculum”. She notes, appropriately, that teachers need to understand the assumptions and traditions of both. Inbar-Lourie (2008) also considers the exploration of assessment initiatives and research to be a necessary component of the knowledge for all teachers, and not just for particular professionals as Brindley (2001a) had proposed.

From domains to processes for developing language teachers’ language assessment literacy

Although it is fruitful to define the knowledge base for language assessment literacy, it is not a sufficient basis for actually developing the knowledge and understanding of teachers. This is particularly so in the context of achieving the dual aims of teacher learning – to transform teacher practices to benefit student learning and to improve teachers’ understanding of their own practices. In the lead article of a special issue of TESOL Quarterly (volume 32, number 3, 1998) on research and practice in English language teacher education, Freeman and Johnson argue for a reconceptualization of the knowledge base of TESOL teacher education. Drawing on work in general education, the shift in teacher learning that they propose is from gaining decontextualized bodies of knowledge to bodies of knowledge that are contextualized in the complex realities of human interaction and teachers’ contexts of practice. They argue that the knowledge base “must begin with the activity of language teaching and learning; the school and classroom contexts in which it is practiced; and the experience, knowledge, and beliefs of the teacher as participant” (p. 413).

In proposing this shift Freeman and Johnson acknowledge that teacher learning requires cognitive and affective involvement and teachers’ willingness to examine their own knowledge and beliefs and their enactment in their local practices for the purpose of improvement or change. They also acknowledge the situated nature of teacher learning. Their position vis-à-vis discipline-based knowledge as secondary to the activity of language teaching and learning itself, has been contested (see Yates & Muchisky, 2003; Tarone & Allwright, 2005; and Freeman and Johnson’s response to Tarone and Allwright in Freeman & Johnson, 2005). The argument centres on the nature and relative contribution of particular kinds of knowledge, as well as on the deeper philosophical issue discussed above of the situated teacher-assessor and how knowledge itself is understood. Freeman and Johnson’s (1998) position highlights that, regardless of the relative emphasis that is accorded to domains of knowledge as opposed to their enactment in practice, at the very least, it is necessary to include both. The key consideration here is to understand how teachers integrate the diverse domains of knowledge into their practices and understanding.

The increasing attention to sociocultural theories of learning extends this debate by highlighting that language learning and, indeed, teacher learning are focused on how meaning is situated in concrete human activity (Freeman, 2007; Lantolf & Johnson, 2007). Freeman (2007) maintains that the object and process of teacher learning have been redefined through the recognition in teacher education of the need to take into account the “inner” worlds and lives of teachers, their “beliefs, assumptions and knowledge” (see Woods, 1996), and their “personal practical knowledge” (see Golombek, 1998). For Freeman (2007) this recognition means that teacher knowledge is best seen “as an intersubjective or contingent social phenomenon … that blended elements learned through socialization, like any form of learning, with those that are explicitly taught in professional education” (p. 901).

Working in teacher education, Shulman (1987) developed the concept of “pedagogical content knowledge”, which he distinguished from discipline or subject matter knowledge. He recognized that teachers need to work with a distinctive form of knowledge that renders discipline knowledge teachable and learnable. In other words, discipline knowledge per se cannot simply be transmitted on the assumption that it will be incorporated by the learner. The knowledge itself needs to be modified for the purposes of teaching. Ball, Thames & Phelps (2008) have further developed the concept of “pedagogical content knowledge” to include two sets of interrelationships: knowledge of content and teaching and knowledge of content and students. Thus “pedagogical content knowledge” becomes knowledge that combines discipline knowledge rendered appropriate for teaching, as well as being rendered appropriate to the knowledge of students. Applied to the development of language teacher assessment literacy this concept invites an acknowledgement that the domains of knowledge need to be blended with the assessment life-worlds of teachers. It is necessary to recognize that the knowledge is contextualized in the realities of teachers’ contexts of practice – as pedagogical or practical and experiential knowledge. This knowledge is a part of teachers’ personal frameworks of knowledge, beliefs and values, that is, the conceptual and social resources they bring to both their practice and their ongoing learning.

In relation to developing language assessment literacy on the part of teachers, therefore, it is necessary to consider not only the knowledge base in its most contemporary representation, but also the processes through which this literacy is developed. In line with contemporary, sociocultural learning theories, these processes should recognize the “inner” world of teachers and their personal frameworks of knowledge and understanding and the way these shape their conceptualizations, interpretations, decisions and judgments in assessment. As argued elsewhere (Scarino, 2005), the understanding of the interplay of the intellectual, the social and ethical positioning that characterizes teachers’ “knowing” and the notion of “ethical knowing” is a kind of knowledge that extends beyond the knowledge base and capabilities of teachers to include their values and dispositions.

In recent times another dimension has been added to teacher professional learning – consideration of its impact, specifically in terms of student learning. Timperley and Alton-Lee (2008), for example, reviewed 97 empirical studies that identified the kinds of teacher knowledge that have demonstrated a positive impact on outcomes for diverse learners (p. 328). In describing the theoretical underpinnings of their synthesis, they acknowledge that much has been written about the fact that there is no direct relationship between a particular act of teaching and what students learn. They recognize that this is the same for teacher learning:

How teachers interpret and use the available understandings and skills is also a complex process. Teachers, like their students, have prior learning experiences and bring different conceptual and social resources that are influenced by their cultural heritages to the learning experience. To make a difference to their students’ learning, however … the content of what teachers learned needed to result in some changes to their practice, because it is teaching practice that influences the learning opportunities for students. (2008, p. 340)

Timperley and Alton-Lee (2008, p. 342) develop a framework for the analysis of the effectiveness of professional learning experiences that includes the relationships among:

the wider sociocultural environment

the professional learning environment

the content of professional learning opportunities

activities constructed to promote learning

(interactive) learning processes

responses of diverse teacher learners/communities

the impact on diverse student learners.

These domains bring together aspects of the knowledge base and processes of teacher learning discussed above, as well as the consideration of teacher transformation and impact on student learning. Each category indicated above is further elaborated. In this way, Timperley and Alton-Lee identify a complex network of relationships among diverse domains of knowledge, ways or processes of knowing and consequences of knowing. This network extends beyond the considerations of language assessment literacy that have been taken into account to date. At the same time, their framework also points to the complexity of developing teachers’ capabilities, for transforming of student learning. This complexity needs to be understood and embraced.

If the goal of assessment literacy is to effect change in teacher practice and understanding, it is necessary to reconsider both the knowledge base and ways of working to develop assessment literacy that take into account teachers’ personal frameworks of knowledge, beliefs and values. Sociocultural theories of learning provide a valuable, intellectual framework that allows for developing language teacher assessment literacy.

In the section that follows, I provide extracts from two collaborative projects where I have been involved in developing and investigating the language teaching, learning and assessment practices of teachers of languages. Each of these illustrates aspects of working with teachers’ preconceptions, beliefs and understandings in developing language assessment literacy.

Examples of some aspects of developing language assessment literacy

The examples I present here are instantiations of the intricacies involved in developing teacher assessment literacy at the interaction point between the knowledge base of the field and teachers’ knowledge, practical experience and values. They are taken from projects carried out at the Research Centre for Languages and Cultures at the University of South Australia. They foreground teacher experience of the process and, in particular, teachers’ preconceptions and tacit understandings that need to be uncovered, critically explored and, at times, challenged in developing assessment literacy. Because of limitations of space I cannot fully describe each project, nor fully represent the data gathered and the analyses undertaken.¹ Rather, I have chosen to illustrate particular moments of the experience where the teachers’ personal knowledge and their preconceptions about assessment are evident. They emerge from the dynamic, inquiry-driven process of working with teachers over iterative cycles of development, reflection and debriefing. The first example illustrates (1) the challenge that an experienced teacher faces in expanding the construct of communicative competence to include consideration of the language and culture nexus; and (2) the way in which her preconceptions, shaped by the institutional requirements and a traditional orientation to assessment, influence her understanding of what is possible both in eliciting and judging students’ learning. The second example illustrates the role of feedback and its internalization as an integral part of the process of developing assessment literacy.

Example 1: Assessing intercultural understanding in language learning

The three-year study, from which the extracts below are taken, investigated the ways in which teachers of languages assess intercultural understanding in learning languages. Assessing this dimension was recognized as an aspect of teaching and learning which was “difficult to assess” and not a part of the regular assessment practices of teachers of languages. The study involved two cycles of development, teaching, assessing and debriefing with a group of highly experienced teachers of a range of languages, who were invited to experiment with assessing intercultural understanding in the context of their language learning programs. I acknowledge here that the research orientation offered a particularly valuable dimension that is not typical of most professional learning programs. The design of the study permitted an iterative process of discussion among the researchers and teachers, which I see as important in developing teacher learning. While the experimentation/investigation was conducted by teachers working individually in their own schools, the ongoing discussion involved extended debriefings with the group of participants as a whole. All discussions were recorded and fully transcribed for subsequent thematic analysis. The transcriptions revealed teachers’ struggle with the construct and deep questioning of their own practices and judgments. The struggle is simultaneously theoretical, practical and institutional. In the extracts below, Marnie, a teacher of Chinese, reflects upon her work with her Year 11 class of students (all girls) on a unit of work entitled Women in China. The class, individually and together, had explored diverse texts, prepared oral presentations, researched and profiled significant women, analysed blogs and prepared their own blogs – all on the theme. These activities yielded various student responses which were then available to the teacher for assessment. In the final debriefing session, Marnie reflects specifically on the challenge that assessment presented for her.

Extract 1 – Marnie’s reflection 1

This focus (on intercultural sensitivity) is a way of making language learning as personally relevant to students and for them to engage … it’s a way for them to understand themselves better, but also to understand and challenge the way they view the world … I can experience it with them, but I don’t know that I can assess it because I’m so much a part of it as well. I don’t know then, if I can step back and really assess it because I’m too involved with what we’ve gone through together because it is something that you can’t just watch them, you have to be a part of it.

Extract 2 – Marnie’s reflection 2

The struggle I really had with … how do I take this really broad view that allowed the students flexibility to move at their own pace, but then bring them into quite specific criteria that I was then assessing …

One of the struggles that I think I really had … was when it came to a structured task, which I worried a little that I had guided them in some way to the responding … And one of the things I found was for the less capable language students, the struggle I had with the reflection in English. They were able to say some of the most amazing things. But when it came to the Chinese, they could make a statement, but they couldn’t really expand or explore that further … Well, do I then look at their English component and see that they have developed that idea, while I can see what they might be wanting to say in Chinese, because they haven’t got that in there because of their language. Do I still credit them for that? But I don’t want to penalise them twice for language use but also then penalise them because they haven’t said enough even though that’s purely because of their language.

In Extract 1, Marnie considers the tension that comes from her understanding of assessment as needing to be objective. She recognizes that her students’ learning in this unit had been different, experiential and reflective, but she has difficulty reconciling her roles as teacher, facilitator, fellow experiencing-participant and, at the same time, assessor. Her framing of assessment within a traditional paradigm does not permit consideration of the subjectivity of her experience as assessor. As such, her framing creates uncertainty for her and sets limits to what she sees as possible in assessment. She recognizes that she needs to stand back from the process of teaching and learning in order to assume her role as assessor. The debriefing invited her to equally stand back from her own preconceptions in order to make space for alternative conceptions. In Extract 2, she reflects on standardization in assessment. She notes the tension between being open and flexible in teaching and learning, but not having the same flexibility at the point of assessment with a class she is preparing for a public examination in the following year. A further struggle that she experiences in this instance is the difference in level of sophistication of content in some students’ responses to reflection tasks in English and the content expressed in Chinese. The institutional syllabus requires responses to tasks in Chinese but also a response to one task in English. It provides criteria for assessing each separately. Marnie recognizes that there is a disparity in the level of sophistication in the responses in the two languages, particularly for her weaker students. She is required to treat the performances as separate but in practice they are not for they belong to one and the same student. It raises the theoretical issue of the relationship between language and content in assessment. These struggles reflect her uncertainty about how to interpret and act upon the particular phenomena that arise in the context of particular instances of assessment. As an experienced teacher she raises questions about her own practices, which is an important part of the process of becoming assessment literate. At the same time, her questioning reveals preconceptions about assessment that come from the institutional and policy requirements and an essentially traditional orientation to assessment. These requirements and the long-established, traditional orientation are not readily put aside. The purpose of the debriefing was to invite Marnie to reconsider her own preconceptions and the way they shape her interpretation, decisions and judgments. The debriefing sessions enabled discussion of these complex dimensions of particular instances of assessment, the provision of theoretical input, and dialogue to compare ways of working with tensions that are a part of the practice of assessment. Through dialogue it becomes possible to explore both the tensions and the teachers’ preconceptions and how they enter into the teacher’s interpretations, decisions and judgments. At times, what is needed is a process of “unlearning” (Cochran-Smith, 2000); at other times, what is needed is a re-contextualizing of the experience both of which also entail the incorporation of theoretical input. Dialogue of this kind, focused on contextualized instances of practice, provides the conditions for new learning and becomes an integral part of developing teacher assessment literacy.

Example 2: A case study of intercultural language learning and assessment

This example is taken from an extended case study with an individual teacher involved in a larger project designed to develop teachers’ understandings of intercultural language learning and assessment. The teachers were asked to develop a year-long program of work that incorporated intercultural language learning goals, pedagogies and assessment processes. The case study was sustained over a period of 18 months. It involved Melissa, a teacher of Indonesian, and myself and another colleague as both facilitators of teacher learning and researchers. The extracts below illustrate the process of providing written feedback to the teacher on her year-long Year 12 Indonesian program and assessment plan and her way of subsequently incorporating the feedback into her own thinking and into the criteria for assessing her students’ performance.

This feedback highlights the need to consider diversity and variability in intercultural teaching and learning. The comments, followed by suggestions, focus on how to sustain the theme of diversity and variability of ideas, opinions, experiences, meanings, understandings and so on in Melissa’s year-long program of language and culture teaching, learning and assessment.

Extract 1: Written feedback to Melissa, December 2006

Comment 1: The issue of ‘variability’ (intra- and inter-culturally speaking) seems not to be included in this Program. However, the ‘theme’ in Module 3 is ‘The Indonesian-Speaking Communities’, making it explicit that there exist variable ‘communities’.

Suggestion 1: It would be interculturally-pedagogically relevant to introduce ‘variability’ from the very first module and sustain it throughout the program. Thus, it can be assessed as an intercultural, ‘learning goal’, and ‘outcome’: For example: In what demonstrable ways do our learners understand (or are aware of, sensitive, critical, engaged with) ‘variability’ in all the forms and behavioural manifestations it takes in human communication/interaction, intra-culturally and inter-culturally, within and across the cultures/languages?

Suggestion 2: In regard to assessing students’ intercultural understanding and learning throughout year-long teaching programs, the following criteria are suggested:

Student exemplifies ability to:

interconnect knowledge from various relevant/previous sources;

perceive these interconnections and compare them analytically, i.e. ‘knowingly’ and overtly in interaction with all participants in the classroom;

understand that in doing so, s/he is engaging in inter-active, self-reflecting ways of learning that are public, therefore ‘debatable’ and ‘challenge-able’;

understand that constructing ones’ own responses from variable resources provides for a critical, multi-and-variable perspective on ‘social issues’ across languages/cultures;

respect reciprocal multi-and-variable perspectives on the social, cultural and linguistic construction of human action and knowledge in all spheres of human communication and interaction;

self-signify this process of learning and educational interaction as intra- and intercultural, thus applicable to all professional and social spheres of communication and interaction with respected-and-respecting ‘others’.

In Extract 2 below, Melissa’s written reflection nine months later distills and incorporates the feedback given, providing us with her insights into how she had reconsidered her own practice.

Extract 2: Melissa’s written reflection, September 2007

The intercultural person reflects on experiences of diversity in positive ways seeking to understand what meanings are being created and what possible alternative meanings could be available. Learning to be intercultural involves learning how to reflect on experiences of linguistic and cultural diversity in communication in order to understand how they transpire. This involves recognizing that only by understanding and monitoring our own linguistic and cultural identities and practices can we engage with different ways of knowing, and reflect sensitively and critically on successes, failures, uncertainties and future developments in interacting with others. To be interculturally aware we need to come to see ourselves from both within our own linguistic and cultural identity and also to have insight into how what we see as familiar can become strange when viewed from a different perspective and that both views have value. We also need to reflect on how we respond to differences and how we engage with the linguistic/cultural practices of others and the consequences our ways of engagement have for us socially and interpersonally.

In particular, Melissa included in her instructions to students in assessing the unit of work, Suggestion 2 of the feedback, about expectations:

I am interested in your ability to:

interconnect inter-related knowledge from various sources;

perceive these interconnections and compare them analytically, i.e., ‘knowingly’ and overtly in interaction;

understand the process of learning and educational interaction as intracultural and intercultural;

understand that you are engaging in interactive, self-reflecting ways of learning.

I am hopeful that through learning Indonesian you will be able to respect reciprocal multi-and-variable perspectives on the social, cultural and linguistic construction of human action and knowledge in all spheres of human communication and interaction.

Melissa incorporates phrases from the feedback directly into her own reflection and in the expectations she sets out for her students. The written reflection comes from a reconsideration of the phenomenon of language teaching, learning and assessment and of her own role. Her decision to incorporate the feedback in her statement of requirements and criteria also comes from this reconsideration.

Because the discussion and feedback process continued over a period of several months and was documented in several different ways, it was possible to trace the new learning and its implications for assessment in the way the teacher articulates her own learning and incorporates it into the task requirements and criteria that she provides her students. As one of the researchers involved in this case study, I did not expect the direct incorporation of feedback in this manner. This is especially so as she had initially found the feedback confronting and indicated that she could not change her program and assessment in any way because it had been developed in line with system requirements. This process of incorporation on its own, however, does not necessarily reflect learning on the part of the teacher. In this instance, further evidence was captured in interviews both with the teacher and with her students, as well as in classroom observations. The teacher had made the feedback her own. This experience raises questions about both the intricate process of giving and receiving feedback and the expectations of how the feedback will be acted upon. These questions prompt consideration of the nature, extent and role of feedback in developing language teacher assessment literacy, for it is through feedback that teachers begin to examine their own preconceptions and the way these shape their interpretations, decisions and judgments.

Discussion and reflection

The examples discussed above provide fragments of some of the experiences and reflections of teachers and of researcher dialogue with teachers in the context of developing the assessment literacy of teachers of languages. They illustrate the role of teacher preconceptions, beliefs, personal theories and practices, that is, their interpretive framework, which cannot be bypassed in developing teacher assessment literacy. Assessment literacy needs to be considered in relation to the theoretical knowledge base as an essential source of input, as well as to teachers’ interpretive frameworks which are shaped through their particular situated personal experiences, knowledge, understanding and beliefs. They are obliged to integrate simultaneously the complex theoretical, practical and institutional dimensions of the assessment act and an understanding of self in relation to these. I have also noted the challenge of demonstrating transformation or change on the part of the teacher and, in line with Timperley and Alton-Lee (2008), I have noted the need, but cannot claim evidence, for the impact of teacher learning on student learning.

Moss (1996, 2004, 2008) has argued over an extended period of time for the need to expand understandings and practices in assessment. Drawing on sociocultural theories of learning, on hermeneutics as the theory of interpretation and, in particular, on the work of Gadamer (2004), Moss argues for a view of assessment that includes an understanding of assessment as interpretive (Moss, 1996, 2004). This has implications for considering the act of assessment itself, in a critical way, as a necessary part of developing assessment literacy – a part that examines the nature of the act as involving interpretation at every point: in conceptualizing the construct, in considering tasks that are intended to elicit students’ performances and the way these tasks are interpreted by students, in interpreting and applying criteria and standards for judging performance and in interpreting evidence as part of processes of validation. It also implies a consideration of diverse traditions which Moss sees as a natural resource for learning. The goal in examining diverse traditions, for Moss, is not to reconcile differences but to understand deeply the assumptions on which different perspectives are based (Moss et al., 2009). Moss proposes a way of working (Moss, 1996; Moss & Schutz, 2001) that is intended to develop deep understanding. This notion of understanding, however, is not passive but rather, it is active in the Gadamerian sense of “dialogue”. Gadamer (2004) conceives understanding as a dialogue in which the partners are seeking to understand the subject matter at hand and each other. This dialogue involves examining the preconceptions that each partner brings, which are shaped by their histories of experience, traditions and context. These preconceptions, assumptions or fore-understandings are always at work in the interpretations that people make – whether they acknowledge them or not. These need to be uncovered and examined critically in dialogue with others in order to come to understand the phenomenon at hand, as well as, and importantly, the people involved and their perspectives. Gadamer (2004) describes understanding growing out of dialogue as a “fusion of horizons” – the horizons reflected in the partner’s preconceptions. He explains the process of “coming to understand” as follows:

Reaching an understanding … necessarily means that a common language must first be worked out. This is not an external matter of simply adjusting our tools; nor is it even right to say that the partners adapt themselves to one another, but rather … to reach an understanding in dialogue is not merely a matter of putting oneself forward and successfully asserting one’s point of view, but being transformed into a communion in which we do not remain the same. (Gadamer, 2004, p. 371)

To understand requires a mutual process of making sense of each other’s contributions (the subject-matter) and, at the same time, each other (the person). Participants in the dialogue necessarily examine the preconceptions of the other, as well as their own, and are transformed in the process.

This view of ‘understanding’ as a complex dynamic and dialogic act of engagement and reflection has implications for developing language assessment literacy. It also raises the question of when it is that a teacher can be considered to be “language-assessment literate”. I would argue that teachers need to come to understand their own frameworks of knowledge, understanding and values – their own preconceptions – and how they shape their conceptualizations, interpretations, judgments and decisions. They also need to come to understand how these frameworks have the power to limit or expand their horizons of understanding and action. It is through such processes of interrogating one’s own and others’ preconceptions that the way is opened up to new learning and the transformation that develops self-awareness as teacher-assessor. Each experience of assessment and reflection on the assessment experience leads to an adjustment of one’s own framework of knowledge and understanding. I see this as the goal and measure of success in developing assessment literacy in teacher education.

Conclusion

Developing the language assessment literacy of teachers in in-service teacher education necessitates a consideration and integration of not only the knowledge base required for language assessment, but also interrelated understandings of language, culture and learning. In addition, we need to take into account the interplay of both variable theoretical knowledge and teachers’ own personal (fore-)understandings of this knowledge and especially how this knowledge is re-cast for the purposes of teaching, learning and assessment. Further, we need to take into account the processes through which teacher-assessors can critically examine their own assessment practices. Finally, we need to recognize the process of assessment – and indeed the development of teacher literacy itself – as interpretive, entailing the examination and re-examination of one’s own and others’ preconceptions from diverse perspectives in order to understand the act of assessment and oneself as an assessor. It is this integrated knowledge and interpretive understanding or meta-awareness, understood as complex and dynamic, that sustains the development of teacher knowledge in general and, specifically, the development of language assessment literacy.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

References

Ball

D. L.

Thames

M. H.

Phelps

(2008). Content knowledge for teaching: What makes it special? Journal of Teacher Education, 59, 389–407.

Birenbaum

(1996). Assessment 2000. Towards a pluralistic approach to assessment. In Birenbaum

Dochy

F. J. R. C.

(Eds.), Alternatives in assessment of achievement, learning processes and prior knowledge (pp. 3–30). Boston, MA: Kluwer Academic.

Black

P. J.

Wiliam

(1998). Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice, 5(1), 7–74.

Brindley

(2001a). Language assessment and professional development. In Elder

Brown

Hill

Iwashita

Lumley

McNamara

O’Loughlin

(Eds.), Experimenting with uncertainty: Essays in honour of Alan Davies (pp. 126–136). Cambridge: Cambridge University Press.

Brindley

(2001b). Outcomes-based assessment in practice: Some examples and emerging insights. Language Testing, 18(4), 393–408.

Byram

(1997). Teaching and assessing intercultural communicative competence. Clevedon: Multilingual Matters.

Cochran-Smith

(2000). Blind vision: Unlearning racism in teacher education. Harvard Educational Review, 70(2), 157–190.

Council of Europe. (2001). Common European Framework of Reference for Languages. Cambridge: Cambridge Univesity Press & Council of Europe.

Darling-Hammond

(2000). Teacher quality and student achievement: A review of state policy evidence. Education Policy Analysis Archives, 8(1). Retrieved from http://epaa.asu.edu/epaa/vol8.html

10.

Fox

(2008). Alternative assessment. In Shohamy

Hornberger

N. H.

(Eds.), Encyclopaedia of language and education, Vol. 7: Language testing and assessment (2nd ed., pp. 97–109). New York: Springer and Business Media LLC.

11.

Freeman

(2007). Research ‘fitting’ practice: Firth and Wagner, classroom language teaching, and language teacher education. The Modern Language Journal, 91, 893–906.

12.

Freeman

Johnson

K. E.

(1998). Reconceptualising the knowledge base of language teacher education. TESOL Quarterly, 32(3), 397–417.

13.

Freeman

Johnson

K. E.

(2005). Response to Tarone and Allwright. In Tedick

(Ed.), Second language teacher education: International perspectives (pp. 25–32). Mahwah, NJ: Lawrence Erlbaum.

14.

Gadamer

H.-G.

(2004). Truth and Method (2nd ed., Weinsheimer

Marshall

D. G.

, Trans.). New York: Continuum.

15.

Gipps

C. V.

(1999). Sociocultural aspects of assessment. Review of Research in Education, 24, 355–392.

16.

Gipps

C. V.

Stobart

(2003). Alternative assessment. In Kellaghan

Stufflebeam

D. L.

(Eds.), International handbook of educational evaluation: Practice (pp. 549–575). Dordrecht: Kluwer.

17.

Golombek

(1998). A study of language teachers’ personal practical knowledge. TESOL Quarterly, 32, 447–464.

18.

Haertel

E. H.

Moss

P. A.

Pullin

D. C.

Gee

J. P.

(2008). Introduction. In Moss

P. A.

Pullin

D. C.

Gee

J. P.

Haertel

E. H.

Young

L. J.

(Eds.), Assessment equity and opportunity to learn (pp. 1–16). New York: Cambridge University Press.

19.

Inbar-Lourie

(2008). Constructing a language assessment knowledge base: A focus on language assessment courses. Language Testing, 25(3), 385–402.

20.

James

(2006). Assessment, teaching and theories of learning. In Gardner

(Ed.), Assessment and learning (pp. 47–60). London. SAGE Publications.

21.

James

Pedder

(2006). Professional learning a condition for assessment for learning. In Gardner

(Ed.), Assessment and learning (pp. 27–43). London: SAGE Publications.

22.

Kramsch

(2006). From communicative competence to symbolic competence. Modern Language Journal, 90(2), 249–252.

23.

Lantolf

J. P.

Johnson

K. E.

(2007). Extending Firth and Wagner’s (1997) ontological perspective to L2 classroom praxis and teacher education. The Modern Language Journal, 91(5), 877–892.

24.

McNamara

(2003). Tearing us apart again. The paradigm war and the search for validity. EUROSLA Yearbook, 3, 229–238.

25.

McNamara

Roever

(2006). Language testing: The social dimension. Malden, MA & Oxford: Blackwell.

26.

Moss

P. A.

(1996). Enlarging the dialogue in educational measurement: Voices from interpretive research traditions. Educational Researcher, 25(1), 20–28.

27.

Moss

P. A.

(2004). The meaning and consequences of reliability. Journal of Educational and Behavioural Statistics, 29, 241–245.

28.

Moss

P. A.

(2008). Sociocultural implications for assessment: Classroom assessment. In Moss

P. A.

Pullin

D. C.

Gee

J. P.

Haertel

E. H.

Young

L. J.

(Eds.), Assessment, equity and opportunity to learn (pp. 222–258). Cambridge: Cambridge University Press.

29.

Moss

P. A.

Girard

B. J.

Greeno

J. G.

(2008). Sociocultural implications for assessment II. Professional learning, evaluation and accountability. In Moss

P. A.

Pullin

D. C.

Gee

J. P.

Haertel

E. H.

Young

L. J.

(Eds.), Assessment, equity and opportunity to learn (pp. 295–332). Cambridge: Cambridge University Press.

30.

Moss

P. A.

Girard

B. J.

Haniford

L. C.

(2006). Validity in educational assessment. Review of Research in Education, 30(1), 109–162.

31.

Moss

P. A.

Phillips

D. C.

Erickson

F. D.

Floden

R. E.

Lather

P. A.

Schneider

B. L.

(2009). Learning from our differences: A dialogue across perspectives on quality education research. Educational Researcher, 38(7), 501–517.

32.

Moss

P. A.

Schutz

(2001). Educational standards, assessment and the search for consensus. American Educational Research Journal, 38(1), 37–70.

33.

Poehner

M. E.

(2010). Dynamic assessment. A Vygotskian approach to understanding and promoting L2 development. Lexington, KY: Springer.

34.

Rowe

K. J.

Hill

P.W.

(1998). Modelling educational effectiveness in classrooms: The use of multilevel structural equations to model students’ progress. Educational Research and Evaluation, 4(4), 307–347.

35.

Scarino

(2005). Introspection and retrospection as windows on teacher knowledge, values, ethical dispositions. In Tedick

D. J.

(Ed.), Second language teacher education: International perspectives. New York: Lawrence Erlbaum.

36.

Scarino

(2010). Assessing intercultural capability in learning languages: A renewed understanding of language, culture, learning and the nature of assessment. The Modern Language Journal, 94(2), 324–329.

37.

Seedhouse

Walsh

Jenks

(Eds.). (2010). Conceptualising ‘learning’ in applied linguistics. Basingstoke, UK: Palgrave Macmillan.

38.

Shepard

L. A.

(2000a). The role of assessment in a learning culture. Educational Researcher, 29(7), 4–14.

39.

Shepard

L. A.

(2000b). The role of classroom assessment in teaching and learning (CSE Technical Report 517). Center for the Studying of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles.

40.

Shohamy

(2001). The power of tests. Harlow, UK: Pearson Education.

41.

Shulman

(1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57, 1–22.

42.

South Australia. Department of Education and Children’s Services (n.d.). South Australian Curriculum Standards and Accountability Framework. Retrieved from www.sacsa.sa.edu.au

43.

Tarone

Allwright

(2005). Second language teacher learning and student second language learning: Shaping the knowledge base. In Tedick

(Ed.), Second language teacher education: International perspectives (pp. 5–23). Mahwah, NJ: Lawrence Erlbaum.

44.

Teasdale

Leung

(2000). Teacher assessment and psychometric theory: A case of paradigm crossing? Language Testing, 17(2), 163–184.

45.

Timperley

Alton-Lee

(2008). Reframing teacher professional learning: An alternative policy approach to strengthening valued outcomes for diverse learners. Review of Research in Education, 32, 328–369.

46.

Victorian Curriculum and Assessment Authority (n.d.) Victorian Essential Learning Standards. Retrieved from http://vels.vcaa.vic.edu.au/index.html

47.

Woods

(1996). Teacher cognition and language teaching. New York: Cambridge University Press.

48.

Wright

S. P.

Horn

Sanders

W. L.

(1997). Teacher and classroom contexts effects on student achievement: Implications for teacher evaluation. Journal of Personnel Evaluation in Education, 11, 57–67.

49.

Yates

Muchinsky

(2003). On reconceptualising teacher education. TESOL Quarterly, 37(1), 137–146.