Developing the assessment literacy of university proficiency test users

The interpretation and use of proficiency test scores for university admission

Standardized proficiency tests are widely used around the world for assessing the readiness of students for whom English is an additional language to study in English-medium universities. In many universities, especially at the undergraduate level, the scores from one of these tests may be the only evidence of English language ability used. Spolsky (2008, p. 300) suggests that ‘an oversimplified view of the ease of producing meaningful measurement is held by the general public and educational administrators alike.’ He argues that the language testing field must assume some responsibility for this state of affairs. He claims that language testers have mistakenly focused on reducing measurement error in tests rather than trying to understand the risk of making decisions about the fate of human beings using fallible language tests. They have also continued to produce unidimensional scales that fail to do justice to the complex multidimensional nature of language ability. Most importantly, Spolsky (2008, p. 302) argues:

We have also continued to ignore a critical factor in our use of proficiency measures to make admissions for immigration or other decisions about individuals, the fact that proficiency is dynamic and that a single measure at one time does not permit predicting what it will be later.

In relation to making university admissions decisions, Spolsky (2008, p. 303) advocates the use of an aptitude test as well as a proficiency test to provide ‘a better prediction of how well and how quickly a foreign student would fit into an English-language environment.’

While more than one form of evidence relating to English proficiency is considered in some UK and US university contexts (Banerjee, 2003; Van Nelson, Nelson & Malone, 2004), this is normally not the case in other countries. In Australia, for example, decisions about English proficiency are usually based on a single source of evidence, such as scores from a standardized language test. The problems associated with this practice are exacerbated by the increasing commodification of proficiency test scores in a highly competitive educational marketplace: cut scores decisions in Australian universities may be based not so much in relation to the language skills needed to study successfully but on boosting revenue by recruiting the greatest possible number of full-fee-paying international students (Victorian Ombudsman, 2011).

As Messick (1996) has suggested, the validity of a test hinges critically on the interpretation of test scores and the uses to which they are directed. University staff need to be educated to take greater responsibility for the valid and ethical uses of proficiency test scores. These responsibilities are set out clearly in the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999, p.112), the guidelines on test use of the International Test Commission (2000) and the International Language Testing Association’s (2007) Code of Practice. These standards and guidelines imply that university members of staff need to develop a strong understanding of the tests that they interpret and use for high-stakes admissions decisions. This responsibility includes setting educationally responsible minimum entry scores, understanding the limitations of scores from a single proficiency test, as Spolsky (2008) suggests, and, where feasible, using additional data about such factors as an individual applicant’s language aptitude, motivation and language learning background in making decisions about their readiness to commence study in an English-medium institution (Rees, 1999). A key question which arises here is what particular knowledge and skills university staff need to acquire about proficiency testing in order to fulfill their responsibilities as test users and how they might best do so.

Interpretation and use of the IELTS test

This article focuses on understanding the assessment literacy needs of university staff working with one high-stakes, standardized English proficiency test, the IELTS (International English Language Testing System), in Australian universities. In recent years there has been a growing interest in studying how this test is used in higher education institutions in a number of countries (Coleman, Starfield & Hagan, 2003; Coley, 1999; Deakin, 1997; McDowell & Merrylees, 1998; Rea-Dickins, Kiely & Yu, 2004; O’Loughlin, 2011).

Coleman, Starfield and Hagan (2003) examined the attitudes of both students and staff (administrative and academic) towards IELTS in three institutions in Australia, China and the UK. They found that, while all participants in the study were generally positively disposed towards the test, overall students were more knowledgeable than staff about the test and more convinced that the institution’s IELTS entry level was appropriate for the course of study they were undertaking. Staff generally felt that the IELTS scores should be higher and that many students’ English language ability was not adequate for their chosen course of study. Perhaps the most troubling finding was that the university staff (administrative and academic) in the three participating institutions demonstrated low understanding of the meaning of IELTS scores.

In her doctoral research project, Banerjee (2003) examined the use of proficiency test scores, including the IELTS, in the selection of students for postgraduate degree programs at a UK university. She found that the selection of international students at the University of Lancaster was a complex, holistic decision-making process based on the recommendation of an academic staff member taking into account a wide range of criteria. Yet, like Coleman, Starfield and Hagan (2003), Banerjee (2003) found that academic admissions officers were not very knowledgeable about the meaning of proficiency test scores, including IELTS.

In a study conducted at another UK university, the University of Bristol, Rea-Dickins, Kiely & Yu (2007) also found that university admissions staff were not always sufficiently knowledgeable about the meaning of IELTS scores. They argued for stronger training of admissions tutors in order that they become better informed about the meanings of IELTS score profiles. This included awareness of and access to the IELTS website.

O’Loughlin (2011) examined the use of IELTS within a large faculty of a major Australian university. The author reported first, that there was no rational basis for setting IELTS minimum entry scores in this context. There was also no tracking of student success in order to validate these entry requirements. Second, applicants’ entry scores were not considered in relation to other relevant individual factors as recommended in these guidelines. Third, IELTS scores were not used to guide future English language learning. This suggested that there had been no beneficial educational consequences flowing from use of the test. The author concluded therefore that the selection policy and procedures in this context did not match the standards required for the valid and ethical interpretation and use of test scores.

O’Loughlin (2011) also reported variable levels of knowledge about IELTS (both of the test and the scores it produces) amongst university staff, including a lack of understanding as to what different IELTS scores imply about a student’s language ability, their readiness for university study, and their need for further English development. However, it was noted there may be little motivation for some of these test users to improve their assessment literacy within the Australian university admissions regime where readiness to study in the medium of English is judged simply on whether an applicant has attained the minimum IELTS scores (or other acceptable alternatives) required by the university.

IELTS is the most widely accepted English proficiency test in Australian universities although a broad range of alternatives including other proficiency tests, educational qualifications, preparatory English programs and previous university study in English are also deemed to satisfy their English entry requirements (Coley, 1999). Each university sets its own entry standards for local and international students although these standards must be consistent with the Federal Government’s regulations for issuing international student visas. In order to enter a program of study, all applicants must normally meet just two key criteria: the minimum academic and English requirements for that program. These two requirements are treated separately in the selection process, and applicants must demonstrate that they have clearly fulfilled each of them before they are selected. This is the current policy context in which the assessment literacy of IELTS test score users in Australian universities needs to be explored.

Aims of the study

The study investigated the assessment literacy needs of a range of university staff in relation to the IELTS test in Australian higher education and how well they are currently being met. These needs relate to a range of possible issues, including the purpose and content of the test, the meaning of test scores, the appropriateness of cut-off levels, the test’s validity, reliability, predictive power and comparability with other accepted forms of evidence of English proficiency. The key research questions addressed in this study were as follows:

Methodology

The study focused on the assessment literacy needs of a range of staff at two large metropolitan Australian universities, hereafter referred to as University A and University B. The two universities were included in the study to obtain a broader sample of participants. In both universities more than 25% of the student population has been international over the last five years. The IELTS test is the most well-established and well-known form of English proficiency evidence used for student selection at both institutions.

The study aimed to recruit volunteer participants working in a range of roles across both universities including admissions, marketing, academic, and both pre-course and in-course support language staff. It was anticipated that some staff would be currently using IELTS test scores directly in their work while others would be using them more indirectly or perhaps not at all. It was decided, however, that no staff who volunteered would be excluded from the project since their readiness to participate indicated that the IELTS test had some relevance to their work. Participants identified themselves as either test users or non-users at the first stage of data collection. The focus in this article is on the data gathered from the test users.

The study focused on both the ‘subjective’ needs (i.e., those identified by the study participants themselves as IELTS test users) and their ‘objective’ needs (i.e., those identified by other parties such as, in this study, the owners of IELTS (the British Council, Cambridge ESOL and IDP/IELTS Australia), as well as by the researcher and research assistant in relation to assessment literacy. The specific objective needs of IELTS test users were analysed by referring to the IELTS Handbook (2007), the IELTS Guide (2009), the IELTS official website (www.ielts.org), as well as the websites of the two universities targeted in the study.

In the first instance, data was gathered using an online survey powered by Survey Monkey™ (www.surveymonkey.com). Except for one open-ended item, the survey used exclusively forced-choice items. The first section of the survey asked respondents to identify their professional roles, whether they used the IELTS test and for what purposes. Only those who self-identified as test users in this first section were then directed to the second section of the survey in which they answered questions about the IELTS test information they used, the sources they used to access it and how useful it was to their work. In the third section, all respondents were required to read and evaluate the IELTS Guide (2009) in terms of its usefulness and informativeness and then to choose between other methods of learning about the test.

Semi-structured interviews were conducted on an individual basis with all survey respondents who volunteered to take part approximately one month after they had completed the survey. In the interviews these participants were first shown their questionnaire responses to refresh their memories and then invited to clarify and elaborate on their questionnaire responses. Field notes were taken during the interviews and audio-recordings were made of the interviews for subsequent analysis.

The survey responses were analysed in terms of comparative frequencies and, where appropriate, percentages. Each interview was summarized and the main themes were identified. Salient quotations which reflected these themes were then transcribed and coded in terms of whether they were (a) representative of all interviewees, (b) representative of a particular sub-group, for example marketing or academic staff, or (c) individual comments.

Results

Participant information

A total of 84 participants (43 participants from University A and 41 participants from University B) completed the online survey. In Questions 2 and 3 of the survey, participants were asked to identify whether they used the IELTS test in their jobs as well as to specify their work area. A total of 50 (23 from University A and 27 from University B) of these participants identified themselves as IELTS test users in the survey. The focus of this article is on the combined data collected from these participants who directly used the test in their work. Figure 1 provides a breakdown of the work areas of these test users in terms of frequencies and percentages.

OPEN IN VIEWER

Figure 1.

Work areas of IELTS test users (N = 50).

Staff interviews were conducted following completion of the staff surveys. Semi-structured interviews were conducted with all of the survey respondents who volunteered to take part in them. Fifteen (7 from University A and 8 from University B) of the 50 IELTS test users who completed the survey were interviewed. These interviewees included 2 admissions, 6 marketing, 2 academic 3 language and 2 other staff members.

Research questions

The results for each of the three research questions are reported below.

Research Question 1: What are the assessment literacy needs of university IELTS test score users?

This question was addressed using data from Questions 4 and 5 of the survey and associated interviews. Question 4 asked: ‘What do you need the IELTS test for in your current job? Please tick ALL responses that apply.’ Figure 2 shows the uses of the IELTS test for the 50 IELTS test users in terms of frequencies. The two main purposes identified by the survey respondents at both universities were advising prospective students about English language entry requirements and making student admission decisions. The interviews clarified which categories of staff carried out these two functions. While only admissions and some academic staff made admission decisions, staff in all categories (including marketing, language and other staff as well as admissions and academic staff) had an advisory role with prospective students. The range of matters these staff might provide to students included the IELTS entry requirements for particular programs, the English programs that were available to prepare for the test, where and how to register for the IELTS test, the nature and scoring of the test, and how much English instruction they might need to attain the scores required to enter different academic programs. However, individual interviewees normally mentioned only one or sometimes two of these topics. It is noteworthy that only five respondents used the IELTS test to set cut-off levels for university entry. This suggested either that the survey respondents were not sufficiently senior to do such work or that setting and revising minimum entry requirements were not frequently undertaken at either university. Only five respondents indicated and identified other purposes for using the test, which were communicating with other university staff and overseas agents, and making scholarship decisions.

OPEN IN VIEWER

Figure 2.

Purposes for using the IELTS test (N = 50).

Question 5 on the survey asked the respondents ‘What specific information about the IELTS test do you use in your job? Please tick ALL responses that apply.’ Fifteen topics, identified by the researcher and researcher assistant as potentially most relevant to the work of university staff on the basis of information about the test given on the IELTS official website (www.ielts.org), the IELTS Handbook (2007), the IELTS Guide (2009) and the IELTS Scores Explained DVD (2009), were listed as options for this topic. These topics were therefore included as the core informational needs of university staff. Figure 3 shows the results for the 50 IELTS test users for each of the fifteen topics listed in terms of frequencies. It indicates that more than 50% of the IELTS test users at both universities reported using information on the last four of these topics: the minimum IELTS entry scores for entry to courses at their university, the different components of the IELTS test, how long the IELTS test scores are valid, and the relationship between IELTS test scores and other evidence of English proficiency accepted by their university. Between 33% and 50% of respondents indicated that they also accessed information about the distinction between the Academic and General Training modules, the validity and reliability of the IELTS test scores, the meaning of the overall IELTS band scores and the recognition of the test locally and internationally.

OPEN IN VIEWER

Figure 3.

Information about the IELTS test used in the workplace (N = 50).

In the interviews, participants were asked whether any of the other types of specific information that they didn’t initially select in Question 5 of the survey might be helpful in their jobs. Ten of the 15 interviewees answered ‘no’ to this question because they did not require additional information about the IELTS test for their work. For example, one marketing staff member suggested that such information ‘would not be relevant because I operationalize a policy which is already in place’. In a similar vein, an admissions staff member stated that “a lot of these things [i.e., specific information about IELTS] I don’t know, but it wouldn’t make a difference [if I did]”. The other five test users suggested they might use some of the information they had not initially selected in Question 5. For example, one marketing staff member suggested she might use the information about the free online test report verification service that was explained in the IELTS Guide (2009). In more general terms, another marketing participant indicated she would be personally interested to know more about the test so that she could allay some of students’ concerns about the test.

The survey and interview results for this research question suggested that university staff are mostly concerned with knowing about or having access to ‘surface’ information about the IELTS test, especially the minimum test scores required for entry to particular courses at their respective universities. To a lesser extent they also use information about more ‘interpretative’ aspects of test quality such as the meaning, validity and reliability of the test scores. However, given that many of them also advised prospective students, one might expect more of them to have accessed and used a wider range of information about the test such as how to best prepare for the test, the nature and scoring of the test and how the scores are reported.

Research Question 2: How well are these needs currently being met?

This question was addressed using data collected from Question 6 and 7 of the survey and associated interview questions. Question 6 in the survey asked ‘Which sources of information about the IELTS test do you mainly use in your job? Please TICK all responses that apply.’ The majority of users (84% – 42/50) reported that they accessed their institution’s English language entry regulations which are available on each university’s website. Slightly less than half of them (42% – 21/50) used the IELTS official website. Only six of the 50 users indicated that they accessed the IELTS Guide (2009) which could be downloaded from the IELTS website. None of the respondents in this study used the IELTS Scores Explained DVD (2009) which must be purchased from the IELTS partners. Figure 4 shows the results for this question as frequencies.

OPEN IN VIEWER

Figure 4.

Sources of information about the IELTS test accessed by users (N = 50).

Of the 15 IELTS users interviewed, nine said they accessed their institution’s English language entry regulations available on their websites, mainly for checking information about the minimum acceptable IELTS scores for particular courses. Nine interviewees said they were aware of one or more of the other sources of information listed and the remaining six had no knowledge of them. Only two of them knew about the DVD but had not used it. Six of them said that they would ask colleagues in the language centre or IELTS test centre attached to their institutions if they required additional information about the test. For example, one marketing staff member said she did not know about the sources listed in the survey other than her own university’s English language regulations and said she “would ask for information from colleagues” in their affiliated language centre if needed.

Most interviewees suggested that there needed to be links on their respective institution’s website where information about its English language entry regulations were given to the IELTS official website. However, subsequent checking showed that both university websites did include these links, although they were perhaps not sufficiently prominent. One marketing staff member also suggested that better information could be included on her institution’s website to make it “more user-friendly to students by giving information about the test, how to practise and prepare”.

Question 7 of the survey asked ‘How useful is the information you currently access about each of the following aspects of the IELTS test? Please tick the appropriate number on the scale from 0–3, 0 = Not Useful, 1 = Slightly Useful, 2 = Useful and 3 = Very Useful’. The aspects listed were the same as those in Question 5. Figure 5 shows the results for this question in terms of frequencies.

OPEN IN VIEWER

Figure 5.

Usefulness of the information accessed about the IELTS test (N = 50).

On the one hand, 60% (30/50) of respondents saw the information provided about the minimum IELTS test scores for entry into specific courses at their university as being most useful. On the other hand, 48% of IELTS users (24/50) considered that the information about (a) how candidates can prepare for the test and (b) the IELTS test centres and how to register for the IELTS test’ was not useful. Again, this is surprising given that 78% (39/50) of them indicated that they advised prospective students about English language requirements in Question 4 of the survey (see Figure 2). Presumably, this advisory role would require them to provide more information than simply the minimum test scores required for entry into a particular course to these students.

The most common theme in the interviews was that if a selected topic was marked as not or slightly useful, it was because the interviewees did not see it as relevant to their job. One academic staff member involved in selection explained that information about certain aspects of IELTS (e.g. how the different components of the IELTS test are scored) was not useful. He indicated that knowing about them would make no difference to the selection process as it currently operated where decisions are based solely on whether the applicants have achieved minimum test scores and that his judgment was therefore not required. Comparing past practices to those of the present day he suggested that

you would work a lot harder and try to know more about these things when I was able to exercise some judgment in borderline cases, but once people are beholden to numbers that judgment goes out of the window … why do you have to bother knowing about things … with IELTS? The way the university does it, it’s just magic numbers.

This comment underscores a serious problem with current selection practices in Australian universities where admissions staff members only need to check whether an applicant has achieved the minimum acceptable IELTS scores to enter their course. As this interviewee suggests, they are usually not permitted to exercise judgment around admissions decisions and therefore do not require any real understanding of the test for this purpose. The rationale for this approach is that it is more equitable (i.e., every applicant is judged in exactly the same way) and efficient than a more multi-evidenced, holistic one. However, given the uncertainty of proficiency test scores (Spolsky, 2008), the fairness of entry decisions about English proficiency based on a single set of test scores is highly questionable.

In Questions 8 and 9 of the survey, data was also gathered on the perceived usefulness of a publication published by the IELTS partners for educational institutions, governments, professional bodies and commercial organizations, the IELTS Guide (2009). The respondents were first required to read its contents before answering Question 8: ‘How informative is each section of the IELTS Guide?’ on a four-point scale: not informative, slightly informative, informative and very informative. All sections of the IELTS Guide (2009) were found to be informative or very informative by 74% or more of respondents. The interviewees who dissented from this majority view claimed that it gave insufficient detail about the meaning of the different band levels and needed to include more research evidence to support its claim that it is the best test to use. Question 9 was an optional open-ended item which asked them ‘What other information do you think is missing from the IELTS Guide?’ One third of the comments recorded (12/36) focused on the need for more information about the meaning and interpretation of IELTS scores. An important issue raised by several respondents here was the so-called ‘jagged profile’ where there is considerable variation between one or more of an applicant’s band scores in the four tests of listening, speaking, reading and writing. This can mean that the applicant does not meet the specified minimum scores in one or more of these individual tests even though they have attained a satisfactory overall score. The same academic staff interviewee quoted above suggested in relation to this issue:

It would be lovely to see a person with [an overall band score of] 7.5 or 8.0 with a band of 5.0 or 5.5 in one area. Then you would be able to show samples of that to your PVC [Pro-Vice Chancellor] and say ‘Why wouldn’t I take that student?’ It would mean something, because the PVC probably knows less than me about this.

This comment highlights potential instances where discretionary judgement could be appropriately used to make admissions decisions. However, this would require a higher level of understanding of test scores than is necessary under the current selection regime, including an understanding of how the scores in each of the four sub-tests are derived and the relative significance of each of these scores as indicators of applicant’s language abilities to pursue their preferred course of study. A score of 5.5 in listening, for example, may not have as negative implications for academic success than the same score for writing. Finally, several interviewees suggested that the IELTS Guide (2009) could be improved by including more targetted material about the test that is relevant for different staff roles.

Overall, the results for research question 2 indicated that the needs of most, but certainly not all, IELTS test users in this study were reasonably well met. More than 80% of respondents at both universities relied on their institution’s English language entry regulations and, to a lesser extent, the IELTS official website for information about the IELTS test. They considered that the most useful information provided to them related to the minimum IELTS entry scores for entry to specific courses, followed by the relationship between IELTS test scores and other evidence of readiness to study in the medium of English, and the international and local recognition of the IELTS. This suggested they took a rather narrow view of their informational needs, especially in relation to advising prospective students. A strong majority of the IELTS users found all sections of the IELTS Guide (2009) to be informative although some respondents would have liked to know more about how to understand and interpret IELTS scores.

Research Question 3: What other approach(es) could be adopted to meet these needs?

The question was addressed using data from Question 10 of the survey and related interview questions in the interviews. Question 10 asked respondents ‘What other ways of learning about the IELTS test would be useful? Please tick all responses that apply.’

Figure 6 provides the results for Question 10. It indicates that the most popular alternatives to the IELTS Guide were online tutorials and information sessions. In the interviews the majority (10/15 or 67%) of the interviewees preferred these ways of learning about the IELTS test to the more static document, the IELTS Guide (2009). The main reason given by the interviewees who most preferred online tutorials was the ease of access and control in terms of how and when they could use them. They also emphasized their potential comprehensiveness and links to other helpful resources such as academic journal articles and research papers.

OPEN IN VIEWER

Figure 6.

Other useful ways of learning about the IELTS test (N = 50).

However, a small minority of interviewees were not in favour of online tutorials. One marketing interviewee said that, “you have a lot of on-line things: and you wouldn’t do them … unless they are mandatory, people wouldn’t do them”. The interviewees who preferred information sessions and workshops highlighted their interactive nature as the chief advantage. For instance, those interviewees who favoured workshops saw them as an opportunity for staff to deepen their understanding of IELTS scores through group discussion facilitated by experts.

An interesting alternative to the options listed in Question 10 of the survey suggested by two of the interviewees was the opportunity for staff to take a free IELTS test. According to a language staff interviewee, this strategy would help to dispel some misconceptions about the test on the part of some test users:

I am not sure what admission and other staff members know about IELTS … I think it [sitting the test] would let them learn more directly about some … basic things, like there is no pass and fail in IELTS … there are a lot of myths and misconceptions like that … IELTS in a way has become a magic word.

This is an important recommendation that underscores the value to university staff of sitting the test in order to demystify it. This kind of first-hand knowledge of the test is all too rare amongst student advisors, marketing, admissions and academic staff.

According to one admissions staff member, sitting the test would also help student advisors to have “more empathy and understanding when talking to students” which, in turn, would “make the students feel confident you know what you are talking about.” A major problem currently is that applicants are not always sure that they are given accurate advice about preparing for and taking the test, and then understanding the meaning and implications of their test scores.

Discussion

From one perspective, the implications of the main findings in this study are relatively straightforward. The IELTS partners should produce mostly online educational materials, including a tutorial, that build the assessment literacy of test users on a fairly limited range of topics so that they can fulfill their main roles in advising prospective students and making admission decisions within the current assessment régime in Australian universities. It is also desirable to cater to different kinds of university staff within the online tutorial format. In recent years, the IELTS partners have made a considerable effort to place more information in the public domain about the content and purpose of the IELTS test, the meaning and interpretation of band scores and the ways IELTS scores can be interpreted and used validly, reliably and responsibly in decision-making in higher education contexts. In response to the findings of this study in particular they are in the process of developing an online tutorial for university test users to supplement their revised test user guide, information sessions, workshops and multi-media products.

Yet, from a deeper perspective, there are fundamental questions to be addressed about current admission policies and practices in Australian universities which, in turn, raise issues for what assessment literacy about the IELTS test (and indeed all of the other measures used for the same purpose) is now and what it might become in the future. Currently, university applicants are assessed in a simple, lockstep manner. They must first of all meet the academic requirements and, second, the English entry requirements of a particular course at the university of their choice. As far as English proficiency tests are concerned, they must have obtained the minimum acceptable scores (which are often subject to downward pressure because of competition with other universities to attract full-fee-paying international students), and there is usually no flexibility around this decision. However, given the limitations and “inevitable uncertainty” (Spolsky, 1995, p. 358) of all proficiency test scores in terms of their validity and reliability, they should be carefully interpreted in relation to other relevant information about applicants’ language ability to strengthen the admissions decision. Such a shift in approach by Australian universities would require a major change in selection policy and procedures. The IELTS partners are clearly aware of this important issue. The IELTS Handbook (2007, p. 5), in particular, suggests that “receiving institutions should also consider a candidate’s IELTS results in the context of a number of factors including age and motivation, educational and cultural background, first language and language learning history.” This crucial recommendation correctly implies that IELTS minimum entry scores should not necessarily be rigidly applied in the decision-making process but, instead, should be interpreted in relation to these kinds of individual factors. Institutions need to take responsibility for gathering relevant information about individual applicants and implementing this more contextual, holistic approach to the interpretation and use of test scores. However, they need guidance from the IELTS partners, language testing experts and university authorities on how to do so. Similar procedures, which have been adopted at individual universities in countries such as the UK and the USA, also need to be mapped and evaluated to provide the IELTS partners with models of good practice. This is an important area for future development of resources for IELTS users. Adaptive and interactive online tutorials that provide more advanced, in-depth training to those members of staff who require it may be the best option currently available for this purpose.

If admissions staff in Australian universities were required to make informed, holistic judgments about the language proficiency of international applicants, then they (and the academic, marketing and English language staff who would need to understand how admission decisions were made) would need to be much better informed about the IELTS and other proficiency tests. Within such an admissions culture the term “test wiseness,” as redeployed by Taylor (2009), could be taken to mean the understandings that university staff need to acquire to promote and/or uphold valid and ethical test use as well as positive test impact. Of particular relevance here is one aspect of Fulcher’s (2012, p. 125) definition of assessment literacy, that is, “familiarity with test processes [and] awareness of [the] principles and concepts that guide and underpin practice” which would include validity, reliability, test fairness, interpretation and use of test scores, and related ethical considerations. Clearly, most test users do not need either the knowledge, skills and abilities to construct and evaluate language tests or an understanding of the wider historical, social, political and philosophical contexts of language testing, the other two of Fulcher’s components of assessment literacy. However, better knowledge of the processes, principles and concepts of language testing would enable university staff to exercise their responsibilities as informed test users and, it is hoped, to become advocates of valid and ethical practice (including both advice-giving and decision-making) in their educational institutions.

Conclusion

This study has identified the need for university proficiency test users to be educated for the valid and responsible interpretation and use of test scores and not simply trained to check whether applicants have met pre-specified minimum scores. This is consistent with a more cautious and informed view of language testing. A key element of this education is understanding the fallibility of proficiency test scores. Such a perspective necessitates adopting a more holistic judgment of an individual’s communicative language ability which, in turn, relies on using multiple sources of evidence to make important decisions affecting their life chances such as university entry. This kind of approach may help to mitigate the potentially negative effects resulting from overreliance on scores from a single proficiency test in this decision-making process. It may also help test users to provide more personalized and informed advice to university applicants about language learning and testing. This is a very important role performed by university staff but one which has not been given sufficient attention in the literature. Given their focus on the interpretation and use of test scores, these test users have more restricted but certainly no less important assessment literacy needs than language testers and teachers who need to acquire a much broader knowledge base ‘grounded in theory and epistemological beliefs, and connected to other bodies of knowledge in education, linguistics and applied linguistics’ (Inbar-Lourie, 2008, p. 396). The challenge is to make university staff aware of and take seriously their responsibilities as ethical test users whose advice-giving and decision-making may have far-reaching impacts on the lives of their prospective students.

Finally, the methodology used in the study on which this article is based provides a working model for other institution-based research into language assessment literacy. Such local studies could help to initiate a wider agenda of research sponsored by university authorities, as well as government and test agencies, since all stakeholders and test users have an interest in language assessments being understood and used appropriately. This kind of system-wide research would also help to enrich the theorization of assessment literacy for the various audiences to which language tests address themselves.