Test candidates’ attitudes and their relationship to demographic and experiential variables: The case of overseas trained teachers in NSW,Australia

Abstract

One measure of the impact of a high-stakes test is the attitudes that test takers hold towards it. It has been suggested that positive attitudes produce beneficial effects while real or anticipated negative experiences can result in the development of attitudes that erode confidence and potentially impact negatively on performance. This study investigated test taker attitudes by exploring the opinions, beliefs, and feelings of a group of overseas trained teachers preparing for a professional gate-keeping test, and examining correlations between attitudes and demographic and experiential factors. The participants were 105 candidates who were enrolled in a preparation course for the Professional English Assessment for Teachers. They were asked to complete a written survey questionnaire with three parts: to determine the nature of their attitude towards the test, to explore the relationship of attitudes and demographic data, and to investigate their perceptions of the sources of their attitudes. Results indicated that there was a slight predominance of negative attitudes, particularly among candidates who had unsuccessfully attempted the test. The main reported sources which correlated with a negative attitude were personal experiences and feelings as well as the impact of other people: notably teachers and other candidates.

Keywords

Attitude demographic English language proficiency experiential impact overseas professional overseas teachers

In Bachman and Palmer’s (1996) test-usefulness framework, one important component is test impact, which refers to how tests might affect different stakeholders. Part of test impact can be accounted for by test takers’ attitudes toward a test. However, there is no evidence in the literature to indicate this important issue has been adequately investigated. This might be because attitude is conceived as a construct related to face validity, an issue which Bachman (1990) had dismissed in his earlier writings, referring unequivocally to a ‘post mortem’, ‘burial’ or ‘interment’, and describing it as a ‘dead concept’ (pp. 285–287). Test taker reactions have not generally been seen as central to test validation, although the likelihood of a negative attitude towards a test impacting on test takers’ performance has been pointed out (Elder, Iwashita, & McNamara, 2002), and it has been acknowledged that negative attitudes may have serious implications for test validity (Elder & Lynch, 1996; Nevo, 1985; Spolsky, 1995). Messick (1989) also explicitly recommended that test taker perceptions be considered as a crucial source of evidence for construct validity. As such, the investigation of test taker attitudes should be considered by test constructors and users because they will affect the test’s usefulness (Chan, Schmitt, DeShon, Clause, & Delbridge, 1997; Stricker, Wilder, & Bridgeman, 2006).

The English proficiency level of overseas trained teachers is assessed in various ways in different contexts, depending on whether candidates will be teachers of English or other subjects and on the status of English in the country in question. For example, in Quebec the English Exam for Teacher Certification (EETC), is a requirement for graduation and for certification of teachers undertaking their pre-service training; Hong Kong teachers must attempt the Language Proficiency Assessment for Teachers of English (LPATE); and in Taiwan a high intermediate score in the General English Proficiency Test (GEPT) or equivalent is required for English teachers, while teachers of other subjects are required to attain an intermediate level.

In Australian state schools English is the language of instruction, so teachers’ language proficiency has major implications for the quality of education. As teachers’ roles across the curriculum involve modelling and shaping the language development of young people, it is expected that their English ability will be of a high standard. The testing instrument which will be discussed in this paper, the Professional English Assessment for Teachers (PEAT), reflects this requirement.

PEAT consists of four separate papers in which the skills of reading, writing, speaking and listening are assessed. All texts and simulations relate to school workplace contexts, and details of the tasks and exemplars are available to candidates on the website of the administering institution (UNSW Global, 2008). On their first sitting, candidates must attempt all the skills. If they do not pass all four papers, they are allowed to retake only the ones they have failed. The result for each section remains valid for a period of two years. Tests are held every six weeks and candidates are permitted to sit every second administration until they have achieved a pass grade in all the skills. The pass grade, an ‘A’, requires test takers to demonstrate their ability to understand and produce a range of text types that are found in the educational workplace.

PEAT was designed after wide industry consultation and research, and underwent extensive trialling and validation. It must be emphasized that it is not the purpose of this study to challenge its validity. However, regardless of actual test quality, candidates may respond in a range of ways to the assessment requirements and potentially self-limiting feelings of anxiety and resentment can arise. For many overseas trained teachers (OTTs), PEAT represents an unexpectedly challenging and sometimes daunting obstacle to the resumption of their professional lives, and negative attitudes are frequently expressed. The direct experience of two of the authors of this article, one a senior head teacher in a college offering preparation courses for PEAT, and the other with prior involvement in the development and implementation of the test, attest to the intensity of some candidates’ reactions. There are three common objections: first, many candidates object to being required to do the test at all; second, some believe that PEAT is more difficult than is necessary and that some local teachers would fail the test if they were required to take it; and finally, some candidates are suspicious of the validity and underlying purpose of PEAT.

A range of sources of frustration are reported. Teachers from outer circle countries (Kachru, 1985) are required to sit the test although they may have been speaking English all their lives and view themselves as completely proficient. Candidates frequently apply for exemptions on this basis and feel indignant when these are not granted. Exemptions are also sometimes sought by Languages Other Than English (LOTE) teachers, for whom English is not used extensively in classroom teaching; however, because teachers are required to use English for other roles in the school, it is not government policy to grant these. Candidates cannot gain access to their test papers and while those who are unsuccessful in a skill can apply for a re-mark report, they often remain puzzled by their apparent failure to progress. The expense of repeated sittings (currently AUD$400 for the complete test) can also contribute to discontent. Furthermore, many of the communities to which test takers belong are very close-knit, and news of any negative experiences travels readily to candidates who have yet to attempt the test, promoting anticipatory anxiety.

On the other hand, it must be emphasized that there are many candidates who are strongly supportive of the rights of Australian students, and who recognize the need for teachers to provide good models of language and literacy. Anecdotally, the candidates who are positive about the test and do not resent having to do it seem to be more readily able to focus on their language skills development. It has also been suggested that learners’ attitudes towards a language proficiency test may affect their performance on that test (Rasti, 2009), and although in the case of PEAT this link has not been demonstrated (nor is it the aim of this study to attempt to do so), it would be helpful for all concerned to understand more about the nature of candidate attitudes and their origins. As such, this study has the potential to inform test preparation teachers and course designers of problematic areas and to contribute to all stakeholders’ understanding, possibly rendering the process of language assessment less stressful for test candidates.

The main objective of this study, therefore, was to probe beyond the anecdotal evidence of candidate resistance to the test; firstly to explore whether it was in fact as widespread as it appeared, and secondly to identify demographic and general attitudinal factors which may be predictors of candidates’ attitudes, as well as exploring possible sources.

The research questions were as follows:

What is the overall pattern of overseas teachers’ attitudes to the test, and to what extent is this consistent with the anecdotal evidence?

Which demographic or experiential factors correlate with candidates’ reported attitudes to the test?

What experiences or sources of information do candidates report as having influenced their attitude towards the test?

Is there any relationship between the direction of candidates’ overall attitude to the test and their reported sources of information and experience?

Theoretical background and review of the literature

Attitude, defined by Allport as ‘a mental and neural state of readiness organised through experience, exerting a directive or dynamic influence upon the individual’s response to all objects and situations in which it is related’ (1971, p. 13) has been widely explored in studies of second language acquisition (Gardner, 1985; Gardner & Lambert, 1972; Oller, Hudson, & Fiu, 1977). Baker (1992) identified three components of attitude: affective, cognitive, and conative (relating to purposeful action). In learning, the experience of success and failure can influence attitude. According to the resultative hypothesis (Hermann, 1980), a positive attitude can result from success in language learning. This view of the relationship between success, attitude and integrative motivation is still widely supported today (Strong, 2006).

Affective reactions and beliefs can be viewed as a component of test impact which extends to the way in which assessment affects society as a whole (Hughes, 2002). Shohamy (1998) also uses the related term consequential validity, akin to Messick’s ‘consequential aspect of construct validity’ (Messick, 1996, p. 251), and in her research she has used empirical methods to explore the consequences of language tests in both educational and political spheres.

According to Bachman and Palmer’s (1996) model of test usefulness, one important aspect of impact is washback (how a test influences the way that language is taught and learned), and the attitudes of all stakeholders can have implications for this (Hamp-Lyons, 1997; Messick, 1996). For both teachers and candidates, positive attitudes towards the fairness and efficacy of a test can promote a sense of empowerment rather than helplessness, and create conditions for learning opportunities to become more effective.

While attitudes of teachers and academics have been widely investigated in relation to their effects on learning and assessment, the literature on attitudes of candidates to tests is much more limited. Studies of the opinions of teachers and university staff are typified by the investigation of McDowell and Merrylees (1998) on attitudes to IELTS. In this study, academics expressed high levels of approval of IELTS, but no test takers were surveyed or interviewed. A more comprehensive stakeholder attitudinal study was undertaken by Coleman, Starfield and Hagan (2003), in which the participants included teachers and students. Areas discussed included controversial ones such as perceptions of unprincipled activity (bribes, fraudulent certificates) incorporated in order to explore the levels to which anecdotal reports of abuse corresponded with actual experience. Some doubts were expressed regarding reliability, particularly among students reporting on their perceptions of the oral interview. The study focussed on the experience, beliefs and opinions, and did not explore affective reactions to the test.

Rasti (2009) also used questionnaires and interviews to investigate the attitudes of Iranian candidates who had attempted IELTS, looking at the relationships between test taker characteristics (sex, age, and educational background) and attitudes to the test. Of 60 participants, almost 80% had an overall positive attitude, and no significant relationships were found between participants’ attitudes and characteristics. Perhaps surprisingly, the IELTS score obtained did not seem to affect candidates’ attitudes.

In their study of washback and the TOEFL test, Alderson and Hamp-Lyons (1996) observed classes and interviewed both teachers and students about their attitudes to the test. This research was undertaken prior to the development of the TOEFL IBT, and thus only the ‘pencil and paper’ TOEFL was considered. The majority of teachers expressed a negative attitude to the test, feeling that it did not satisfy the criterion of authenticity. The study also found that washback was a more complex construct than had been hitherto acknowledged, but that attitude was influential in determining some teaching practices.

In China, a study by Han, Dai and Yang (2004) surveyed 1194 university and college teachers, and found a high incidence of negative attitudes towards the CET. No students were surveyed in Han et al.’s study, but Li (1999) administered surveys to both students and teachers to measure attitudes to the CET-Spoken English Test. A major study of washback conducted by Gu (2005) also involved investigating the attitudes of a wide range of CET stakeholders, including administrators, teachers and students.

Depending on the test taker’s purpose, all the above tests can be considered high-stakes, but differ from PEAT in a number of important ways. Unlike IELTS, TOEFL and CET, PEAT is a small-scale operation, with each administration accommodating a maximum of 60 candidates. The nature of this testing operation and the fact that commercial-in-confidence restrictions apply to the availability of its validation data mean little is published about it. Exemplars are available, but candidate numbers are too low to support the production of preparation textbooks, or to fund on-going research. Therefore, it is difficult for teachers or candidates to have an informed opinion about the quality of PEAT. In a society where there is considerable sensitivity to issues of discrimination and the rights of all to fair and equal treatment, the limited information in the public domain means negative attitudes and beliefs, however unjustified they may be, cannot be readily dispelled.

Operationalizing the construct of attitude

In the research we reviewed for this study, we found that the construct of attitude had been defined and operationalized in several different ways depending on the researchers’ area of focus, but none of these were fully suitable for the purposes of our study. References to Baker’s 1992 three-way division into cognitive, affective, and conative, were common in the general SLA literature on attitudes (e.g. Mantle-Bromley, 1995), but less so in the specific work on those of test takers, where a more unitary construct tended to be adopted. For example, Rasti (2009) noted that attitude was usually defined as ‘a disposition or tendency to respond positively or negatively towards a certain thing such as an idea, object, person or situation’ (p. 111), and did not problematize or further subdivide this conceptualization of the construct.

Terminology is still at a developing stage in this field, with consistency yet to emerge. To consider just one study: in their work on test taker perceptions of difficulty, Elder, Iwashita and McNamara (2002) refer to test taker reactions on nine separate occasions. This includes one reference to affective reactions, possibly indicating that there may be other kinds (p. 262), and once to ‘test-taker attitudes and reactions’ (p. 363), which suggests attitudes and reactions may be viewed as different (although not treated separately in the data analysis). The authors state that attitude to a task is ‘defined in our study as enjoyment’ (p. 356), and this is well justified in terms of the questions being investigated in their study, but cannot be generalized to other contexts. The term test taker perceptions was also used on numerous occasions in this article, the majority of times in collocation with ‘of task difficulty’; in Baker’s taxonomy this usage would probably be classed as cognitive rather than affective. Attitude is a hypothetical psychological construct, which cannot be measured directly, and the field would probably benefit from more explicit standardization in the use of the terms applied to label it, as well as more discussion of its indicators and sub-components.

The information we sought to capture in our questionnaire was designed to indicate an ‘overall’ construct of attitude, which was measurable using Likert scales and could be expressed numerically. The study was informed and motivated both by pre-existing theory and by factors of concern that had been observed in test administration and preparation classes: evidence of stress, pessimism and anger; negative beliefs; and opinions about the need for change. As such, we designed the questionnaire to be as broad as possible and we accepted it was beyond the scope of our current study to attempt to separate the component factors more meaningfully. A factor analysis conducted on the pilot questionnaire enabled us to identify items that did not load onto one factor although there was no clear reason for this beyond the possibility of ambiguity in their wording, or because they were too linguistically complex for the participants to interpret satisfactorily. These items were eliminated. Although we had used the terms ‘reactions’ or ‘perceptions’ in initial stages of the project, in the final analysis we did not find them to be helpful in describing the construct. Reporting on behaviours was outside the scope of our study, but we were able to distinguish two strands of the cognitive component of the construct: beliefs (that a proposition is or is not true) and opinions (that an actual or hypothetical action should or should not happen). We also identified emotions (corresponding to Baker’s ‘affect’).

Figure 1 shows how these constitute the construct of attitude. As the diagram shows, there may be interrelationships between these factors; for example, a belief that the test was unfair could lead to anger, which could reinforce an opinion that an individual or nationality should be exempt from taking the test. Indeed, factor analysis of the final 20 questions on attitude failed to resolve into affective or cognitive, suggesting that these all do, in fact, load onto one factor. For the purposes of investigating correlations with potential causal factors, we labelled this ‘overall’ attitude.

Figure 1.

The construct of overall attitude as operationalized in the present study

The study

Participants

The participants in this study were drawn from three colleges of technical and further education in a metropolitan city in Australia. Of the cohort of 105 overseas-qualified teachers, 11 were male and 94 female. They were invited to complete the attitude questionnaire at the beginning of their course of study.

In order to look for possible clusters in the incidence of certain attitudes, we roughly classified the teaching subjects represented within the group into three categories of Sciences (including Chemistry, Biology, Mathematics, and Physics); Social Sciences (including History, Geography, Economics, Commerce, Business Studies, and Physical Education); and Languages (including ESL, Chinese, Japanese, Korean, Turkish, French, and Arabic).

The candidates came from a range of countries of origin. This range was too diverse for countries to be considered individually, so for the purposes of comparison they were grouped according to broad geographical regions: Group 1 included 53 teachers from India, Bangladesh, and Sri Lanka; Group 2 included 23 teachers from Vietnam, China, Japan, Korea, Taiwan, the Philippines, and Fiji; and Group 3 included 29 teachers from a broader category consisting of Egypt, Syria, Iran, Iraq, Afghanistan, Turkey, Russia, Serbia, Bosnia, Czechoslovakia, and Peru.

The time the candidates had spent in Australia varied from less than two months to as much as 17 years. Using the median (three years) they were divided into two groups of longer-term residents (N = 42) and new arrivals (N = 59).

Some of the candidates had direct or indirect experience of private or government schools, either through casual teaching or through having their children enrolled as students. The participants were thus divided into having teaching experience (N = 34), or not (N = 69); having children at school (N = 51), or not (N = 50). The latter was also thought to confer a possible advantage in that as parents they would have had the chance to develop some familiarity with spoken and written genres of the Australian education system. In terms of their experience of sitting for the PEAT test, they were divided into two groups of those having attempted the test once or more (N = 47) and those with no experience of sitting for the test (N = 56).

Table 1 summarizes participants’ demographic and experiential features.

Table 1.

Participants’ demographic and experiential characteristics

Category	Group	Frequency & percentage
Country of origin	G1	53 (50.5%)
	G2	23 (22%)
	G3	29 (28%)
Area of teaching	Sciences	49 (47%)
	Social sciences	11 (10%)
	Languages	36 (34%)
Length of residency	New	59 (56.2%)
	Old	42 (40%)
Teaching experience in Australia	Yes	34 (32.4%)
	No	69 (65.7%
Children at school	Yes	51 (48.6%)
	No	50 (47.6%)
Experience of PEAT	Yes	47 (44.8%)
	No	56 (53.3%)

As part of their background profile, candidates were asked whether they expected to pass the test, either soon or at some time in the future. Most indicated they expected to pass the test; 92 out of 96 responses to the item ‘I expect to pass the PEAT at some time in the future’ belonged to the category of ‘agree and strongly agree’ and only four disagreed or strongly disagreed with this item. Also, in response to ‘I think I will pass the PEAT soon’, 70 candidates chose ‘agree or strongly agree’ and 20 marked ‘disagree or strongly disagree’.

Data collection instrument

To collect the required data for the study a questionnaire was developed, piloted, then administered to the participants. The questionnaire included three parts. The first part consisted of demographic questions as well as questions designed to gauge participants’ expectations about passing the test, which were discussed in the ‘Participants’ subsection above, their experience of Australian schools, their opinions about Australian teachers and their opinions about the migrant experience in general. The second part included 20 items on participants’ attitude to PEAT (see Table 2) and the third part of 15 items on the sources of information participants thought had contributed to the formation of their attitude to PEAT. The attitude items in the first part, and the second and third parts of the questionnaire, used a five point Likert-scale with 5 indicating ‘strongly agree’ and 1, ‘strongly disagree’. Overall attitude, derived from section two of the questionnaire, was operationalized as indicated in Figure 1 and was measured through 20 items related to the components of attitude shown in that figure.

Table 2.

The frequency and percentage of participants’ responses to different items on the attitude section (part 2) of the questionnaire

Item	Agree + Strongly Agree Frequency (Percentage)	Don’t know Frequency (Percentage)	Disagree + Strongly Disagree Frequency (Percentage)
1. All overseas trained teachers should have to do PEAT	45 (43%)	2 (2%)	56 (53%)
2. The tasks in the test are a good reflection of what teachers have to do in their work	72 (69%)	5 (5%)	27 (26%)
3. I feel angry that I have to do the PEAT	44 (42%)	8 (7.6%)	49 (46.7%)
4. I sometimes feel like giving up	57 (54%)	1 (1%)	44 (42%)
5. If I had known all about the PEAT, I would not have migrated to Australia	22 (21%)	4 (3.8%)	77 (73.3%)
6. The PEAT makes me depressed	62 (59%)	11 (10.5%)	29 (27.6%)
7. Some Australian born teachers would not pass the PEAT¹	76 (72.4%)	19 (18%)	8 (7.6%)
8. The PEAT is a fair test	38 (36.2%)	18 (17%)	47 (44.8%)
9. The PEAT is necessary	52 (49.5%)	11 (10.5%)	39 (37%)
10. It’s fair that I should have to pass the test before I can work in a DET school²	53 (50.5%)	9 (8.6%)	41 (39%)
11. My English is good enough: I should not have to do PEAT	49 (46.7%)	9 (8.6%)	44 (42%)
12. People from my country shouldn’t have to do PEAT	20 (19%)	12 (11.4%)	72 (68.6%)
13. I feel anxious about the PEAT	73 (69.5%)	9 (8.6%)	23 (21.9%)
14. This test is just a way of excluding migrant professionals from the workplace	45 (43%)	10 (9.5%)	50 (47.6%)
15. I feel embarrassed because I haven’t passed the PEAT yet	41 (39%)	10 (9.5%)	52 (49.5%)
16. It is possible to be good enough to work in a school and still not pass	73 (69.5%)	11 (10.5%)	20 (19%)
17. I feel confident about the PEAT	49 (46.7%)	8 (7.6%)	45 (43%)
18. My feelings about the PEAT are getting worse	24 (22.9%)	11 (10.5%)	68 (65%)
19. The tasks I will have to do in school are different from the ones in the test	47 (45%)	16 (15%)	37 (35%)
20. I don’t mind having to do the PEAT	39 (37%)	7 (6.7%)	56 (54%)

The questions related to the attitude construct in the following way. Items designed to gather information about emotions, (3, 4, 6, 13, 15, 17, 18, and 20) contained an indicative word or phrase, such as ‘angry’, ‘embarrassed’, ‘confident’, ‘don’t mind’, or referred directly to ‘my feelings’. Items designed to investigate beliefs about the test (2, 8, 9, 14, 16, and 19) contained a statement related to the current test such as ‘PEAT is a fair test’. Opinion questions (1, 5, 7, 10, 11, and 12) referred to alternative possibilities and contained a hypothetical ‘would’ or ‘should’.

The reliability of this part of the questionnaire was calculated using the Cronbach’s Alpha index. Listwise deletion, based on all variables in the procedure, excluded 21 cases for missing data and included 84 (80%) as valid cases. The Cronbach’s Alpha was 0.918 indicating a high internal consistency for the operationalized construct.

The third part of the questionnaire had 15 Likert-scale items asking participants to report different sources of information and experience they thought informed them to develop their attitude toward the test. The reliability of this section of the questionnaire was also calculated using the Cronbach’s Alpha index. Listwise deletion based on all variables in the procedure excluded 27 cases for missing data, and included 78 (74.3%) as valid cases. The overall Cronbach’s Alpha for this section was 0.77.

Data collection and analysis procedures

To collect the data the questionnaire was administered during the first three weeks of participants’ 18-week-long PEAT preparation courses. Administrators of the questionnaire explained its purpose and obtained written permission from each participant surveyed; all participants were assured their responses would be treated confidentially and anonymously.

The following statistical analyses were performed on the collected data.

To check the reliability of the second and third part of the questionnaire, Cronbach alpha as an index of internal consistency was used.

To address Research Question 1, participants’ attitudes to different aspects of PEAT, frequencies and percentages of their responses to each individual item on the second part of the questionnaire were calculated. To do this, the categories of ‘strongly agree’ and ‘agree’ were merged as was the categories of ‘strongly disagree’ and ‘disagree’. To find out if the differences in participants’ responses were statistically significant or not, a non-parametric test of Chi-square was run on the observed frequencies.

While participants’ responses to each individual item in the second part of the questionnaire provide useful information about how they reacted to each question, we were also interested in a collective index of the participants’ attitude. In order to achieve this, we reversed the scores for the seven items that were worded in such a way that agreement with the statement would indicate a positive attitude, and used these to calculate the means (Table 3). It would have been equally possible to reverse the statements which indicated negative attitudes instead and produce an overall positivity score, but as we were primarily interested in the correlation of negativity and other factors this approach seemed more logical and transparent.

We added up the value of each individual response on the 20 items to get an overall score for each participant’s attitude to PEAT. We calculated the mean of this score, and named it ‘overall attitude’, and used it to determine the relationship between participants’ attitude to the test and their personal and demographic characteristics (Table 4). Furthermore, based on the median of the overall attitude score, participants were divided into ‘more positively’ and ‘more negatively’ oriented groups. In order to address Research Question 2, we then ran a cross-tabulation between these two groups and three demographic (experiential) variables to find out if any of the three experiential factors were predictors of the tendency to fall into one or other attitude group (Table 5).

Factor analysis was used to investigate the underlying factors of candidates’ perceptions of the sources on information that had shaped their attitude to PEAT (Research Question 3). Based on the items loaded on each factor as presented in Appendix 1, three factors were identified. To find out if there was any relationship between the attitude to the test, and the three underlying sources of the participants’ perception of their attitude (Research Question 4), a correlation was calculated (Table 6).

Table 3.

Item means for each of the three components of the attitude construct

Belief items	M	Opinion items	M	Emotion items	M
2*	2.38 (1.14)	1*	3.09 (1.41)	3	3.09 (1.4)
		5	2.25 (1.28)	4	3.2 (1.42)
8*	3.23 (1.25)	10*	2.9 (1.21)	6	3.47 (1.35)
9*	2.82 (1.24)	11	3.15 (1.24)	13	3.69 (1.17)
14	2.92 (1.32)	12	2.41 (1.07)	15	2.98 (1.41)
16	3.71 (1.11)			17*	3.07 (1.28)
19	3.14 (1.21)			18	2.48 (1.26)
				20*	2.83 (1.25)
M	3.03	M	2.76	M	3.10

Table 4.

Candidates’ attitudes to Australian teachers and the migrant experience and the relationship with their overall attitude to PEAT

	No	Agree + Strongly Agree	Disagree + Strongly Disagree	M	SD	Pearson correlation
1e1. Native speaker teachers in Australia are sometimes inaccurate in their written language	82	72	10	3.79	.90	.326^**
1e2. Native speaker teachers in Australia are sometimes inaccurate in their spoken language	96	70	26	3.69	1.02	.303^**
1e3. Australian-educated teachers should also have to do the PEAT	83	67	16	3.56	1.22	.304^**
1f1. In Australia migrants are discriminated against	81	38	37	1.31	.62	.519^**
1f2. In Australia, migrants and Australian born people have equal access to employment opportunities	84	41	53	1.15	.77	−.392^**

correlations are significant at p < .01 level

Table 5.

Results of the cross-tabulations on demographic/experiential variables and attitude to PEAT

	Overall attitude		Total	χ ²
	More positive	More negative	Total	Asymp. sig. (2-sided)
Residency in Australia				.142
New	27	20	47
Old	13	19	32
Teaching experience				.04
Yes	9	17	26
No	33	23	56
Test experience				.003
Yes	13	26	39
No	28	14	42

Table 6.

The top five mean scores of the sources of forming attitude to PEAT

Item	M	SD
2. My test preparation teachers’ opinions about the PEAT	3.87	.91
12. My feelings about my ability to pass	3.82	1.05
1. My personal experience attempting the PEAT	3.67	1.08
5. Information I have read in the exemplar and website	3.66	1.04
6. My contact with DET regarding the PEAT	3.45	1.04

It is important to note that this table only identifies the sources as influential; the resulting attitudes may of course have been either positive or negative.

Results and discussion

Candidates’ responses

Table 2 presents the frequency and percentage of responses on individual items of the attitude part of the questionnaire.

To check if the differences in the observed frequencies of the categories of ‘strongly agree & agree’ and ‘strongly disagree & disagree’ were statistically significant, the non-parametric Chi-square test was used. Results indicated that the difference between the categories for all the 20 items was statistically significant (df = 2, p < .001) meaning the observed frequencies could be generalized beyond the sample. The following patterns could thus be extracted.

In Table 2, items 9 and 10 show that the study participants generally support the use of language tests for OTTs and the majority also believe that OTTs should satisfy language requirements before being allowed to work as teachers in Australia. The majority of candidates did not feel that as representatives of their source country (item 12) they merited exclusion from assessment; however, the response to item 1 suggested that some candidates believed there were circumstances under which exemptions could be made.

Some of the affective responses were clearly negative, with a majority (69.5%) reporting they felt anxious as a result of PEAT, while 59% were depressed. In addition, 54% responded that they sometimes felt like giving up. While only 42% were angry and 39% embarrassed, it could still be argued that even this level of agreement might be cause for concern in terms of test impact. However, the majority (65%) did not consider their feelings were getting worse and, consistent with the results of question 1c in the first section of the survey, almost half of the cohort reported they were confident they would pass.

In order to further explore the attitude construct discussed in the ‘Operationalizing the construct of attitude’ section, we looked separately at the questions, which had been designed to investigate each component. It is informative to see the pattern of responses to the items of each of the three components of the attitude construct; namely, belief, opinion, and emotion. Table 3 presents item means for the conceptualized components of the construct of attitude. As explained in the ‘Data collection and analysis procedures (c)’ subsection above, in order to achieve an overall consistent value for each component the means of the ‘positively framed’ items (starred) have been reversed.

As can be seen from Table 3, the item means for each of the three components of the attitude construct are greater than 2.5, showing quite an even distribution but with a slight overall tendency toward negativity. Based on the mean magnitudes, the greatest negativity towards the test is represented in the area of emotions.

An overall attitude score for the cohort was also calculated with a possible maximum score of 100. This was calculated using the reversed scores on positively framed items and hence a more positive attitude was represented by a lower score. This calculation yielded a mean of 60.36, a median of 59, and the smallest mode value of 42 (as multiple modes existed), and a standard deviation of 15.85.

Candidate attitudes and demographic/experiential factors

The second research question addressed by this study concerned the relationship of demographic and experiential factors with attitudes to the test. This was important to establish because if any action is to be taken to counter potentially self-defeating attitudes, it is necessary to establish a profile of the kind of candidate who is likely to hold them.

Participants’ opinions about both Australian teachers’ speaking and writing proficiency and their migrant experience, and the relationship between these and their overall attitude to PEAT, are presented in Table 4.

The Pearson correlation shows the relationship between each individual’s agreement with each statement and overall attitude to PEAT. There was a significant correlation³ between negative perceptions of the treatment of migrants in Australian society in general and negative attitudes towards the test. In addition, the significant correlations between negative attitude and agreement with the statements in section 1e of the questionnaire showed that candidates were more likely to hold negative attitudes to the PEAT if they had doubts about the language proficiency of teachers who were already working in Australian schools.

In order to investigate other key variables, the cohort was divided into two groups falling above and below the median attitude score (59), and these were labelled ‘more positive’ and ‘more negative’ respectively. It should be emphasized these are relative categories only, and did not represent overall positive or negative attitudes. Cross-tabulations were then performed to see if any of the demographic factors were predictors of the tendency to fall into one or other attitude groups. Results of the cross-tabulations are presented in Table 5.

As can be seen in Table 5, a significant difference was found between those who had teaching experience in either government or private schools and those who did not in terms of their attitude to PEAT. Teaching experience appeared to lead to a more negative overall attitude to the test. It had been expected that the experience of sitting for the test might result in more negative attitudes, and this was confirmed. Unambiguously, test experience (and by implication in this cohort, the experience of failure) correlated with more negative attitude. Length of residency in Australia, however, did not predict any significant difference among the teachers in terms of their attitude to PEAT.

Candidates’ perceptions of the origins of their attitudes

To answer research questions three and four, besides the more objective demographic, experiential and demographic data, candidates’ perceptions of the factors that had shaped their experiences were also investigated.

In terms of the individual item mean scores, the five most commonly reported sources of attitude for the whole cohort are presented in Table 6.

In order to investigate possible sources of negativity, Principal Component Analysis (PCA) with Varimax rotation was used to perform factor analysis on the third part of the questionnaire (Appendix 1). Three underlying factors were extracted and named as ‘macro context’ (items 7, 8, 9, 10, 13, 14), ‘other people’ (items 2, 3, 4, 6), and ‘personal feelings and experience’ (items 1, 5, 11, 12, 15) based on the items which loaded on each factor.

At the level of underlying factors, to find out if participants’ attitude to the test correlated with any of the three underlying sources for the development of such an attitude, namely, ‘macro context’, ‘other people’, and ‘personal feelings and experience’, a correlation was run among the three factors and each participant’s overall attitude score. As can be seen in Table 7, there was a correlation between the overall negative attitude to the test and two of the underlying sources: ‘other people’ and ‘personal feelings and experiences’.

Table 7.

The correlation between overall negative attitude to PEAT and the three factors perceived to shape attitude

	Factor 1 (Macro context)	Factor 2 (Other people)	Factor 3 (Personal experience & feelings)
Overall attitude	−.131	−.265*	−.320**
Sig.	.233	.015	.003

This suggests the macro context did not contribute to participants’ development of more negative attitude to the test, rather it was their personal feelings and experience and input from colleagues, classmates and teachers which were seen as significant.

To summarize the findings of the study, we can see that the answer to our first research question was that overall the participants’ attitude toward the test was only slightly negative, and that there were numerous examples of positive responses. Where negativity was found, it tended to be centred on emotions. Of the variables exploring demographics, experiential factors and attitudes not directly related to the test (question 2), the factors indicating a significant difference among the participants in terms of their attitude toward PEAT were as follows: having test experience, having teaching experience at schools, a negative attitude to Australian teachers, and a negative attitude to the treatment of migrants in Australia in general. These were the real, direct and tangible experiences of the participants. Based on the participants’ perceptions of what information sources contributed to their attitude to PEAT (questions 3 and 4), the results of the correlation confirmed the importance of personal experience and indicated candidates thought other people had also been influential in shaping their attitudes.

Despite the fact that the three components of the attitude construct loaded on a single factor, the fact that different means were obtained for each component was a positive indicator of its usefulness. However, in processing the data we became aware that very minor differences in wording might lead to different interpretation of items. For example, question 20, ‘I don’t mind having to do the test’ was designed to identify a positive emotion, while question 10, beginning ‘It is fair that I should have to . . .’ was structured to elicit an opinion. In future research, it would be possible to investigate the statistical robustness of the questionnaire more along the lines suggested by Alderson and Banerjee (2000). Moreover, it would be interesting to extend the study if it became possible to gain access to a wider range of participants, rather than only the ones in test preparation classes.

Conclusions

Language tests can be stressful and unpleasant experiences in many ways, but it has been shown (Rasti, 2009) that not all test candidates hold overall negative attitudes towards them. Even if a conclusive link between test effectiveness and candidate attitude has yet to be demonstrated, the experience of preparing and presenting at the test can be deeply influenced by the manifold components of attitude.

In our study, the matching of attitudes to demographic and experiential factors yielded results that were interesting and sometimes surprising. Despite anecdotal evidence that candidates from particular countries of origin where English is widely used were more likely to express their discontent at various aspects of the test experience, no significant correlations were found between underlying attitude and nationality. This was interesting because there are certainly important questions surrounding the rights of speakers of World Englishes to have their variants accorded equal recognition with those of the inner circle, and it is possible that in the future the designers of gate-keeping tests will have to take this into account to a greater extent than they have done. It is also not impossible that a nationality effect does exist, but the analysis in this study failed to capture it because of the large number of countries of origin represented in the study, and the fact that respondents were not grouped according to the status of English in these countries. This could be considered in future studies.

The fact that teaching area was also found not to be a significant predictor again conflicted with anecdotal evidence; as was discussed in the introduction, the debate about whether teachers of languages other than English should be exempt has been a vigorous one, and it was anticipated this might be reflected in the attitudinal data. Similarly, it was counter to expectation that the length of time the candidates had spent in Australia did not prove to be a predictor. The most likely explanation for this is that length of residence is not really a single factor, but consists of a complex cluster of independent variables, whose impact on attitude may cancel each other out. All migrants experience their time in Australia in different ways, depending on factors such as their level of integration with the broader community, the opportunities for language and personal development afforded by their work, and a range of unpredictable individual factors. Further investigation of this area may shed more light on some of these questions.

The correlation between experience of Australian schools and more negative attitudes was also quite surprising. In general it has been the practice of test preparation courses to encourage participation in schools in order for socialization of overseas trained teachers into Australian workplace genres to occur naturally. Quite a number of candidates have taught casually or worked as teaching assistants. It could be hypothesized that the impact on attitude might be due to an increased sense of self-confidence at being able to function in the role, leading to greater resentment of the PEAT as obstacle. However, data from the current study cannot confirm this. Nor can we conclude that teaching experience is not useful to test performance in other ways, and further research should investigate this before any changes to preparation courses can be recommended.

Some of the specific beliefs expressed by the majority of candidates were slightly alarming. For example, beliefs concerning the low level of language proficiency of some local teachers were widely held. Whatever the origins or level of veracity, this could suggest, as a logical extension, a sense of injustice at being required to sit an exam that candidates suspect local teachers could not pass. The cohort’s reported limited experience of Australian schools suggested this opinion must have been based at least partly on hearsay, but it would be interesting to further investigate this.

In addition, it is cause for concern that general negative attitudes to the experience of migrants in Australia were seen to correlate with more negative attitudes to the test. Counterbalancing these views presents a challenge to stakeholders who seek to help candidates shed self-limiting beliefs, because it is difficult to modify attitudes to a test if it is perceived as part of a broader pattern of exclusion and discrimination. However, the fact that almost three quarters of the cohort would not have changed their plans to migrate to Australia even if they had been fully informed about the PEAT suggested they did not view it as an insurmountable obstacle.

Implications of the study

In the light of the findings of this study, it would probably be beneficial for the providers of PEAT, and indeed the designers of any high-stakes professional entry test, to reflect on ways in which candidates could obtain more detailed information about the objectives, and the effectiveness (validity, reliability and overall integrity) of the test. Better ways need to be found to reassure all candidates that there is no discriminatory agenda, that their potential contribution to Australian schools is valued, and that rigorous standards are imposed on local as well as overseas teachers with the aim of providing quality education to Australian primary and secondary students.

The study also confirmed that at least in the candidates’ own perception, one of the major factors shaping opinions and attitudes were ‘other people’. The three items loading most strongly onto this factor were the following: colleagues and friends, test preparation teachers, and other candidates. While it is difficult to control the spread of rumours and hearsay around a community of candidates, particularly when they are interacting regularly in a classroom situation, the importance of the role played by test administrators and test preparation teachers in maintaining morale and dispelling unhelpful negative beliefs merits greater attention.

As a part of the evaluation of test effectiveness, the results of this study suggest that is important not only to monitor candidate attitudes, but also to investigate their sources. In this way groups at risk can be identified, and their concerns and potential misapprehensions appropriately addressed.

Footnotes

Appendix

Appendix 1:

Rotated component matrix^a for section 3

	Component
	1	2	3
My personal experience attempting PEAT	−.041	−.068	.659
My test preparation teachers’ opinions about PEAT	.047	.620	.128
The opinions of my colleagues or friends	−.033	.822	.037
Information I have heard from other candidates	−.069	.653	.142
Information I have read in the exemplar and website	.150	.398	.447
My contact with DET regarding PEAT	.357	.455	.047
My contact with UNSWIL regarding PEAT	.530	.270	−.179
My experience of Australian schools	.684	.016	.237
My personal experience of racism or discrimination	.558	−.270	.153
My general attitudes to how migrants are treated in Australian society	.747	−.118	−.054
My opinions about Australian English	.410	.058	.593
My feelings about my ability to pass	.144	.250	.672
My experience of Australian teachers working in schools	.543	.273	.116
My general views on social justice	.595	.122	.442
the cost of the test	.006	.109	.540
Extraction Method. Principal Component Analysis.
Rotation Method. Varimax with Kaiser Normalization.

Rotation converged in 5 iterations.

Notes

References

Alderson

J. C.

Banerjee

(2000). Impact and washback research in language testing. In Elder

Brown

Grove

Hill

Iwashita

Lumley

McNamara

O’Loughlin

(Eds.), Experimenting with uncertainty: Essays in honour of Alan Davies (pp. 150–161). Cambridge: Cambridge University Press.

Alderson

J. C.

Hamp-Lyons

(1996). TOEFL preparation courses: A study of washback. Language Testing, 13, 280–297.

Allport

G. W.

(1971). Attitudes. In Thomas

(Ed.), Attitudes and behaviour. Harmondsworth: Penguin.

Bachman

L. F.

(1990). Fundamental considerations in language testing. Oxford: Oxford University Press.

Bachman

L. F.

Palmer

A. S.

(1996). Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.

Baker

(1992). Attitudes and language. Clevedon, UK: Multilingual Matters.

Chan

Schmitt

DeShon

R. P.

Clause

C. S.

Delbridge

(1997). Reactions to cognitive ability tests: The relationships between race, test performance, face validity perceptions, and test-taking motivation. Journal of Applied Psychology, 82(2), 300–310.

Coleman

Starfield

Hagan

(2003). The attitudes of IELTS stakeholders: Student and staff apperception of IELTS in Australian, UK, and Chinese tertiary institutions. IELTS Australia Research, 5, 20–34.

Elder

Iwashita

McNamara

(2002). Estimating the difficulty of oral proficiency tasks: What does the test taker have to offer? Language Testing, 19(4), 347–368.

10.

Elder

Lynch

(1996). Public perceptions of basic skills tests and their ethical implications. Paper presented at the 20th Language Testing Research Colloquium, Monterey.

11.

Gardner

(1985). Social psychology and second language learning: The role of attitude and motivation. London: Edward Arnold.

12.

Gardner

Lambert

(1972). Attitudes and motivation in second language learning. Rowley, MA: Newbury House.

13.

(2005). Positive or negative? An empirical study of CET washback on college English teaching and learning in China. ILTA Online Newsletter, 2.

14.

Han

Dai

Yang

(2004). Problems with College English Test as emerged from a survey. Foreign Languages and Their Teaching, 179(2), 17–23.

15.

Hamp-Lyons

(1997). Washback, impact and validity: Ethical concerns. Language Testing, 14(3), 295–303.

16.

Hermann

(1980). Attitudes and success in children’s learning of English as a Second Language: The motivational vs. resultative hypothesis. English Language Teaching Journal, 34, 247–254.

17.

Hughes

(2002). Testing for language teachers. Cambridge: Cambridge University Press.

18.

Kachru

B. B.

(1985). Standards, codification and sociolinguistic realism: The English language in the outer circle. In Quirk

Widdowson

H. G.

(Eds.), English in the world: Teaching and learning the language and literatures. Cambridge: Cambridge University Press.

19.

(1999). Language test development and College English Test – Spoken English Test. Foreign Languages and Their Teaching, 126, 28–30.

20.

Mantle-Bromley

(1995). Positive attitudes and realistic beliefs: Links to proficiency. The Modern Language Journal, 79(3), 372–385.

21.

McDowell

Merrylees

(1998). Survey of receiving institutions use and attitude towards IELTS. IELTS Australia Research, 1, 15–23.

22.

Messick

(1989). Validity. In Linn

R. J.

(Ed.), Educational measurement. New York American Council on Education/Macmillan.

23.

Messick

(1996). Validity and washback in language testing. Language Testing, 13(3), 241–256.

24.

Nevo

(1985). Face validity revisited Journal of Educational Measurement, 22(4), 287–293.

25.

Oller

J. W.

Hudson

A. J.

Fiu

P. F.

(1977). Attitudes and attained proficiency in ESL: A sociolinguistic study of native speakers of Chinese in the United States. Language Learning, 27, 1–29.

26.

Rasti

(2009). Iranian candidates’ attitudes towards IELTS. Asian EFL Journal, 11(3). Retrieved from: www.asian-efl-journal.com/September_2009_ir.php.

27.

Spolsky

(1995). Measured words: The development of objective language testing. Oxford: Oxford University Press.

28.

Stricker

L. J.

Wilder

G. Z.

Bridgeman

(2006). Test takers’ attitudes and beliefs about the Graduate Management Admission Test. International Journal of Testing, 6, 255–268.

29.

Strong

(2006). Integrative motivation: Cause or result of successful second language acquisition? Language Learning, 34(3), 1–13.

30.

Shohamy

(1998). Critical language testing and beyond. Studies in Educational Evaluation, 24(4), 331–345.

31.

UNSW Global. (2008). PEAT– Professional English Assessment for Teachers. In UNSW Institute of Languages. Retrieved April 19 2010 from: www.languages.unsw.edu.au/testing/PEAT.html.