The influence of prompt on group oral tests

Abstract

This paper reports an investigation into how the prompt may influence the discourse of group oral tests. The group oral test, in which three or four participants are rated on their ability to discuss a prompt, is a format for assessing the spoken ability of language learners. In this study, 141 Japanese university students were videoed in 41 group orals of three or four test-takers. Although the four different prompts written for the test were supposed to be of equal difficulty, they were found to be substantially different in the type and number of questions that comprised them. Analysis of the transcribed interactions revealed significant differences in turns taken, syntactic complexity and fluency of the interactions they elicited. A qualitative examination revealed that the two prompts that elicited longer, more complex turns did so by encouraging test-takers to explain their family circumstances or speculate about their future. Prompts with more factual content elicited shorter, less complex turns and the prompt that test-takers responded to with the least fluency required the test-takers to talk about a more personal subject. The implications for rating and creating prompts are discussed, and the need to tailor them to the purpose of the test.

Keywords

Accuracy complexity fluency group oral test interaction speaking assessment test prompt turn-taking

In the last couple of decades, the direct assessment of spoken language has seen a shift in interest towards tests in which test-takers interact with each other, rather than with an interviewer (see Johnson, 2001, for a review). This shift signifies a move from conceiving of speaking ability as represented by the linguistic features of an individual’s spoken words to one of interactive communication. Various aspects have been researched, including the co-constructed nature of the discourse (Brooks, 2009; Galaczi, 2008; He & Dai, 2006; Swain, 2001), how different ability levels interact (Gan 2008, 2010; Norton, 2005), personality factors (Berry, 2004; Nakatsuhara, 2011; Ockey, 2009), number of participants (Nakatsuhara, 2011) and task type (Van Moere, 2007). While the impact of task has been mentioned in some of these studies to a greater or lesser degree, none have systematically investigated whether different forms of even the same task can result in a significantly different discourse. The lack of literature on tasks might be thought surprising when it is considered that without the presence of an interviewer who may direct the discourse towards the targeted language, the task may assume primary importance in shaping the ensuing interaction.

This article goes some way to filling this niche by researching the role of the prompt in influencing different patterns of interaction during the administration of a large-scale group discussion test. The data comes from a year-end proficiency test at a Japanese university, for which four different prompts were written with the intention of their being of equal difficulty. However, it was noticed anecdotally that the prompts tended to elicit very different responses despite there being no significant difference in their scores; and this research was instigated to investigate this observation empirically. As such, this study is of direct relevance to speaking test developers, who can use the findings to construct tasks more likely to suit their purposes.

The effect of task on oral discourse

In this study, the three main facets affecting the oral discourse of these group oral tests were considered to be test-taker facets, task facets, and contextual factors. Test-taker facets include such factors as their level of language proficiency, gender, background knowledge, and personality factors; task facets may include the questions in the prompt, degree of imposition, task structure and instruction, and number of participants in a group; and contextual factors may include issues of power relations and social distance.

A large body of research conducted on the influence of task has come from investigations into how varying a task’s facets or parameters affects the complexity, accuracy, and fluency of the language elicited (Ellis, 2009; Robinson, 2011; Skehan, 2009). A principal tenet is the generally accepted theory of learning that attention is a limited resource and if a person has more than one element to attend to, cognitive resources have to be prioritized. Newly learnt skills require more attention, but over time are increasingly automated, freeing up attention for other tasks (Schneider & Shiffrin, 1977). More difficult tasks for language learners require more attention, and the competition for resources affects the complexity, accuracy, and fluency of oral production according to the various parameters of the task (Skehan, 2009). Research has shown that (a) more familiar tasks with a clear structure, such as personal information exchange tasks, aid accuracy and fluency over complexity; (b) interactive tasks result in more accuracy and complexity; (c) information manipulation tasks produce more complexity; (d) narrative tasks are related to higher complexity but lower accuracy and fluency; and (e) public performance may reinforce accuracy (Skehan, 2009, p. 511).

Much of this research has focused on non-interactive monologue tasks such as narratives based on pictures (see Ellis, 2009, for a review). Amongst studies that included interaction between participants, Taguchi (2007) investigated the impact on fluency and appropriateness in situations with a high or low degree of difference in power, social distance, and imposition. Her lower-proficiency language users found it more difficult to produce appropriate language and were significantly less fluent on tasks with a high degree of difference in status. Robinson used direction-giving (2001) and narrative tasks (2007) with paired subjects and found that tasks that were more complex in terms of reduced planning time, unfamiliarity, inclusion of more materials, and causality in the narrative required significantly more interaction to complete than simpler ones. More complex tasks generated more clarification-checking questions and a corresponding increase in lexical variety, but were accomplished with less fluency. Another study that used dyads with L2 Dutch speakers also found that more complex comparison tasks lead to less fluency. The participants with more challenging tasks were found to show improved accuracy but with only a limited effect on the complexity of their language (Michel, Kuiken, & Vedder, 2007).

Other relevant literature on the task facet comes from studies on how teachers use questions to set tasks and elicit. Questions can be dichotomized into open and closed. Open questions have no single answer while closed questions have a restricted range of answers, such as yes/no and “or” questions. Simultaneously, questions may be either interpretive, eliciting reasoning of some kind, or factual, eliciting recall. Studies in the field of preschool education and science teaching have found that open and interpretive questions elicited a wider variety of vocabulary expressed in sentences that were more complex (Lee & Kinzie, 2012) and this has been linked to significantly improved language development (Wasik, Bond & Hindman, 2006). In contrast, closed questions in language classes have been found to elicit short and syntactically simple responses (Yang, 2010), although it has been noted that they can elicit expanded responses and, if so, they can be retrospectively categorized as “half-open” (Hargreaves, 1984).

Although this research on tasks is relevant to test developers, in assessment it is not just the elicitation of language that needs to be considered, but how it is scored. Test scores are often insensitive to task variation (Fulcher & Márquez Reiter, 2003). Moreover, if the scoring bands do not reflect pertinent features of discourse, or the raters are not trained sufficiently, even if the language elicited by different tasks varies, it may go unnoticed. A study by Brooks (2009) provides an example of rating bands that obscured the range of language elicited by a peer-interaction speaking test. Brooks compared scores and analyzed the discourse of students who took both paired and interview tests and found that although students scored significantly higher in the paired than the interview test, the richness of their interactions was not reflected in the holistic scoring band used in the study, leading to construct-underrepresented scores. Even students who scored lower in the paired test than the interview used a greater range of interactive functions when talking to their peers (Brooks, 2009). This shows that to ascertain the impact of the task facet it is necessary to analyze the discourse it elicits. Once the impact of such task facets as the prompts on test-takers’ oral production are empirically investigated, then it is possible to translate discourse features into rating descriptors, leading to a higher degree of construct validity.

In the assessment literature, only a few studies have examined the discourse produced in peer-interaction speaking tests. One that did was Gan (2010), who used Conversation Analysis (CA) to compare a high-scoring group oral test with a low one, focusing on one of the test-taker facets, language proficiency. Both groups had four test-takers whose task was to discuss a question related to a movie they had seen. Gan found that the higher-level group engaged collaboratively, while the lower-level group approached it more as an exercise in which they had questions to answer. Gan noted that, besides the test-takers’ level of language proficiency, a task facet feature was present which might have affected the produced discourse: the higher-level group had a single question, whereas the lower-level group was given three supplementary questions. Gan speculated that this may have contributed to the lower-level group’s orientation to task rather than conversation (p. 598).

This echoes an earlier study by He and Dai (2006) of 48 group orals in the College English Test–Spoken English Test (CET-SET) that revealed a low number of candidates using the negotiating functions that the test was supposed to elicit. Like Gan (2010), He and Dai suggest this may be a result of candidates focusing on producing a response in an assessment context rather than collaborating in the discussion as expected.

Another reason He and Dai (2006) give for the paucity of negotiating functions is the lack of interest induced by the task prompt and questions. They surveyed 196 test-takers about the prompts, and 60.2% of the candidates responded that the prompts were uninteresting or dull. Whether this actually affected their ability to use negotiating functions is debatable, but it should be pointed out that like the higher-level group in Gan’s (2010) paper, the prompts in He and Dai (2006) were single questions (“Is it desirable to live in a big city?”). This weakens an inference that could be made from Gan (2010) that a single question prompt may lead to a more collaborative discussion, rather than, for example, the attitude or confidence of the interlocutors.

In contrast to He and Dai’s (2006) participants, Van Moere’s (2006) test-takers had a more positive impression of the multiple question prompts in their group orals. In his survey, 80% of 1088 students thought the multiple question prompts were “effective for making people talk” (Van Moere, 2006, p. 439). While this has no bearing on the quality of the interaction, at least it shows some face validity for the multiple question format of prompt amongst those test-takers.

Finally, in illuminating research conducted by Van Moere (2007), 63 test-takers performed two or three different tasks: a group discussion of a prompt, a consensus-reaching task, and a picture difference task. The group orals were analyzed by interactional function, words and turns, and then by Conversation Analysis (CA). Van Moere found that the discussion task tended to elicit fewer longer turns; the picture tasks a greater number of shorter turns, with the consensus task in between. The interactional functions also varied, with the discussion requiring the most “asking for and providing information”, the picture the most “negotiation of meaning”, and the consensus task a wider variety of the functions (Van Moere, 2007, p. 336). However, only one discussion prompt was used, and it is this paper’s contention that different prompts can elicit significantly different patterns of interaction.

In the literature on direct speaking tests, the influence of the task has been neglected. Although the studies reviewed above have mentioned it in various ways, they do not consider that different prompts of the same discussion task may have a significantly different impact on the interaction of a group oral test. This study will do so by investigating the prompts used in the administration of a group oral test, both in terms of how the prompts themselves differed, and then by the language they elicited. The discourse of group oral tests is investigated by examining the length of turns in words and time and then by the complexity, accuracy, and fluency of the test-takers’ language according to the prompt they responded to. These measures allow the elicited language. Following this will be a discussion in which examples from the transcripts of the group oral discussions will highlight the findings of the quantitative analysis.

For this study, the research questions are as follows:

Do the prompts used in this administration, despite being created with the intention of being of equal difficulty, differ in terms of how open/half-open/closed and factual/interpretive they are?

In an administration of a group discussion proficiency test for which four different prompts were created with the intention of being equivalent in difficulty:

Can the prompts elicit significantly different patterns of responses, as indicated by the number of words spoken, turns taken, or length of turn in words or time?

Can the prompts elicit significantly different levels of syntactic complexity, accuracy, and fluency?

Method

Participants

The data comes from 141 students who were videoed in 41 group oral tests that took place at the end of their first year at university. The students were all 19–20 years old, mostly female (81%) Japanese majoring in English, with a generally uniform educational background. They had all studied English for at least six years at middle and high schools, and had one year of intensive study at university in which they were exposed to 15 hours per week of English-only classes.

The administration of the group oral test

At the university where this study was conducted, the test-takers sit for a reading, listening, grammar and writing test in the morning, and the group oral test in the afternoon. The test contributes 20% towards their grade for core classes and is used to place students into classes. Additionally, the administration uses it to monitor student progress. The group oral test at this institution has attracted considerable attention; for example, Bonk and Ockey (2003), Ockey (2009), and Van Moere (2006, 2007) collected data here.

When the morning session ends, students are reminded of the afternoon’s oral test. Out of consideration of fairness, students are randomly assigned to a group with others who are not their classmates and assigned two raters who are not their teachers. They do not know the room, time or who they will be tested with until the afternoon of the oral test. Thus it can be assumed that with respect to test-taker characteristics, such as personality factors and acquaintanceship, test-takers are randomly distributed in the groups, and their impact may affect groups equally. Table 1 shows the distribution of students and group oral tests by the four prompts in the tests videoed for this study.

Table 1.

Distribution of test-takers and group orals by prompt.

	Test-takers	Female – Male	No. of group orals
Mobile	35	32 – 3	10
Outdoors	27	20 – 7	8
Singles	29	21 – 8	9
Family	50	41 – 9	14
Total	141	114 – 27	41

The university’s native English-speaking staff who rate this test are trained in a two-hour session. For each test, the raters choose one of the four prompts at random and allow a minute for the test-takers to read it and prepare before telling them to start the discussion. The students discuss the prompt without interruption for up to 10 minutes, but if the raters have sufficient information to assign a grade they can end the test earlier. The average length of the oral tests used in the current study was 7.02 minutes. The rating scale can be found in the Appendix.

For security reasons four prompts were created for the administration of the test, and they were not piloted before the test. The prompts were created according to the guidelines developed from previous administrations: they should be on a topic similar to those used in their classes, be of equal difficulty, and have enough content for the discussion to last 10 minutes. They should all include an initial context-setting statement followed by a series of questions ranging from the concrete to more abstract, one of which requires test-takers to enumerate advantages and disadvantages of the topic. To ensure participants’ understanding, a Japanese translation is provided. The prompts used in the current study were on mobile phones, outdoor vs. indoor activities, single vs. married life and traditional families (hereafter referred to as “Mobile”, “Outdoors”, “Singles”, and “Family” (see Table 3).

Equality of group composition

The first step was to ensure that the students who took each prompt, as well as the students within each group, had an equivalent incoming level of proficiency. The morning test, which included 141 reading, grammar, and listening multiple-choice items and was taken by 783 students, including this study’s participants, was used for this purpose. A Rasch analysis of the test using Winsteps (Linacre, 2010) produced acceptably high reliability estimates of 0.90 for the students, and 0.99 for the items. A one-way Analysis of Variance (ANOVA) of the 141 individuals’ Rasch-adjusted scores revealed no significance difference when grouped by prompt (F = 0.893, df = 3, p = 0.446), nor were significant figures found in the average scores of each prompt’s group (F = 0.985, df = 3, p = .410), implying that the participants in the four prompt groups and the composition of groups had the same English proficiency level. To rule out the possibility of females and males having different levels of ability (Berry, 2004), students were grouped by gender, but no significance was found (F = 0.336, df = 1, p = .563). Another ANOVA revealed no significant differences for the student’s oral scores, either in the five bands or in the total (see Table 2).

Table 2.

Descriptive statistics of average group scores by prompt and ANOVA results.

	No.		Pronunciation	Fluency	Vocabulary	Grammar	Complexity
Mobile	10	$\bar{x}$	2.71	2.73	2.68	2.66	2.85
Mobile	10	SD	0.26	0.22	0.20	0.27	0.34
Outdoors	8	$\bar{x}$	2.71	2.79	2.60	2.69	2.85
Outdoors	8	SD	0.33	0.28	0.10	0.24	0.46
Singles	9	$\bar{x}$	2.57	2.51	2.45	2.44	2.63
Singles	9	SD	0.18	0.27	0.22	0.21	0.28
Family	14	$\bar{x}$	2.55	2.59	2.51	2.51	2.61
Family	14	SD	0.26	0.33	0.34	0.28	0.35
df			3	3	3	3	3
F			1.198	1.849	1.587	2.003	1.437
Sig.			0.324	0.155	0.209	0.130	0.247

Video transcription and coding

The group oral tests were video-recorded and transcribed by native English-speaking postgraduate students who were trained to record every utterance and ascribe it to the correct person. Every transcript was then rechecked by one of the authors and coded into AS-units (Foster, Tonkyn, & Wigglesworth, 2000), which have been used in other task-based studies (e.g., Foster & Tavakoli, 2009). The AS-units were coded by a trained graduate student and checked by one of the authors. Finally, for inter-coder reliability an experienced MA qualified colleague who was trained in the AS-Unit, clause, and error-free clause procedures checked an entire script that had been chosen at random. Percent agreement per line of transcript was calculated to be at an acceptable 88.4%. The divergent codings were discussed and the resolution resulted in changes being made to the coding system, and the transcripts corrected.

Analysis of turns, words, and time

In order to answer research question 2a, the words and turns in the transcripts were counted. Non-words, such as “ums”, “ahs”, incomplete and immediately repeated words, were omitted (Lennon, 1990; Taguchi, 2007). A turn was defined as an utterance that was responded to, or a coherent sequence of words that could have been responded to (Eggins & Slade, 1997).

The length of the turn can be considered not only in terms of words, but also in time, and so all turns over 10 seconds were timed, as well as all the silences within these turns, using the program Audacity (Mazzoni & Dannenberg, 2010). The period of 10 seconds was chosen to eliminate minimal responses from this data. Only one test-taker did not have a single turn over 10 seconds, and was thus considered an outlier and not included in the analysis of this part of the study.

Time is also important because, as noted above, raters can end the test when they have sufficient data to give a rating. Since longer tests potentially give participants more opportunity to speak, the data was normalized by dividing all counts by test time. This was deemed the best method of normalizing, since it captured all of the language elicited without loss of data – an important consideration given that some test-takers talked more towards the end of the test than at the start, and vice versa.

Analysis of syntactic complexity, accuracy, and fluency

Fluency was measured by the rate of speech and quantity of unfilled pauses, which have been found to be significant markers of fluency (Lennon, 1990; Riggenbach, 1991; Towell, Hawkins, & Bazergui, 1996). For the “speech rate” index all understandable English syllables, including repeated words and false starts were counted, while non-lexical fillers, such as “um” and “er”, and Japanese interjections were excluded. This figure was divided by the turn’s time and multiplied by 60 to arrive at the rate per minute (Towell et al., 1996). Unfilled pauses of one second or more within a long turn were timed, and this figure was divided by turn time to give a “pause proportion” index, which is a measure of breakdown fluency (Tavakoli & Foster, 2008). The software Audacity (Mazzoni & Dannenberg, 2010) was used to measure pauses, since this allowed precise calculation of lengths of silence from a waveform.

Additionally, a measure of repair fluency (Tavakoli & Foster, 2008) was given by the proportion of repetitions, false starts, and self-corrections not necessary for communication, or “maze words” (Loban, 1976), to total words spoken.

Syntactic complexity was measured by the ratio of clauses to AS-units and the average length of utterance, which was calculated as the number of words per AS-unit (Foster & Tavakoli, 2009). Finally, the measurement of accuracy was given by the ratio of error-free clauses to total clauses.

Non-parametric statistical methods were used because the data was composed of frequencies of occurrence, and neither a normal distribution nor homogeneity of variance could be assumed. Unequal variances were confirmed visually in box plots (Larson-Hall, 2010) and by Levene’s test. Kruskal-Wallis, the non-parametric test of significance equivalent to the Analysis of Variance (ANOVA), was therefore used. Effect size was calculated following Burns’ (2000, p. 330) calculation for η² . The Mann-Whitney U test was used as a post-hoc test to identify the statistically significant relationships in the data (Field, 2005).

Results

Research question 1: Analysis of the prompts

The questions were analyzed according to the dimensions of open/closed and interpretive/factual, and the results are presented in Table 3.

Table 3.

Analysis of questions in the prompts.

Mobile prompt*Context-setting statement: These days, lots of people have mobile phones and they are becoming very important in our lives.

Do you have a mobile phone? Why?	Half-open – factual + Interpretive
Do you often use it?	Closed – factual
How do you feel about mobile phones?	Open – interpretive
What are some good points and bad points about them? Why?	Open – interpretive
Could you live without your mobile phone?	Closed – interpretive

Outdoors prompt*Context-setting statement: More and more young people are spending their free time inside the house watching TV, using the Internet and playing computer games.

Do you like to spend your free time inside or do you prefer to do outdoor activities?	Closed – interpretive
Should we all try to spend some time doing outdoors activities? Why or why not?	Half-open – interpretive
Is it healthy to spend all our time indoors?	Closed – interpretive

Singles prompt*Context-setting statement: Around the world, people are marrying later and later.

What are the advantages of being single? Why?	Open – interpretive
What are the advantages of being married? Why?	Open – interpretive
Do you want to get married or would you prefer to be single?	Closed – interpretive
In the future, do you think people will still want to get married?	Closed – interpretive

Family prompt*Context-setting statement: In the traditional Japanese family, men earned the money and women did the housework.

Is your family traditional or not? Why do you think so?	Half-open –interpretive
What are the advantages and disadvantages of the traditional family?	Open – interpretive
Do you think the situation in Japan is changing? Why?	Half-open –interpretive

When presented to test-takers these prompts appeared in paragraph form with a Japanese translation.

It can be seen that the Outdoors, Singles, and Family prompts are composed entirely of interpretive questions. The only two factual questions both appear in the Mobile prompt. Immediately following one of these is an interpretive “why”, encouraging the test-taker to expand their reply (Hargreaves, 1984).

The Mobile, Singles, and Family prompts include both open and closed questions, but the questions in the Outdoors prompt are either half-open or closed, which suggests that they might not elicit such complex responses (Yang, 2010).

It is difficult to predict what impact the content of the prompts would have on the language elicited. Topics that are more familiar may aid accuracy and fluency over complexity (Skehan, 2009); and the frequent use of personal pronouns “you”, “we”, and “our” in most of the questions in each prompt might encourage personal information to be exchanged. It is unknown to what extent students are familiar with discussing this kind of personal information, although it might be anticipated that the Singles topic would push test-takers to disclose personal information that they may find face threatening, and this could affect their fluency (Taguchi, 2007).

The differences among the prompts found by this analysis raise serious questions about their supposed equivalence of difficulty. Had such an analysis been done before the administration of the test, editing might have been considered necessary to make them more parallel in form and content. However, previous research had shown that a similar approach to the creation of speaking prompts did not impact significantly on scores (Bonk & Ockey, 2003) and owing to time constraints and their general conformity to the guidelines, the prompts received approval, and the administration of the test went ahead.

Research questions 2a & b: Analysis of the discourse

A general picture of the language elicited by the prompts may be gained from the descriptive statistics presented in Table 4. The results of the Kruskal-Wallis tests of statistical significance are shown in Table 5, and Table 6 provides the Mann-Whitney U tests of the significant findings.

Table 4.

Descriptive statistics by prompt.

		Words		Turns		Words/Turn		AS-Units		Complexity		Fluency			Accuracy
		1. Words/Test-taker	2. Time/Word	3. Turns/Test-taker	4. Time/Turn	5. Words/Turn	6.Time/Words/Turn	7. AS-Unit/Time	8. AS-Units/Turn	9. Clause/AS-Unit	10. Words/AS-Unit	11. Maze Ratio	12. Speech Rate	13. Pause Proportion	14. Error-Free Ratio
Mobile	$\bar{x}$	147.23	3.48	14.34	35.00	10.74	43.50	5.69	1.75	1.27	5.57	0.11	99.39	0.34	0.62
Mobile	SD	70.55	1.78	7.07	13.98	3.54	15.18	2.89	0.32	0.15	1.05	0.09	26.4	0.12	0.14
Outdoors	$\bar{x}$	141.68	3.37	12.18	56.9	19.88	37.43	6.29	2.56	1.37	5.92	0.10	104.95	0.36	0.65
Outdoors	SD	81.05	1.47	7.78	54.37	25.87	24.26	3.37	1.74	0.28	2.10	0.08	30.34	0.14	0.15
Singles	$\bar{x}$	141	3.49	10.38	56.12	16.63	32.51	8.15	2.13	1.57	6.66	0.14	92.44	0.46	0.63
Singles	SD	64.81	1.26	6.43	33.73	9.55	17.90	3.10	0.613	0.22	0.99	0.094	24.48	0.12	0.08
Family	$\bar{x}$	145.63	4.20	8.10	85.26	22.10	28.30	8.09	2.83	1.50	6.83	0.15	92.53	0.35	0.56
Family	SD	76.02	2.89	4.87	85.34	12.39	18.14	4.37	1.21	0.22	1.77	0.09	23.42	0.13	0.15

Table 5.

Kruskal-Wallis test of statistical significance (degrees of freedom = 3).

	Words		Turns		Words/Turn		AS-Units		Complexity		Fluency			Accuracy
	Words/Test-taker	Time/Word	Turns/Test-taker	Time/Turn	Words/Turn	Time/Words/Turn	AS-Unit/Time	AS-Units/Turn	Clause/AS-Unit	Words/AS-Unit	Maze Ratio	Speech Rate	Pause Proportion	Error-Free Ratio
X ²	0.57	1.021	20.555	24.203	26.151	20.285	16.674	28.062	34.517	20.632	9.872	6.125	16.041	6.874
η²	0.004	0.007	0.147	0.192	0.187	0.145	0.119	0.200	0.247	0.147	0.071	0.044	0.115	0.049
p	.903	.796	.000***	.000***	.000***	.000***	.001***	.000***	.000***	.000***	.02*	.106	.001***	.076

Table 6.

Mann-Whitney U Tests of the significant results.

Prompt	Turns		Words/Turn		AS-Units		Complexity		Fluency
	Turns/Test-taker	Time/Turn	Words/Turn	Time/Words/Turn	AS-Unit/Time	AS-Units/Turn	Clause/AS-Unit	Words/AS-Unit	Maze Ratio	Pause Proportion
Mobile vs. Outdoors	0.236	0.793	0.468	0.081	0.709	0.065	0.489	0.874	0.579	0.347
Mobile vs. Singles	0.005**	0.002**	0.001***	0.001***	0.001***	0.013**	0.000***	0.000***	0.085	0.000***
Mobile vs. Family	0.000***	0.000***	0.000***	0.000***	0.001***	0.000***	0.000***	0.000***	0.028*	0.396
Outdoors vs. Singles	0.424	0.185	0.213	0.482	0.022*	0.892	0.001***	0.023*	0.034*	0.006**
Outdoors vs. Family	0.027*	0.005*	0.001**	0.081	0.036*	0.019**	0.013**	0.017*	0.01**	0.816
Singles vs. Family	0.105	0.022*	0.038*	0.129	0.525	0.003	0.453	0.914	0.744	0.001***

Significant at p < .05; ** significant at p < .01; *** significant at p < .001.

An important finding for the administrators of this test can be seen in columns 1 and 2 of Table 4. These show that words spoken per test-taker were not significantly different whether adjusted for test time or not, indicating that test-takers were not discriminated against by the varying lengths of the tests.

The answer to research question 2a is indicated by the statistics related to the number of turns (columns 3–6), which show that test-takers responded to the prompts in significantly different ways. The statistics for “time per turn” in Table 4 are reported as the average frequency of occurrence in seconds. Thus, a test-taker responding to the Mobile prompt took a turn every 35 seconds on average, while those responding to the Family and Singles prompts had on average 85.26 and 56.12 seconds respectively between taking turns. The Kruskal-Wallis test showed that this statistic, along with the other turn-related ones were significantly different (p < .001), with small effect sizes of between 0.145 and 0.192. The Mann-Whitney U tests showed that these differences came from the Mobile prompt eliciting shorter turns more frequently than the Singles and Family prompts. Also, test-takers with the Family prompt took turns significantly less frequently and longer than those with the Singles and Outdoors prompts.

The figures for AS-Units returned significant findings per time and per turn with small and moderate effect sizes respectively, and the Mann-Whitney U tests are consistent with previous findings. Responses to the Mobile prompt again had significantly less and more frequent AS-Units than the Singles or Family prompts. The Outdoors prompt also had significantly more frequent AS-Units than did the Singles prompt, and fewer AS-Units per turn than the Family prompt. Research question 2a can be answered positively.

The results of the Kruskal-Wallis indicate a positive answer to research question 2b for complexity and fluency. The complexity figures of words and clauses per AS-Unit are significant (both p < .001), with small and moderate effect sizes respectively. For fluency, the maze ratio and pause proportion show significant differences (p < .05 andp < .01 respectively) with small effect sizes.

The Mann-Whitney U tests showed that the complexity of responses to the Mobile prompt were significantly less than for the Singles and Family prompts, as seen in the clauses and words per AS-Unit. The Outdoors prompt also elicited significantly less complexity than the Singles or Family prompts, with lower ratios of clause and word to AS-Unit. The fluency figures resulted from the Singles prompt eliciting a significantly higher proportion of unfilled pauses than all other prompts, as well as having a significantly higher proportion of maze words than the Outdoors prompt. The Family prompt had a significantly higher proportion of maze words than the Mobile or Outdoors prompts.

Finally, no significant differences were found between the prompts in the measure for accuracy, which is an interesting result when considered in conjunction with the rating band for grammar, which includes aspects of both complexity and accuracy. The implications of this are in the discussion section of the paper.

Discussion

Mobile prompt

As we have seen, groups with the Mobile prompt consistently had more turns which used fewer words and less time than other prompts. Extract 1 is an example of a group oral which shows this clearly.

Extract 1. Group oral 10-10. Turn Person 5 C: Do you have a mobile phone? 6 A: Yes. 7 B: Yes. 8 D: Yes of course [A:B:C:D: huh huh huh]. 9 C: Me too. 10 A: How do you feel about mobile phone? 11 D: It’s very useful. 12 C: Yes, convenient. 13 D: When did you start to have a mobile phone? 14 A: When I entered high school. 15 B: Yeah me too. 16 C: Yes me too. 17 D: Me too. 18 A: Maybe I can’t live without the mobile phone [B:ah yes]. 19 D: I send email to my friends every day. 20 A: Do you think what is a good point? 21 B: I think mobile phone is good because {mobile phone has…} mobile phone is easy to communicate with other people. How about you?

Key:

Underline = simultaneous talk

[] = backchannel

{} = maze words

Galaczi (2008) used the innovative approach of operationalizing a topic of conversation as “a spate of talk that referred to a specific visual” (p. 96), since the prompts in her study used images. This analysis will follow her by defining a sufficient but not necessary condition of “topic” as a sequence of talk that refers to a specific question from the prompt. Within the first 20 turns of Extract 1 the participants use four topics (at turns 5, 10, 18, and 20) directly from the prompt. That the Mobile prompt questions were answered so tersely precipitates the use of the prompt to find something else to talk about, leading to greater dependence on it. Since participants may be reluctant to return to questions that have already been “answered” they are, in effect, consuming the scarce resource of questions in the prompt. This pattern of discourse looks similar to the behavior of the lower-scoring students in Gan (2010), who seemed to consider the prompt as a set of questions to be answered rather than as the basis of a discussion.

As noted in the analysis of the questions, the Mobile prompt was the only one that included factual questions, and, as in extract 1, these were typically answered briefly. Test-takers usually developed the discussion by asking further questions about cellphone features, price, and usage, which could be answered just as briefly. This contrasts with responses to the Family and Single prompts, in which the test-takers could produce longer, more complex turns by drawing on their personal background or by speculating about the future. This is underlined by the number of turns over 10 seconds not being significantly different between the prompts (X² = 2.039, p = .564), showing that participants of the Family and Single prompts usually had more to say about the content of their prompt.

Outdoors prompt

The Outdoors prompt elicited the widest range of responses, as shown by the highest standard deviations for words per turn and AS-Units per turn amongst others. As noted in the analysis of the prompts, the questions are either half-open or closed; for the test-takers this made their current activities the most obvious thing to discuss. This was usually done using short turns which were, like the responses to the Mobile prompt, on the here-and-now of their lives, discussing what activities they are most frequently engaged in. This resulted in significantly less complexity in their language than in responses to the Singles or Family prompts.

Extract 2 is taken from an Outdoors prompt group oral which had similar overall words per turn statistics to the response to the Mobile prompt in extract 1, making it the lowest with respect to these figures among responses to the Outdoors prompt.

Extract 2. Group oral 07-5. Turn Person 12 D: What do you usually do? 13 C: um I like to read books. [D:nn] so {if I go to} if I go out I go to the library. 14 A: What kind of book do you read? 15 C: um I like um mystery. 16 A: Mystery. 17 C: Yeah, um {wha-} what do you do? Ah outside? 18 D: Outside? Um I like just walking around. Yeah and go shopping and something. What about you Kasumi? 19 B: hhhhhum {I} I just sleeping. [C:sleeping A:D:huhuhuh] in my house yeah. Ah {to} to go outside too tired for me. So I spend my time {insi-}uh inside. {What do} what do you do? 20 A: un if I go outside I go to Starbucks [C:Starbucks] and drink coffee. 21 D: It’s kind of inside. 22 A: Yeah. 23 C: I think so [B:C:D:huhuhuhuh]. 24 A: And drink coffee and watching people. 25 C: Do you like coffee? 26 A: Yeah. 27 C: Oh ah I’m working at Dotour coffee shop [A:mmm B:oh].

In contrast to the Mobile group oral test in extract 1, the test-takers seem oriented to having a conversation. Unlike extract 1, where the questions were taken directly from the prompt, in turns 14 and 25 the participants ask follow-up questions in response to the previous speaker, and turn 21 also responds directly to what was previously said. Turn 27 provides cohesion by giving relevant information that person A, as a coffee drinker, might be interested in. Along with this conversational involvement with each other, the influence of the prompt can be seen nakedly by the qualification “Ah outside?” in turn 17 and especially in turn 21, in which D passes judgment on A’s comment in terms of the dichotomy presented in the prompt. Nonetheless, if authenticity in testing is accepted as the “degree of correspondence” (Bachman & Palmer, 1996, p. 23) between real world activities and test activities, this exchange can be considered as a satisfactory simulation, and suggests that the phenomenon of non-conversation-like behavior in response to multiple question prompts (Gan, 2010), may be owing to other factors, such as attitude or confidence, as much as the number of questions. It is interesting to note that general language proficiency, as recorded by the participants’ scores on the morning’s test, does not seem to be a factor here, with those in the groups in extracts 1 and 2 having almost the same average score.

The Outdoors prompt also had a significantly lower maze ratio in its long turns than either the Singles or Family prompts, which is consistent with research that has found that familiar topics elicit more fluent responses (Skehan, 2009), but may also be related to the shorter average turn length. An example of a typical Outdoors prompt discussion is shown in extract 3.

Extract 3. Group oral 10-03. Turn Person 27 A: Do you want to play soccer? 28 D: Yes, but I have no place [C:oh] {and} but I have part time job. 29 C: You have place, here [B: yeah] [D: ahh]. 30 A: You should enter the football circle [D: circle]. 31 D: But I have part time job. [A:B:C: ahhh] So I can’t manage{to} to play [C: ah]. 32 B: Um. Do you like TV games? 33 C: Ahhh, un.. ah I don’t like TV games very much [B: unn]. Yes, because it’s too much difficult to win [A:B:C:D huhuhuh]. And you? 33 B: Ah I want to play games but I don’t have. So my parents don’t like it. So ah. {in} in my.. younghood they didn’t gave me [C mm D: oh]. How about? 34 A: {I I} um I dislike TV game, but my brother likes TV game and he borrowed a TV game. {So} but I hate TV game, so I didn’t do TV game [B: hmm].

Of the participants here, person A had statistics that were near the median for maze ratio, D’s figures were in the quartile above the median, while B and C had figures that were among the fewest. The content of the talk here is similar to extract 2 in that it is mostly about their current lives, likes, and dislikes, though B also expands briefly into her personal history in turn 33. Examining the use of vocabulary, the participants’ relative ease in finding appropriate words to express themselves can be seen by the lack of pausing, though B forces herself to circumlocute with “younghood” in turn 33. All participants express themselves mostly by using coordinating conjunctions to string together their clauses, resulting in lower complexity figures. The participants certainly seem to find it less challenging to talk about this prompt compared with test-takers with the Singles or Family prompts, as seen in extracts 5–7.

It is instructive to examine some extreme cases among test-takers with this prompt. Among the 27 students in eight groups discussing the Outdoors prompt, there were four students with exceptionally longer average words per turn statistics, three of whom were in one group. Part of their group oral is shown in extract 4.

Extract 4. Group oral 09-10. Turn Person 4 B: I like outdoors because {when I was a jun-} when I was a high school student I belonged to fencing club and every Saturday and Sunday I practiced fencing {so um… if I have a …free time …I free time}. But I {ca} don’t like to spend my free time in the house so I want to active my free time um. Yes … [A: un] Do you think so too? 5 A: Yes, I agree with you. Actually when I was a junior high school student and high school student I belonged to a handbell club. Do you know handbell? 6 C: Handbell 7 A: Handbell. It’s a kind of {instrument musical} music instrument {and and I didn’t} mm and I couldn’t have opportunities to go outside {but}.. But after graduating {from} from high school student, I started to go to a swimming club, and … un there is a swimming pool outside, and … um {I feel I feel very good and uh under the sun} I feel very good swimming under the sun. So … …{I like} um I feel better…mm .. and I think {there is} it is good for our health {to to be out} to be outside …in my opinion. {Do you think} How do you think?

Instead of focusing on the activities they liked or disliked (as in extract 2), these test-takers expanded their responses by talking about their history of playing sports, allowing them to build a personal narrative into the turn which has similarities to the structure of stories in conversation (Labov & Waletzky, 1967). This can be clearly seen in turns 4 and 5 by speakers B and C. The beginning of the story orients the listener to the time and place, followed by a brief narrative before the resolution which relates it back to the current topic of conversation. That a few individuals could do this shows the possibility of responding in different ways to a prompt. Given this, the consistency of the responses to the prompts in general is quite remarkable.

Singles prompt

Interactions elicited by the Singles prompt tended to follow a similar pattern to the Family prompt by generally having fewer, longer turns, with significantly more complexity than responses to the Outdoors and Mobile prompts.

These tendencies can be seen in extract 5, which shows the AS-Unit analysis of two turns (the context of these turns can be seen in extract 6). Although these test-takers had lower than average clause to AS-Unit figures for responses to the Singles prompt, they show some typical features. Prefacing statements with a qualifying “I think”, reasons given with “because”, and a high incidence of conditional statements as they speculated about their futures, were all commonly found in responses to this prompt. This is in marked contrast to responses to the Mobile and Outdoors prompts that did not require qualifying and reason-giving devices as often, even for the few students who could expand their turns by narrating their experiences.

Extract 5. Group oral 10-06. Turn Person 12 D: I think.. + raising {child} children + is very important thing for life + because…{we} we can learn {many} a lot of things from child../ for example we can make ah many friends {through the}… ah through the our child. / so we need + ah to raise our child.. [A:huhuhuh] 13 B: I want + to get married at 26 years old + because {I I have dream to be..} I have dreams / so if I get married + maybe I can’t work. / and so…{ um I…} I.. want to.. work + after I get married / but it is difficult / so un… {…I get} I want to get married / but un… maybe I will get married later.

Key

/ AS-Unit boundary

+ Clause boundary

{ } Maze Words

Extract 6. Group oral 10-06. Turn Person Turn Length Pause Time 8 D: Do you like child? 7.60 – 9 A: Yesss. 8.51 – 10 C: Yeah I like child but… to grow up child is very 16.46 8.52 ^2.7 difficult.. [A: ah] {so I.. umm…} so what do you ^1.0 ^1.3 ^3.5 think about how to grow child? 11 A: But it’s troublesome for us to bring up child but it is fun. 9.08 – 12 D: I think.. raising {child} children is very important 27.76 7.51 ^1.1 thing for life because… {we} we can learn ^2.2 {many} a lot of things from child.. for example we ^1.2 can make ah many friends {through the}… ah ² through the our child.. So we need ah to raise ¹ our child [A:huhuhuh]. 13 B: I want to get married at 26 years old because {I} 39.67 18.41 I {have dream to be..} I have dreams. So if I get ^1.3 married maybe I can’t work.. and so… um {I…} ^1.5 ^2.7 ^2.3 I.. want to.. work after I get married but it is ^1.7 ^1.7 difficult so un… …{I get} I want to get married ⁶ but un… maybe I will get married later. ^1.2 14 C: Are you Chinese? 1.67 – 15 B: No, I’m Korean in Japan. (C: oooh) 1.82 – 16 C: Do did you grow up in Japan? 2.20 – 17 B: Yes. 0.47 – 18 C: You don’t know {about Korea Korea..} about Korea? 13.54 3.78 ^1.1 .. In Japan {women} women get married later ^1.6 and later.. how about Korea? ^1.1

Responses to the Singles prompt were remarkable for having a significantly higher proportion of pauses than all other prompts. In the transcripts, it appears that the introspection about personal matters required by this prompt was largely responsible for the high rate of pauses for these 19–20-year-olds. It is a topic that they may well talk about less than fluently in their native language. Those who paused the most were usually talking about getting married or having children, perhaps reflecting the finding that more face-threatening situations may negatively affect fluency (Taguchi, 2007).

Extract 6 provides an example that includes longer turns by three of the four participants (B, C and D). Of these, B’s pause proportion figure was close to the median for responses to the Singles prompt (0.50), while C and D were among the most fluent in terms of low pause proportion figures (0.31 and 0.27 respectively). The difference in pause proportions in this extract may be indicative of the problems they had dealing with this topic. Participant C in turn 10, despite having one of the lowest overall pause proportions, shows evidence of struggling to construct her response, using a circumlocution (“grow up child”) and a false start (“so I…”), before surrendering the turn to another participant. The pause proportion of this turn (0.518) is well above her average in the discussion and contrasts with her turn in line 18, which is closer to her norm. Test-taker B’s struggle with the topic can be seen in turn 13 by the frequent short pauses and a longer pause of six seconds as well as abandonment (“I have dreams to be… I have dreams”) and repetitions. In contrast to both of these performances, participant D’s pausing is consistent with her performance in the rest of the discussion. Although D also has disfluencies in her speech, this participant has minimal pauses, and shows her greater ease with the language by her ability to access vocabulary that C could not (“raising a child”).

In the scoring, the students responding to this prompt did get a lower average score in the fluency band (see Table 3), but it was not significantly different to the other prompts. It seems the wording of the bands may have led raters to categorize the fluency of these test-takers as “hesitant… but don’t [sic] impede communication completely” rather than as belonging to the lower band which is described as “long unnatural pauses” making “communication with a [native speaker] difficult”. Given the statistical significance of the pauses, the lack of significant difference in the scores has some interesting ramifications that are discussed toward the end of this section.

Family prompt

The Family prompt elicited the fewest and longest turns in words and time. Test-takers tended to use them for two main topics: to explain who was responsible for earning money and doing housework in their own family to frame the extent they could be considered “traditional”; and to describe the situation of traditional families in Japan. Usually the explanation of their family came in the earlier turns, in the form of a statement followed by question to pass the turn to the next participant. This was in contrast to the Mobile and Singles prompts, which were nearly always started by questions that were taken from the prompt. Extract 7 comes from a test that was middle of the range for words per turn for the Family prompt.

Extract 7. Group oral 10-05. Turn Person 1 C: Um my family is {traditional family}.. Ah not traditional family. Ah both my parent {get} get a job [D:mm]. How about your family? 2 B: My family is traditional type [C:mm]. My father work and ah my mother is doing housework. Um but I think it is {not} not so bad [C:mm]. My father can relax after home [C:mm] and go home and my mother mm cook him dinner so {he} he can relax [ D:C:mm]. 3 A: My family is same as you, traditional family [C:mm] and my father work and my mother do the housework [ C:mm] But usually I um watch the news yesterday, so here was single mother is increasing [ C:mm] and the situation is changing in Japan, I think. [ C:mm] How about you? 4 D: Um my family is traditional Japanese family {and and} ah and I think ah traditional family is ah normal in Japan [C:mm] {because because ah women I think women can’t..} I think men can get high position easier than women [C:mm]. So I think men {can get} can earn {money} more money than women. So I think {Japa- traditional Japa-} traditional Japanese family is the better. {What do which do you think?} Ah what do you think? [ C:aohhh] Traditional or {man and women both} both man and women get the job… is better? [ C:ahm] 5 C: I think {both} both person get the job is good {so…} um so.. recently there are ah a lot of {ta-} ah situation mm. I think it is good.

As was common in responses to this prompt, in the first four turns the test-takers state their general opinion, seemingly at the expense of listening and responding directly to what each other had said. This can be seen in turn 3 above, when A states her opinion on Japan and then asks “How about you?” to D. This question is ambiguous; it could refer to what A had just said about the Japanese situation, it could be asking about D’s opinion generally, or it could just be a signal that the speaker has finished. However, D evidently does not need to clarify this: for him it is the cue to give his opinion in response to the prompt, and since these are the initial moments of the test, this is the opinion that he spent the preparation time ruminating on. By comparison, the question D asks at the end of turn 4 gets answered directly, since everybody has already given their general opinion.

The significantly higher figures for the clauses and words per AS-Unit and Maze ratio in responses to the Family prompt show less fluency and more complexity than the Mobile and Outdoors prompts, which is consistent with other research into language elicited by narrative tasks (Skehan, 2009). The greater complexity was mostly due to the subordination required to give reasons for their position, as for the Singles prompt. Although the levels of complexity in the Singles and Family prompts were similar, the lower ratio of unfilled pauses suggests that the subject of “family” was more familiar to the test-takers.

Implications for scoring the test

In the rating of this test, it might be thought that those who had the Singles and Family prompts could gain higher grades in the grammar band, since responses to these prompts were significantly longer and more complex than the Mobile and Outdoors prompts. At the same time, the significantly shorter, quicker turns in response to the Mobile and Outdoors prompts may have given them more opportunity to show their skill in managing the conversation, and hence a higher score in the communicative skills band. In the scoring of fluency, those responding to the Singles prompt might be thought to be at a disadvantage since they had significantly higher pause proportion figures than test-takers responding to other prompts, and the significantly lower maze ratio figures of those who took the Outdoors prompt suggest they might benefit compared to those with the Family and Singles prompts. However, the test-takers’ scores show that this did not happen. Those taking the Mobile and Outdoors prompts had a similar average score, higher than those who took the Family and Singles prompt, though it was not statistically significant, suggesting that the apparent greater ease of these prompts may have influenced the raters generally rather than in specific bands. While the lack of significance in the scores means that at least the test-takers appear not to have been overly discriminated against by the prompt assigned to them, the negative consequences are that the group oral may be considered a blunt tool for assessing speaking proficiency, and it may encourage self-deceptive beliefs about the efficacy of the test. It is important then to examine this issue from practical and theoretical perspectives.

From a practical perspective, it may be symptomatic of the problem of translating discourse features elicited by oral groups into rating descriptors or rater training on scoring band use. An examination of the bands’ wording is instructive. In the grammar band, it can be seen that the descriptor for each level refers to both grammatical accuracy and complexity. Given that there was no significant difference in the accuracy measure, it leaves open the possibility that the raters were swayed more by grammatical accuracy than how the test-takers were using grammar to create meaning. This implies the necessity to train raters to value what test-takers can do with grammar as much as their accuracy. Further research is necessary to investigate the respective salience of accuracy and the productive use of grammar in learner language, and from this improved descriptors can be created, and training for raters reformed accordingly.

Regarding fluency, it seems the raters were unable to translate the significantly higher proportion of unfilled pauses found in responses to the Singles prompt into significant differences in the test-takers’ scores. Possible reasons are the lack of significant difference in the rate of speech overshadowed the statistics for pausing and the maze ratio, or the raters found it difficult to distinguish the first three grades of the fluency rating band. If this is the case, setting standards for fluency at different grades and raising awareness of these in training could lift the ability of the raters to distinguish performances.

From a theoretical perspective, the construct that the bands are attempting to measure should be considered. The lack of correspondence between the scores and significant measures used in this study may result from the measures used being too sensitive or specialized for the raters to distinguish. Although the original impetus for conducting this research resulted from the authors’ impressions that differences existed in the data, more research is needed to establish what degree of statistical difference is necessary before human judges can reliably notice they are different.

Overall, the findings of this paper suggest short and long term objectives for improving the scoring of the group oral. In the short term, if the rating descriptors functionally represent the features of the oral discourse produced in group oral tests, and if raters’ attention is drawn to the features of interaction raised here, the rating of test-takers’ speaking ability should improve. In the long term, research needs to be conducted on the relative importance of grammatical accuracy and productive use when making meaning, and on establishing the relationship between tests of statistical significance and human perception of fluency.

However, before such programs can be considered, the strongest recommendation this study can make is that prompts that elicit similar interactions must be created, lest students be penalized unfairly due to the prompt that they were assigned.

Limitations of this study

Although this study has found some significant differences, limitations need to be considered. Firstly, the relative uniformity of the participants should be pointed out. The test-takers were 19–20-year-old Japanese university students with similar language learning backgrounds; a less homogenous test-taker body might not have reacted as consistently. Also, the data comes from videos of the test; retrospective interviews that would provide support for the findings could have been conducted had the students been available. Finally, because the data was collected from an actual language test rather than a laboratory setting, it was not possible for the test-takers to take more than one prompt, which would have settled this issue definitively. Nonetheless, it remains a strength of this study that it uses data from a real world setting where the results do make a difference to the participants, something that cannot be guaranteed in laboratory settings.

Conclusion

This study has provided evidence that the prompt may influence the shape of the ensuing discussion. Those students with prompts that asked them to discuss factual or here-and-now topics such as the Mobile and Outdoors prompts had conversations with turns that were usually significantly shorter and less complex. A few test-takers who took the Outdoors prompt could produce longer and more complex turns by using their personal experience to construct a narrative. The Family and Singles prompts required test-takers to explain their family context or speculate on their personal future, which tended to require longer, more complex turns. It seems likely that the hesitancy in response to the Singles prompt was due to the personal topic they were required to explain. While the results were statistically significant, research conducted in more controlled conditions where participants take more than one prompt can can confirm the prevalence of the findings of this investigation.

This study also provides evidence that multiple question prompts need not lead participants to focus more on answering questions at the expense of the discussion. In extract 2 the collaborative exchange that takes place in response to a multiple format question makes it seem more a matter of conversational skill than the number of questions in the prompt, especially when compared with the participants of extract 1. When it is considered that test-takers who work collaboratively can achieve higher scores in rating bands that measure interaction (Brooks, 2009; Galaczi, 2008), it seems to confirm a hierarchy of skills in interaction which requires further investigation. To answer this question longitudinal data would be useful to examine the development of interactive skills over time. Given the impact of various factors on the test-takers’ oral performance as found in studies like Van Moere (2006) and Ockey (2009), such a study could answer the question whether it is possible for test-takers to perform consistently enough to show measurable improvements in the group oral discussion test.

Such questions are important for the viability of this form of test. For now, the findings of this study underline the importance of specifying the construct that is being assessed. If it is desirable to sample the test-takers ability to use extended discourse in a conversational context, then prompts that require explanation of the test-taker’s context, such as the Family prompt, should be used. If interactive skills are thought important, more factual prompts, such as the Mobile or Outdoors prompts, require the skills of nominating new topics, managing turns and keeping the conversation alive. This study also demonstrates the need to be cautious about prompts that lead to talk on personal matters, as it may negatively impact on the test-takers’ fluency.

The group oral test is a convenient way of testing that puts the focus on interaction among peers, giving it the potential to create positive backwash on teaching and learning. However, the extra test-takers add dimensions to the complexity of various influencing aspects, and may threaten its reliability as a proficiency test. It is only by studying the impact of such factors that they can be understood and taken into account when designing test tasks. This investigation into the impact of the prompt has cast some light onto one more factor that affects the interaction of the group oral, and thus has contributed to the literature on this subject.

Footnotes

Appendix

Appendix:

Oral descriptor bands

	PronunciationThink about: pronunciation intonation word blending	FluencyThink about: automatization fillers speaking speed	GrammarThink about: use of morphology complexity of syntax (embedded clauses, parallel structures, connectors)	VocabularyThink about: range of vocabulary	Communicative skills/strategiesThink about: interaction confidence conversational awareness
0.5	Very heavy accent; uses Japanese katakana-like phonology and rhythm; words are not blended together	Fragments of speech that are so halting that conversation is not really possible; NS would think person had virtually no English	Does not use any discernible grammatical morphology	Shows knowledge of only the simplest words and phrases taught in junior high school or beginning high school	Shows no awareness of other speakers; may speak, but not in a conversation-like way
1.0 1.5	Somewhat katakana-like pronunciation; does not blend words together; they are pronounced in isolation	Slow, strained speech; constant groping for words and long unnatural pauses; communication with a NS would be difficult	Doesn’t have enough grammar to express an opinion clearly; makes frequent errors; no attempt at complex grammar	Lexis not adequate for task; cannot express opinion properly with the limited words used	Does not initiate interaction; produces monologue only; shows some turn-taking; may say, “I agree with you,” but not relate ideas in explanation; too nervous to interact effectively
2.0 2.5	May not have mastered some difficult sounds of English, but would be mostly understandable to a naïve NS ; makes some attempts to blend words	Speech is hesitant; some groping for words and unfilled spaces are present but generally don’t impede communication completely	Relies mostly on simple (but appropriate) grammar; has enough morphosyntax to express meaning; complex grammar is attempted but may be inaccurate	Generally has enough lexis for expressing some opinions but does not demonstrate any particular knowledge of vocabulary	Responds to others without long pauses to maintain interaction; shows agreement or disagreement with others’ opinions
3.0 3.5	Pronunciation is good but has still not mastered the sound system of English; accent does not interfere with comprehension; can blend words	May use some fillers; rarely gropes for words but speech may still not be quick	Shows ability to use some complex grammar; may make errors but they are only in late-acquired grammar	Shows some evidence of some advanced vocabulary	Generally confident; responds appropriately to others’ opinions; shows ability to negotiate meaning quickly and relatively naturally
4	Speaks with excellent pronunciation and intonation; has practically mastered the sound system of English	Excellent fluency; uses fillers effectively; shows ability to speak quickly in short bursts	Uses both simple and complex grammar effectively; may make occasional errors but they are only in late-acquired grammar	Shows evidence of a wide range of vocabulary knowledge	Confident and natural; asks others to expand on views; shows how own and others’ ideas are related; interacts smoothly

Acknowledgements

We would like to thank the anonymous LT reviewers for their efforts in pointing out improvements to this article.

Funding

This work was supported by Hankuk University of Foreign Studies Research Fund of 2013.

References

Bachman

L.F.

Palmer

A.S

. (1996). Language testing in practice. Oxford: Oxford University Press.

Berry

(2004). A study of the interaction between individual personality differences and oral performance test facets (Unpublished doctoral dissertation). Kings College, University of London, London.

Bonk

W.J.

Ockey

. (2003). A many-facet Rasch analysis of the second language group oral discussion task. Language Testing, 20, 89–110.

Brooks

(2009). Interacting in pairs in a test of oral proficiency: Co-constructing a better performance. Language Testing, 26(3), 341–366.

Burns

(2000). Introduction to research (4th ed.). Sydney: Longman.

Eggins

Slade

(1997). Analysing casual conversation. London: Cassell.

Ellis

(2009). The differential effects of three types of task planning on the fluency, complexity, and accuracy in L2 oral production. Applied Linguistics, 30(4), 474–509.

Field

(2005). Discovering statistics using SPSS (2nd ed.). London: Sage.

Foster

Tavakoli

(2009). Native speakers and task performance: Comparing effects on complexity, fluency, and lexical diversity. Language Learning, 59(4), 866–896.

10.

Foster

Tonkyn

Wigglesworth

(2000). Measuring spoken language: A unit for all reasons. Applied Linguistics, 21(3), 354–375.

11.

Fulcher

Márquez Reiter

(2003). Task difficulty in speaking tests. Language Testing, 20(3), 321–344.

12.

Galaczi

E. D.

(2008). Peer–peer interaction in a speaking test: The case of the first certificate in English examination. Language Assessment Quarterly, 5(2), 89–119.

13.

Gan

(2010). Interaction in group oral assessment: A case study of higher- and lower-scoring students. Language Testing, 27(4), 585–602.

14.

Gan

(2008). Extroversion and group oral performance: A mixed quantitative and discourse analysis approach. Prospect, 23(3), 24–42.

15.

Hargreaves

D. H.

(1984). Teachers’ questions: Open, closed and half-open. Educational Research, 26(1), 46–51.

16.

Dai

(2006). A corpus-based investigation into the validity of the CET-SET group discussion. Language Testing, 23(3), 370–401.

17.

Johnson

(2001). The art of non-conversation: A reexamination of the validity of the oral proficiency interview. New Haven, CT: Yale University Press.

18.

Labov

Waletzky

(1967). Narrative Analysis: Oral Versions of Personal Experience. In Helm

(Ed.), Essays on the Verbal and Visual Arts. Seattle: University of Washington Press. pp. 12–44.

19.

Larson-Hall

(2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.

20.

Lee

Kinzie

M.B.

(2012). Teacher question and student response with regard to cognition and language use. Instructional Use, 40(6), 857–874.

21.

Lennon

(1990). Investigating fluency in EFL: A quantitative approach. Language Learning, 40(3), 387–417.

22.

Linacre

J. M.

(2010). Winsteps® Rasch measurement computer program. Beaverton, OR.

23.

Loban

(1976). Language development: Kindergarten through grade twelve. NCTE committee on research report no. 18. National Council of Teachers of English, Illinois.

24.

Mazzoni

Dannenberg

(2010). Audacity. A free, cross-platform, audio editor. Version 1.2.6. Retrieved from http://audacity.sourceforge.net/

25.

Michel

M. C.

Kuiken

Vedder

(2007). The influence of complexity in monologic versus dialogic tasks in Dutch L2. IRAL, 45(3), 241–259.

26.

Nakatsuhara

(2011). Effects of test-taker characteristics and the number of participants in group oral tests. Language Testing, 28(4), 483–508.

27.

Norton

(2005). The paired format in the Cambridge speaking tests. ELT Journal, 59(4), 287–297.

28.

Ockey

G. J.

(2009). The effects of group members’ personalities on a test-taker’s L2 group oral discussion test scores. Language Testing, 26(2), 161–186.

29.

Riggenbach

(1991). Toward an understanding of fluency: A microanalysis of nonnative speaker conversations. Discourse Processes, 14(4), 423–441.

30.

Robinson

(2001). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22(1), 27–57.

31.

Robinson

(2007). Task complexity, theory of mind, and intentional reasoning: Effects on L2 speech production, interaction, uptake and perceptions of task difficulty. IRAL – International Review of Applied Linguistics in Language Teaching, 45(3), 193–213.

32.

Robinson

(2011). Task-based language learning: A review of issues. Language Learning, 61, 1–36.

33.

Schneider

Shiffrin

R. M.

(1977). Controlled and automatic human information processing: 1. Detection, search, and attention. Psychological Review, 84, 1–66.

34.

Skehan

(2009). Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 30(4), 510–532.

35.

Swain

(2001). Examining dialogue: Another approach to content specification and to validating inferences drawn from test scores. Language Testing, 18(3), 275–302.

36.

Taguchi

(2007). Task difficulty in oral speech act production. Applied Linguistics, 28(1), 113–135.

37.

Tavakoli

Foster

(2008). Task design and second language performance: The effect of narrative type on learner output. Language Learning, 58(2), 439–473.

38.

Towell

Hawkins

Bazergui

(1996). The development of fluency in advanced learners of French. Applied Linguistics, 17(1), 84–119.

39.

Van Moere

. (2006). Validity evidence in a university group oral test. Language Testing, 23(4), 411–440.

40.

Van Moere

. (2007). Group oral tests: How does the task affect candidate performance and test scores. Doctoral dissertation. Lancaster University, UK.

41.

Wasik

B. A.

Bond

M. A.

Hindman

(2006). The effects of a language and literacy intervention on Head Start children and teachers. Journal of Educational Psychology, 98 (1), 63.

42.

Yang

C. C. R.

(2010). Teacher questions in second language classrooms: An investigation of three case studies. Asian EFL Journal, 12(1), 181–201.