Abstract
This qualitative study reports an investigation of the nature of interactional competence at various levels of achievement in the context of role-play speaking assessment. The focal point of this study is on how examinees jointly accomplish the interactional work involved in proposal sequences in role-play interaction. Based on a conversation analysis of a corpus of role-play interaction, I argue that distinct sequential organizations and interactional features found across examinees’ levels serve as critical validity evidence for assessing interactional competence. Various shift markers and stepwise transitions were present in higher-level examinees when they initiated and shifted actions in role-play interaction. However, lower-level examinees’ opening turns were typically forwarded without establishing a shared understanding relevant to an upcoming action. When the examinees responded to various proposal sequences, coherent and sufficient topic organizations were recurrent in higher-level performances. The examinees, regardless of levels, managed to close the role-play interaction well. I discuss the implications of the demonstrated link between the recurrent interactional features and examinees’ interactional competence for future research into speaking assessment and teaching.
Keywords
Investigations into second language (L2) interactional competence as a target domain in the assessment of spoken language ability have been a growing focus of research (Roever & Kasper, 2018). However, lingering research issues include the difficulty of isolating particular resources that an individual examinee employs during assessment interaction (Young, 2000) and the need for more robust evidence of interactional competence at various performance levels (Seedhouse, 2012). Without addressing these issues, claims of construct validity in assessment that seek to target interactional competence lack a clear empirical basis. This qualitative study takes up the issue of validity evidence of interactional competence in role-play assessment by examining how interactional features found in examinees’ interactional work differ at various performance levels. The analytical focus of this study is on how examinees manage the interactional work involved in proposal sequences (e.g., launching a proposal sequence, signaling a transition, and responding to a proposal) in role-play assessment. Recurrent interactional features of 38 examinees of varying levels, selected from a larger database of 102 examinees’ role-play interactions, were analyzed by using conversation analysis (CA). Based on the findings, I argue that CA’s analytical attention to real-time details of role-play interaction captures critical interactional phenomena indicative of varying levels of interactional competence, essentially serving as the evidential basis of construct validity.
L2 interactional competence
The critical discussions of the communicative competence paradigm established interactional competence as a target domain in the assessment of spoken language ability (Hall, 1995; McNamara 1997) and the term “interactional competence” was introduced in the field of applied linguistics in the 1980s (Kramsch, 1986; Schmidt, 1983). Carried further by various disciplines, including ethnomethodology and CA, interactional competence is generally defined as participants’ competence for social interaction, which is socially grounded and shared with social group members. Participants draw on a set of routinized yet context-specific structures of talk-in-interaction to accomplish social actions while managing social identities (Hall & Pekarek Doehler, 2011; Roever & Kasper, 2018; for a more comprehensive discussion). When it comes to interactional competence in L2, various factors, such as interactional competence in first language (L1) and L2 proficiency, can shape learners’ L2 interactional competence. Drawing on interactional abilities in their L1, L2 learners “recalibrate” features of social interaction while using linguistic resources at their disposal during their L2 development (Pekarek Doehler, 2018, p. 6). Areas of investigation include what comprises L2 interactional competence, employing the descriptive focus of CA (e.g., Lee, 2006; Waring, 2012), and how L2 learners develop interactional competence (e.g., Hall, Hellermann, & Pekarek Doehler, 2011; Hellermann, 2008). In recent discussions (e.g., Sert, Kunitz, & Markee, 2018), the role of linguistic resources for social actions in the development of L2 interactional abilities (Pekarek Doehler, 2018) and the teachability of interactional competence (Waring, 2018) have been addressed as well. This line of research illustrates that CA’s descriptive analyses allow for the identification of recognizable objects of learning in various domains of L2 use.
In terms of assessing interactional competence, given the complexity of delineating the scope of the construct, aligning the intended construct of interactional competence with learners’ performances is not a simple endeavor, particularly when different assessment methods are involved. Further, interactional competence is jointly constructed by speakers, rather than a primary attribute of an individual participant (Jacoby & Ochs, 1995; McNamara, 1997; Young & He, 1998, p. 7). The co-constructed nature of interactional competence poses several challenges in language assessment situations where making an inference of an individual examinee’s language proficiency is essential (Kasper & Ross, 2013; Taylor & Wigglesworth, 2009; Young, 2000).
In order to strengthen the assessment practices of interactional competence, we need to ensure that the recognizable interactional phenomena elicited from test instruments are fully construct-relevant as the evidential basis for the target construct and valid research claims (Kane, 1992; Norris & Ortega, 2003; Purpura, Brown, & Schoonen, 2015). In developing an empirical basis of spoken language ability, one approach employed is to analyze language test data at various levels (e.g., Iwashita, Brown, McNamara, & O’Hagan, 2008; Seedhouse, 2012), which is also the approach adopted in the present study. This study draws on the conceptualization of interactional competence grounded in ethnomethodology and CA, which focuses on investigating participants’ mutually shared practices for organizing social interaction, such as turn-taking, repair, disagreeing, and opening/closing a conversation. Thus, this conceptual approach warrants CA as an analytical framework to examine the interactional work of role-play assessment discourse in this study. In the next section, CA’s methodological contributions to the production of validity evidence of interactional competence are discussed.
Assessing L2 interactional competence
An increasing body of CA research on speaking assessment discourse has highlighted the contribution of CA in improving the practices of assessing interactional competence (e.g., Kormos, 1999; Lazaraton, 1992, 2002; May, 2011; Roever & Kasper, 2018; Ross, 1992; Ross & Kasper, 2013; Sandlund, Sundqvist, & Nyroos, 2016; Young & He, 1998). By now, CA is a well-established research methodology in applied linguistics (Kasper & Wagner, 2014). The fundamental concern of CA is to examine the sequential organization of talk-in-interaction and how participants orient to and achieve orderly conversation by employing an emic perspective that takes the viewpoint of participants (Sacks, Schegloff, & Jefferson, 1974). Rather than focusing on why speakers say what they say, CA’s interest lies in how turns are constructed and how participants orient to each other’s turns in talk-in-interaction focusing on various types of organizations, such as turn-taking, sequence organization, preference organization, and repair (Drew & Heritage, 1992; Sidnell, 2013). Essentially, CA’s analytical apparatus provides a useful methodological approach for defining and operationalizing interactional competence.
Previous research investigated how interaction in various types of speaking assessment discourse is organized using CA and found numerous analytical topics for language assessment. For example, these efforts resulted in well-established findings of how interaction is organized in oral proficiency interview (OPI) discourse (e.g., Kasper, 2013; Kasper & Ross, 2007, 2013; Seedhouse & Harris, 2011; Young & He, 1998) and paired speaking tests (e.g., Galaczi, 2008, 2014; Sandlund & Sundqvist, 2011). OPIs are mainly organized as question and answer sequences, in which interviewers take a main role of leading an interview using scripts. Thus, one of the main analytical foci in OPI research has been how interviewers’ interactional styles affect examinees’ performances. To illustrate, interviewers who deviate from interlocutor scripts can give an advantage to some examinees. Interviewers who use frequent backchanneling during an interview can provide an opportunity for examinees to develop topics further, in comparison to those who move tightly from one question to the next (Brown, 2003, 2006; Seedhouse & Egbert, 2006). Although OPIs can elicit various components of interactional competence, substantial differences between OPIs and ordinary interaction were found in terms of turn-taking, sequence organization, and topic nomination (e.g., Johnson & Tyler, 1998), providing evidence that the OPI is a type of specialized institutional talk. Further, the fixed interactional role of OPI examinees, who mainly respond to an interviewer’s questions, limits the capacity of OPIs to elicit interactional competence in various social activities.
Role-plays, on the other hand, can introduce a wide range of social activities by including various contextual variables (e.g., a relationship between interlocutors, pragmatic actions, settings) (Kasper & Rose, 2002), which allows examinees to accomplish social actions in talk-in-interaction. Previous CA studies confirm that elicited role-play interaction shares diverse interactional features that occur in ordinary conversation (Al-Gahtani & Roever, 2012, 2018; Huth, 2010; Okada, 2010; Okada & Greer, 2013; Stokoe, 2013). Role-plays also allow for more sequence initiation from examinees compared to OPIs (Seedhouse & Nakatsuhara, 2018). These findings mean that the simulated role-play interaction data, although it is not fully authentic, can support inferences as to examinees’ ability to accomplish social actions while engaging in meaningful interaction.
Apart from the studies on how interaction in speaking assessment discourse is organized, an emerging body of CA research reports a range of fine-grained interactional features that provide the concrete evidence of examinees’ interactional competence (e.g., Roever & Kasper, 2018). For example, Al-Gahtani and Roever (2012) reported that learners at varying proficiency levels managed the request sequence in role-play interaction differently. The lower-level participants’ requests were typically forwarded without pre-sequences. Pre-sequences refer to the sequences that come before a specific action, such as a request or invitation, which function as preparation for an upcoming action by establishing a topic or confirming that the conditions necessary for the accomplishment of the action are present (Schegloff, 2007). In addition to the pre-sequences in role-plays, examinees’ OPI performances were examined using CA. He (1998) described the interactional features of an examinee who failed the OPI, including a lack of elicited elaboration in responses and off-topic responses that prompted the interviewer to repeat questions. With a larger sample size, Seedhouse (2012) investigated the type of interaction that explains high and low scores given to OPI examinees and reported that the most important indicators of the scores given to the examinees were as follows: (a) the learner’s ability to answer questions; (b) coherent topic development; (c) the amount of repair; (d) identity construction; and (e) the length of turn. Both He (1998) and Seedhouse (2012) reported that low-level examinees were not able to provide elicited responses. Such a pattern is also suggested in other test formats. In the context of paired/group speaking tests, Gan (2010) and Galaczi (2008) reported that one of the features that distinguished higher-level from lower-level examinees was the degree of mutuality established among the participants. Mutuality reflects the degree to which the shared meaning is established from one turn to the next, which is evident in the different amount of participants’ contributions to the resulting interaction. These concrete interactional features of speaking assessment interaction can serve as an empirical basis of developing valid rating criteria in performance-based language assessment as well (Fulcher, 1996). For example, Youn (2015) also used CA to identify a range of interactional features that explain examinees’ abilities of achieving pragmatic actions in interaction, which informed concrete descriptions of rating criteria corresponding to varied levels.
In summary, the key contribution of CA in speaking assessment research is to furnish a conceptual and methodological lens to describe the nature of interactional competence specific to different test formats. This description can then function as concrete validity evidence of interactional competence. In terms of necessary additional work, however, there is relatively little attention paid to analyzing role-play test interaction with an explicit focus on the interactional features that can explain different performance levels, except for recent studies (e.g., Al-Gahtani & Roever, 2012, 2018). It remains to be empirically demonstrated whether previous findings are also instantiated in role-play interaction and what interactional features characterize high- and low-level role-play performances. Thus, the additional research necessary includes the extent to which the interactional features demonstrated in examinees’ role-play interaction distinguish among varying levels of interactional competence. For this, we need to establish a robust evidential basis by investigating examinees at various levels with a larger sample size to establish a certain degree of generality (Markee, 2017) of interactional organizations in speaking assessment discourse and to investigate whether recurrent interactional organizations can be applicable to different speaking test formats. In addition, the ways in which examinees at different levels propose ideas in role-play interaction are relatively unknown. Proposal sequences often encompass pragmatic actions (e.g., acceptance, rejection, agreement, disagreement, and negotiation) and distinct sequential organizations (e.g., initiating and shifting topics) (Schegloff, 2007), providing ample opportunities for speakers to employ interactional competence. Given that L2 learners’ abilities to open a conversation and shift topics in interaction develop gradually (Hellermann, 2007; Lee & Hellermann, 2014), examinees’ abilities to manage proposal sequences deserve further attention as validity evidence of interactional competence. Accordingly, in terms of the analytical emphasis of this study, I focus on proposal sequences: how L2 learners propose ideas during role-play assessment interaction.
The present study
In this study I employed CA to investigate the sequential organizations and interactional features with which examinees jointly accomplish various phases of proposal sequences in the context of role-play assessment. The present study addressed the following research questions.
In what ways do examinees manage various phases of proposal sequences in role-play interaction?
Do examinees at different levels manage role-play interaction differently? If so, what interactional resources characterize the differences?
Methods
Data
The data for this study came from 38 examinees selected from a larger database of transcribed 102 examinees’ role-play performances on a range of scenarios. The role-plays were developed to measure L2 learners’ pragmatic abilities in spoken interaction in a previous study (Youn, 2015), informed by a task-based pragmatic needs analysis of an academic setting (Youn, 2018). Compared to a traditionally used pragmatic instrument (e.g., discourse completion task), role-plays allow eliciting examinees’ pragmatic performances over multiple turns in interaction (Kasper & Rose, 2002). Following the discursive approach to L2 pragmatics (Kasper, 2006), the construct being measured through the role-plays is examinees’ abilities to accomplish pragmatic actions in spoken interaction, which entail their abilities to utilize a range of interactional and linguistic resources appropriate to a given context (Al-Gahtani & Roever, 2012, 2018; Mori & Nguyen, 2019). Considering that the construct focuses on pragmatic abilities in spoken interaction, the examinees’ interactional competence, particularly their abilities to organize series-of-turns effectively to accomplish pragmatic actions during talk-in-interaction, is a critical dimension of the construct being measured in the role-plays.
Examinees voluntarily participated in this study and were either graduate or undergraduate English as a second language (ESL) learners enrolled in public universities in North America. Of the 102 examinees, 70% were female and 30% were males. The examinees’ TOEFL iBT® scores ranged from 65 to 111, which is likely in the upper B1 to lower C1 band of the Common European Framework of Reference (Council of Europe, 2001; ETS TOEFL, 2018). The full database includes approximately 11-hours-long audio recordings of examinees’ performances on five different role-plays. These role-plays were not part of high-stakes assessment, but they were developed for the purpose of assessing pragmatic ability in spoken interaction within a research project.
Of the five role-plays that the examinees completed, this study only draws on the two role-plays that involved proposal sequences. In these role-plays, the examinees acted as two classmates working on a group project and were expected to accomplish two actions: (a) finding an agreeable meeting time; and (b) negotiating an agreeable meeting mode between face-to-face and online discussion options (see Appendix A). Each examinee completed the two role-plays with the same partner, with each role-play interaction lasting from one to two minutes. There was no preparation time limit and most of the examinees did not take more than five minutes for planning, except for several lower-level examinees who needed more time to understand the instructions. The test was designed following an open role-play format (Kasper & Rose, 2002). Interactional outcomes were not imposed on the examinees and contingencies were embedded in role-play cards given to each examinee to ensure some degree of authenticity. For example, when scheduling the time, each examinee’s schedule was not shared; when discussing the meeting mode, each examinee was allowed to express his or her own preference between the face-to-face and online meeting choices. Furthermore, in order to achieve standardization among the examinees, each examinee had an equal number of opportunities to contribute to the role-play interaction in terms of initiating, shifting, and developing topics, which were specified in the role-play cards. For example, when two examinees completed the two role-plays, one initiated the conversation in the first role-play (i.e., negotiating an agreeable meeting time); the other then had an opportunity to initiate the conversation in the second role-play (i.e., negotiating a meeting mode). In this way, the dominance of one examinee during the role-play interaction was minimized.
All role-play performances were scored by 12 trained raters using analytic rating criteria (see Appendix B). The data-driven rating rubric was developed to reflect various features of role-play performance, which served as an external measure within the larger research study to divide the examinees into three different levels of role-play performance (High, Mid, Low). The analytic rating criteria included concrete linguistic and interactional features reflected in five rating categories (Content Delivery, Language Use, Sensitivity to Situation, Engaging with Interaction, and Turn Organization) on a three-point scale. The validity of rating criteria was investigated by using qualitative (CA) and quantitative data (reported in Youn, 2015). The individual rater training involved various phases (including the introduction of test instruments and rating criteria) to establish the rater’s shared understanding of distinct scales in the rating criteria descriptions, and scoring practice. CA-based transcripts and concepts (e.g., turn-taking, adjacency pair) were also introduced during rater training, especially for the two rating criteria (i.e., Engaging with Interaction and Turn Organization) that specifically reflected the interactional features. In terms of determining examinees’ performance levels, the logit values derived from a multi-faceted Rasch analysis of rated performances from a previous study (Youn, 2015) were used. The logit scale transforms the probability of a particular response, which offers a more accurate estimate of examinee’s ability in relation to the task difficulty and rater severity (McNamara, 1996; Eckes, 2011). The examinees’ ability logit values below zero were categorized into the low level, indicating that they at least had a 50% chance of successful performance on the role-plays of average difficulty levels. The logit value of 1.4 was used as a cut point that divides between mid- and high-level examinees, as the logit value of 1.4 indicates that examinees have an 80% chance of being successful on the role-plays.
Data analysis
The methodology employed in this study is CA. All 102 examinees’ audio-recorded role-play performances in the database were transcribed following the Atkinson and Heritage (1984) transcription style (see Appendix C). The 38 examinees’ extracts selected from the database were included in this study. The 38 examinees are not intended to represent the entire data set of 102 examinees in the database; rather, the selected extracts are the result of a systematic data analysis and selection procedure. I first followed the general strategy for data exploration with unmotivated looking (ten Have, 2007). For this, I started analyzing randomly selected parts of the transcribed data focusing on basic types of interactional organization, such as turn-taking, sequence organization, preference organization, and repair organization. After this process, the recognizable sequential organizations and interactional features from examinees at varying levels were identified in terms of how they managed various phases of proposal sequences. Then, a collection of examples of the same phenomenon was identified using the inductive search through the database. For example, after identifying ways in which examinees launched the proposal sequence, a collection of launching the proposal sequences was identified and then patterns with regard to the examinees’ levels were examined. In doing so, 58 examinees’ data exemplifying the recurrent interactional features reported in this study were selected from the database. Of these, 38 examinees’ data that represent varying levels, L1s, and gender were included in this study. The selected examinees’ came from 10 different L1 backgrounds: Korean (34%, n = 13), Japanese (18%, n = 7), Chinese (16%, n = 6), Indonesian (11%, n = 4), Farsi (5%, n = 2), Vietnamese (5%, n = 2), Singhalese (3%, n = 1), Spanish (3%, n = 1), Thai (3%, n = 1), and Yoruba (3%, n = 1).
Results
This section is organized in terms of the recognizable phases of proposal sequences (launching a proposal sequence, signaling a transition, responding to the proposal, and closing the proposal sequence) in the role-play interaction. Again, the following extracts are not intended to represent a comprehensive list of proposal sequences; they offer representative examples selected from the examinees at varying levels. In the extracts presented subsequently, Phoenix (P) and Jessie (J) indicate the roles played, but these are almost always different examinees in the extracts.
Launching a proposal sequence
In the data, distinct interactional organizations of launching a proposal sequence were found corresponding to examinees’ levels. Extracts 1 to 3 represent typical opening sequences among higher-level examinees that involve pre-sequences before proposing the meeting time discussion. Pre-sequences refer to the sequences that come before a specific action (e.g., invitation, request), which project the possibility of the upcoming action (Schegloff, 2007).
Extract 1 (P: High, J: High, ID17&18RP1-1)
Extract 2 (P: High, J: High, ID29&30RP1-1)
Extract 3 (P: High, J: High, ID69&70RP1-1)
In Extracts 1 and 2, P displays the prosodic format with a rising tone in line 1 signaling an initiation of longer sequences (Couper-Kuhlen, 2001), followed by a greeting exchange. In Extract 1, before getting down to business, P formulates the mutually assumed knowledge of an upcoming presentation in line 8. J’s response, sure, in line 9 serves as a preferred go-ahead response, advancing the sequence’s trajectory. Such sequential patterns are also shown in Extract 2 (lines 3–4) and Extract 3 (lines 4–5), which serve as a topic proffering sequence (Schegloff, 2007). By proffering the topic, P makes it clear that an action related to the topic (i.e., proposing a meeting time) is coming up. Throughout the interaction in all three extracts, both P and J orient to members’ shared responsibility of deciding an agreeable time as classmates, evident in various linguistic expressions (e.g., we, should, have to, and gotta) and sequential organizations, such as a of course sounds good response to a meeting proposal with no inter-turn delay (e.g., line 10 in Extract 3). Here, the participants jointly construct the actions with a high level of mutuality, demonstrating their orientation to the goal-oriented role-play interaction.
On the other hand, P at a lower level often launches a meeting proposal sequence without pre-sequences as seen in Extracts 4 and 5.
Extract 4 (P: Mid, J: High, ID19&20RP1-1)
Extract 5 (P: Mid, J: Low, ID41&42RP1-1)
Extracts 4 and 5 are typical of the opening sequences initiated by P with a lower level in the data. In Extract 4, P summons J with the rising tone in line 1 and launches a sequence by asking for a specific meeting time upfront in line 3, which blocks any pre-sequences, such as a greeting or establishing shared understanding. After a delay in line 4 of Extract 4, J responds to P by asking for a specific time, displaying orientation toward task completion. In Extract 5, P launches a so-prefaced turn construction unit in line 1 which functions as a topic sequencer to introduce recipient-attentive matters (Bolden, 2008). However, the opening turn in line 1 only includes a statement of the shared responsibility along with a time availability question without establishing shared understanding or a topic relevant to an upcoming action. After a 0.6 second delay in line 2, P states a specific time availability in line 3, orienting to the lack of J’s immediate uptake toward P’s explicit attempt to get an answer from J in line 1. Compared to Extracts 1 to 3, no pre-sequence was shown in the opening sequences by P at lower levels. In addition, more frequent inter and intra-turn delays were present in Extracts 4 and 5, indicating a lower degree of mutuality established between the examinees.
Signaling a transition
Upon completing the first role-play (i.e., deciding an agreeable meeting time), the recording device was paused for a short break so that each examinee had an opportunity to read the second role-play card, in order to minimize pauses due to rereading the role-play card. Since the short break between two role-plays was typically no more than three minutes, both examinees made a reference to the first role-play interaction and made a smooth transition to the second role-play of proposing meeting options to complete the class project together. Thus, what appeared in the first turn in some extracts below (e.g., Extracts 8 and 9) was the negotiation of an agreeable meeting time. Following the open role-play format, the contents were not completely shared among the participants, so the two meeting choices (face-to-face vs. online discussion) were only included in J’s role-play card. The turns in signaling the shift between sequences of actions were regularly constructed when marking the forthcoming new topical content. In the data, three patterns of sequential organizations in signaling the shift were found: (a) extended topic-shift moves, either disjunctive or stepwise (Sacks, 1992), which are further explained below; (b) less successful moves with topic markers present; and (c) absence of any shift markers. These three patterns corresponded to the examinees’ levels. Extracts 6 and 7 demonstrate the disjunctive topic movement accomplished by J at higher levels.
Extract 6 (P: High, J: High, ID7&8RP1-2)
Extract 7 (P: High, J: High, ID93&94RP1-2)
Disjunctive transitions (Jefferson, 1984; Sacks, 1992) refer to participants’ use of prefaces in a turn initial position when signaling a move to the introduction of a new topic. For example, speakers often mark an upcoming action by prefacing it with topic-shift markers, such as the change of state token oh (Heritage, 1984), or recycle previous talk with the preface of speaking of … (Jefferson, 1972). In Extracts 6 and 7, various disjunctive topic-shift markers that signal an upcoming topic (Jefferson, 1984; Sacks, 1987; Schegloff & Sacks, 1973) surfaced in J’s turns, such as alright, you know what (line 1, Extract 6), the change of state token oh (Heritage, 1984) (line 1, Extract 7), and before I leave (line 1, Extract 7). P agrees to the topic shift with no inter-turn delay, indicating a high degree of mutuality established.
A more extended stepwise topic-shift movement which refers to a more gradual way of shifting topics, such as linking a previous topic discussed in previous turns when moving onto new topics, was also found in high-level examinees’ data, as shown in Extracts 8 and 9.
Extract 8 (P: High, J: High, ID31&32RP1-2)
Extract 9 (P: Mid, J: High, ID85&86RP1-2)
Both Extracts 8 and 9 demonstrate stepwise topic movement, where J launches mutually shared knowledge on the agreed meeting time (line 3, Extract 8; lines 1-2, Extract 9), discussed in the first role-play before marking the new topical content. In doing so, in Extract 8, J first marks the new topical content in an open-ended question format in line 6 before proposing two specific meeting ideas in line 9. In Extract 9, J explicitly formulates a turn in line 5 using we and have to, orientating to the members’ shared responsibility. Here, when J proposes the new topical content (i.e., meeting mode), various topic-shift markers were also present, such as you know (line 6, Extract 8), so and now (line 5, Extract 9). During J’s topic-shift movement, P agrees with J’s actions with no inter-turn delay, providing acknowledgement tokens in both Extract 8 (line 5) and Extract 10 (line 3).
In mid-level examinees’ data, the topic-shift movement sequences were also present. However, the sequential progression of topic-shift movement often did not include disjunctive markers or a pre-proposal sequence.
Extract 10 (P: Low, J: Mid, ID97&98RP1-2)
Extract 11 (P: High, J: Mid, ID45&46RP1-2)
In Extract 10, J (mid-level) first summarizes the previous topical content with noticeable inter-turn delays and the prosodic format that indicates a continuing topic sequence in lines 1 and 2. With a 1.7 second pause in line 3, J projects a but-prefaced turn and directly asks the meeting option preference in lines 4 to 5 without any disjunctive markers or a pre-proposal sequence that connect the topic summary and the new topic launch. Similarly, Extract 11 shows J’s incomplete topic-shift movement. J’s initial turn (lines 1–2) includes an incomplete turn construction unit which is cut off as J restarts with I mean and self-repairs. After P’s lack of immediate uptake, J directly launches two specific meeting options in lines 4 and 6 without topic-shift markers or a turn that contextualizes the meeting options discussion. Here, a relatively low degree of mutuality was evident, with P providing no immediate uptake (line 3, Extract 10; line 3, Extract 11). Taken together, the mid-level examinees attempted to signal a transition between actions by summarizing the prior topic or using prefaces (okay, so). However, the organization of topic-shift sequence was less extensive compared to the high-level examinees’ sequences.
Extract 12 shows a typical topic-shift sequence shown in low-level examinees, which is characterized as no pre-expansion along with noticeable inter-turn and intra-turn delays.
Extract 12 (P: Low, J: Low, ID33&34RP1-2)
J summons P with the rising tone in line 1, followed by no immediate response from P. With a 3-second intra-turn pause, in lines 3 to 4, J asks an open-ended question about how they will meet, rather than making a proposal using the two meeting options mentioned in the role-play cards. Here, compared to the extracts discussed before, J does not provide the details that are related to two meeting options, although J was supposed to mention the two meeting mode choices according to J’s role-play card. After a 2.2 second long gap of silence in line 5, P attempts to propose an idea in line 6.
Responding to the proposal
In discussing the meeting mode option, the examinees were allowed to choose between the face-to-face and online discussion meeting options based on their own personal preferences, which resulted in opportunities for engaging in meaningful interaction. The higher-level examinees’ turns typically included sufficient information related to the situation and their interactional moves were strategic which eventually led to an effective negotiation. However, lower-level examinees’ turns lacked coherent and sufficient details regarding a proposed meeting option across sequences, to which the participants oriented by prompting additional turns to seek further information. Extracts 13 and 14 show how the higher-level examinees engage in responding to the proposal.
Extract 13 (P: High, J: High, ID43&44RP1-2)
Extract 14 (P: High, J: High, ID75&76RP1-2)
In Extract 13, J proposes two meeting options to P in lines 4 to 8, pursuing an explicit answer using a wh-question (what do you want) in line 8. P does not immediately align with the action, which is evident in a hearable delay (uhm:) in line 9 and a 2.5 second pause in line 10. Instead of choosing a particular meeting option, P proposes a hybrid meeting mode, namely both face-to-face and online discussions, in lines 11 to 13 (an option not explicitly mentioned in the role-play cards). J does not immediately agree with P’s proposition, evident in a 1.8 second delay in line 15 and an explicit preference for the online discussion option (line 16). However, J eventually agrees with P’s proposition, providing the reason of the third group member being absent in lines 16 to 20. Here, P’s hybrid meeting proposition preempted potential disagreement turns. Extract 14 shows a similar sequential environment in which P proposes a hybrid meeting option in lines 11 to 13, followed by J’s immediate agreement in line 14. In both extracts, P’s turns related to proposing ideas are accompanied by sufficient information with regard to the situation.
On the other hand, some mid-level examinees do not provide an elaborated reason in response to J’s meeting proposal turn, which results in undue between-turn pauses, as the lack of expected detail while exchanging opinions deviates from the normative turn organization, as seen in Extracts 15 and 16.
Extract 15 (P: Mid, J: Mid, ID15&16RP1-2)
Extract 16 (P: Mid, J: High, ID19&20RP1-2)
In Extract 15, J’s explicit inquiry about the meeting mode preference is projected in lines 4 to 5. P then states a meeting mode preference in lines 7 to 8 without explicating the reasons. J then makes P accountable in several subsequent turns. After a noticeable delay in line 9, J displays surprise in line 10. With P’s minimal response in line 12, J explicitly pursues a reason in line 13 using a wh-question. Extract 16 demonstrates a similar sequential environment. Here, J (high-level) proposes the meeting options with more elaborated detail on the online meeting option in lines 6 to 10. However, P (mid-level) offers only an explicitly stated preference in line 11, again with no reason provided. J then orients to the absence of expected explanations by soliciting reasons in line 13. Both Extracts 15 and 16 show that P’s individual turns (mid-level) lacked the sufficient details expected in disagreement sequences. The recipient, in response, orients to this departure from a normative turn-taking structure of disagreements in preference organization (Pomerantz, 1984; Pomerantz & Heritage, 2013).
The low-level examinees also do not fully engage in the proposal sequence, evident in both inter-turns and intra-turns with a noticeably short length of each turn, as seen in Extracts 17 and 18.
Extract 17 (P: Low, J: Low, ID57&58RP1-2)
Extract 18 (P: Low, J: Low, ID5&6RP1-2)
In Extract 17, in response to J’s inquiry of a meeting mode preference in line 9, P states the online discussion preference in line 11. J then offers minimal acknowledgement tokens toward P’s opinion, demonstrated by a hesitation (u:hm) and a prosodic format (°yeah°) in line 12. Orienting to a 1.8 second pause in line 13, P provides a minimal reason in lines 14 to 15. What is distinct in lower-level interaction is that neither participant actively orients to accomplishing the proposal sequence, demonstrated by the short turn length and the lack of a topic prompting turn. Proposal turns (e.g., line 11 and 14 in Extract 17) also do not include sufficient information. Extract 18 demonstrates an even more minimal number of turns. The proposal sequences on the meeting mode only span lines 6 to 9. After a noticeable delay in line 10 with no uptake by J, P moves onto a different action following the information specified in P’s role-play card.
Closing the proposal sequence
In the data, recognizable closing sequences that indicate that the conversation is moving to an end were found, with no evident association with the examinees’ levels. A typical closing sequence consists of pre-closing signals, an arrangement sequence, and a termination sequence (Wong & Waring, 2010). Extracts below are chosen from the examinees at varying levels.
Extract 19 (P: Mid, J: High, ID61&62RP1-2)
Extract 20 (P: Low, J: Low, ID57&58RP1-2)
Extract 21 (P: Mid, J: High, ID67&68RP1-2)
Extracts 19 and 20 demonstrate a typical closing sequence consisting of a pre-closing signal (okay), an arrangement sequence (e.g., line 69 in Extract 19; lines 34 to 36 in Extract 20), and a termination sequence (bye, see you). Some closing sequences were short, such as Extract 21, but closing signals (okay, alright) and an arrangement sequence (I will let you know in line 41) surfaced, demonstrating that the examinees regardless of their levels oriented to a completion of the task at hand.
Discussion
In this study, I aimed to examine the evidential basis of interactional competence through a qualitative analysis of the interactional work involved in managing proposal sequences in role-play assessment using CA. With regard to Research Question 1, the examinees jointly oriented to accomplishing various phases of proposal sequences, such as launching the proposal sequence, signaling the shift between actions, responding to the proposal, and closing the proposal sequence, following the conversational agenda imposed by the role-plays setup. The role-play cards, designed to standardize the role-play interaction and to distribute equal opportunities for each participant to contribute to the interaction, were intertwined with the recognizable phases of managing the role-play interaction. As such, the examinees’ orientation for staying on task and completing the role-plays was evident throughout the interaction, taking up their assigned identities as classmates. Echoing the findings reported in previous CA studies on role-play interaction (e.g., Al-Gahtani & Roever, 2012, 2018; Huth, 2010; Okada, 2010; Okada & Greer, 2013), the sequential environments and interactional features shown in various phases of proposal sequences in this study resembled the features of real-life conversation. In addition, the findings illustrated that the more competent examinees, according to their overall performances on the role-plays, actively utilized a range of interactional resources in their proposal sequences, including a pre-sequence, topic-shift movement sequences, and responding to the proposal. In other words, the examinees employed contextually occasioned interactional resources to establish the shared understanding in managing the proposal sequences beyond grammatical resources. Such an ability to engage in proposal sequences is also reported in classroom learning contexts. For example, in Lee and Hellermann’s (2014) CA-based study on the sequential details of L2 development, the longitudinal data on topic-shifts during story-telling episodes elicited from pedagogical tasks illustrated that the learners’ ability to shift topics and mark the new topic developed over time. Hellermann (2007) also reported that lower-level L2 learners’ ability to establish the necessary context when opening proposal sequences in peer tasks developed gradually. Taken together, the interactional work involved in the proposal sequences is a visible phenomenon of L2 language use and, therefore, should be more explicitly considered as a ratable object for assessment purposes.
With regard to Research Question 2, distinct patterns of sequential organizations and interactional features were found in recognizable phases of proposal sequences corresponding to varying levels of examinees. In terms of opening and signaling the shift sequences, what distinguished the higher-level examinees from the lower-level examinees included diverse topic-shift markers (e.g., you know what) and pre-sequences of formulating mutually assumed knowledge, serving the purpose of establishing shared understanding among the participants. The lower-level examinees often launched into the main topic without any pre-expansions in their opening turns, precluding the participants from accomplishing the shared context. The lack of establishing a mutually shared context in lower-level L2 learners’ interaction was also reported elsewhere, such as in classroom interaction (e.g., Hellermann, 2007). Given the recurrent patterns shown in the conversation opening and shifting sequences in this study, prefacing devices involved in speaking assessment discourse or learner interaction may be a useful gauge for differentiating L2 speakers’ interactional competence. With regard to the sequences where the learners engaged in negotiating various proposals, the features in individual turns and the extent to which mutuality is being accomplished among the participants may also indicate learners’ levels of interactional competence. The individual turns in higher-levels included clear, substantive, and coherent content necessary for proposing opinions. In particular, the normative turn-taking structures involved in disagreement sequences, such as providing accounts or meaningful inter-turn delays, were commonly observed in the higher-level performances. Additionally, the higher-level examinees effectively organized their sequences, such as proposing a hybrid meeting mode upfront with reasonable accounts, which eventually prevented additional negotiation. The lower-level participants’ turns often did not contain substantive topical content nor the normally expected turn-taking structures of disagreement, which are typically accompanied by delays and markers (e.g., well) of noticeable reluctance along with an explanation (Pomerantz, 1984). These differences emerged from the examinees at different levels when negotiating various proposals and further confirm the challenge of building connected discourse for L2 learners.
Without an empirical description of what an intended language construct constitutes, valid and reliable assessment is simply not possible. Based on the findings reported in this study, I argue that the sequential organizations and interactional features found during proposal sequences constitute critical validity evidence for assessing L2 interactional competence and, therefore, should be integrated into speaking assessment (Roever & Kasper, 2018). The findings of the study further contribute to advancing our understanding of what interactional competence entails and how it varies at different performance levels. In order to strengthen this argument, the extent to which those features are context-specific or task-dependent needs to be discussed, especially given the challenge of comparing different varieties of interaction elicited from different test formats (Seedhouse, 2012). Previous findings with regard to interactional organization in various speaking assessment discourse types (e.g., OPIs, paired/group speaking tasks, role-plays) offer insights into how the findings of this study hold some degree of generality (Markee, 2017). For example, the two typical topic transition sequences (i.e., stepwise and disjunctive) found in the higher-level performances in this study were also reported in Gan, Davison, and Hamp-Lyons’s (2009) study of L2 learners’ group speaking performances. Seedhouse’s (2012) findings on the features that distinguish high and low scorers in OPI, such as the extent to which learners’ responses are developed and turn length, also mirror the findings in this study. In the context of role-play, Al-Gahtani and Roever (2012) reported the absence of pre-sequences in lower-level learners’ request performances. The varying degree of mutuality associated with the learners’ levels in this study has also been found in group speaking tasks (Gan, 2010) and in paired speaking tasks (Galaczi, 2014) as a distinct feature that differentiates the high- and low-level performances. Taken together, it can be inferred that these are recurrent sequential organizations and interactional features that are applicable to assessing interactional competence elicited from various speaking assessment tasks.
Nonetheless, the extent to which the identified interactional features are only part of L2 interactional competence is an unresolved issue. Learners’ interactional competence in their own L1s, their personalities, and their strategy use can influence the ways in which they engage in talk-in-interaction. Speakers who are more socially skilled, regardless of L2 language proficiency, could be more interactionally competent in L2 (Hall, Hellermann, & Pekarek Doehler, 2011). Such a gray area calls for more research focusing on the relationship between L2 interactional competence and speaker-related variables to advance the reliable and valid assessment of interactional competence. Another challenge in employing CA for interactional competence assessment research concerns epistemologically incommensurable assumptions underlying in CA and language assessment (Kasper & Ross, 2013). CA focuses on contingent language use and how participants themselves orient to establishing shared understandings (i.e., emic). Conversely, language assessment often relies on predetermined external criteria (e.g., descriptions in rating criteria) and institutional goals that learners’ performance is then measured against (i.e., etic). One possible way to address the conceptual clash is to develop and facilitate more empirically grounded assessment practices for interactional competence, such as training raters with CA-based concepts and developing CA-grounded interaction-sensitive rating criteria. This way, raters can become more aware of how participants themselves achieve shared understandings using different interactional resources. Raters may then award scores accordingly using rating criteria that appropriately reflect interactional features. Although CA’s descriptive analyses of the details of interactional competence offer a useful analytical framework, further research will be critical in order to advance CA-based research that sets out to investigate the validity of interactional competence assessment.
This study offers implications for teaching and assessing interactional competence. The present study demonstrates what is lacking in lower-level learners when it comes to accomplishing various actions in their sequential organizations. Thus, a more explicit pedagogical emphasis on a range of interactional resources involved in spoken interaction is essential. For example, the lower-level learners’ attempts to launch or shift the proposal sequence without pre-sequences or topic-shift markers often resulted in a rather abrupt interaction. The abilities to open the sequence and shift topics in interaction are fundamental elements in talk-in-interaction, reflecting goal-oriented language use in real-life conversation. Given that role-plays enable learners to accomplish various communicative goals and allow opportunities for learners engaging in meaningful interaction, language teachers and testers can adopt role-plays as pedagogical and assessment tasks. At the same time, it should be noted that role-plays may not necessarily be appropriate for lower-level learners whose linguistic resources are limited for managing highly contextual role-play performances. Thus, given the importance of using various linguistic resources (e.g., various topic-shift markers including you know what, speaking of) in proposal sequences, as seen in this study, teaching the necessary linguistic resources for organizing actions (Pekarek Doehler, 2018) would be desirable for lower-level learners. To this end, CA’s analytical apparatus can be integrated into teaching (see Waring, 2018; Wong & Waring, 2010), which will facilitate learners’ explicit awareness on how turns are organized in talk-in-interaction and how a conversation flows sequentially.
Turning to implications for assessment, the findings in this study can be connected to various assessment practices, such as designing speaking tasks that indeed elicit a wide range of interactional features, developing rating criteria descriptions to capture interactional features, and training raters to attend interactional features while scoring. This study draws on the interaction-sensitive rating criteria developed by the researcher in a previous study (Youn, 2015). While the rating criteria were developed based on the CA of selected examinees’ performances, it can be further strengthened in terms of specifying the interactional features. Similarly, currently available scales (e.g., CEFR, Cambridge English speaking tests) that include the descriptions of interactional competence can be compared to the existing conversation analytic findings on speaking assessment discourse to examine whether they are reflective of key interactional features at various performance levels. Although the rating criteria descriptions cannot be too fine grained with regard to practicality, key interactional features that distinguish among different levels need to be well-captured. As a rapidly increasing field within L2 assessment, assessing interactional competence has significant room for growth.
The limitations of the study should be noted. This study only relies on the audio-recorded data, thereby rendering invisible nonverbal resources employed during interaction. Becoming interactionally competent includes an ability to incorporate nonverbal behaviors, such as gaze and gestures, when accomplishing social actions (Plough, Banerjee, & Iwashita, 2018). The audio-recorded data may have more accurately explained some of the inter-turn and intra-turn pauses found in this study’s data. In addition, the findings of the current study do not directly confirm whether the recurrent sequential organizations and interactional features are applicable to the rest of the data with quantitative power. Since the 38 examinees’ data presented in this study do not represent the entire 102 examinees in the database, the extent to which the remaining examinees’ performances in the data exhibit similar sequential organizations and interactional features is unknown, calling for future investigations.
Conclusion
The lingering validity issues in assessing interactional competence are the challenge of isolating interactional resources that an individual examinee employs and how they differ at various levels of achievement. In this study, I demonstrated the critical role of CA’s descriptive focus in the production of validity evidence of interactional competence, which can eventually contribute to the development of interactional speaking tests and rating criteria. It was evident that interactionally competent speakers actively utilized a range of interactional features while maintaining effective sequential organizations when managing proposal sequences involved in role-play assessment. Clear outcomes of the analysis in this study are the recurrent features involved in managing role-play interaction across a refined range of levels selected from a relatively large sample, which also resemble the findings reported in previous research on speaking assessment discourse. Thus, the learners’ ability to manage proposal sequences during assessment interaction should be considered as a key sub-construct of interactional competence. I argue that CA’s analytical attention to the real-time details of role-play assessment interaction captures the interactional phenomena indicative of varying levels of interactional competence, essentially serving as the empirical basis of construct validity. In addition to the quantitative account of validity evidence (e.g., Van Moere, 2006), the qualitative scrutiny of the construct of interactional competence enhances our understanding of what is being measured, which is fundamental to subsequent assessment practices. Future research on using CA to examining various interactional devices of talk-in-interaction in speaking assessment discourse, applicable to research in the context of L2 learning, teaching, and assessment, will make timely and valuable contributions to the field.
Footnotes
Appendix A: Role-play tasks and role-play cards
Appendix B: Rating criteria
| Score | Content Delivery | Language Use | Sensitivity to Situation | Engaging with Interaction | Turn Organization |
|---|---|---|---|---|---|
| 3 | • Clear, concise, fluent • Smooth topic initiations with transitional markers (i.e., smooth turn initiation) |
• Pragmatically appropriate linguistic expressions ( • Good control of grammar and vocabulary that doesn’t obscure meaning |
• Consistent evidence of awareness and sensitivity to situations exists in an appropriate sequence |
• A next turn shows understandings of a previous turn throughout the interaction (i.e., shared understanding) • Evidence of engaging with the conversation exists (e.g., clarification questions, backchannel, acknowledgement tokens) |
• Complete adjacency pairs (e.g., question & answer) • Interactionally fluid |
| 2 | • Generally smooth, but • • |
• Able to use modal verbs in • Linguistic expressions are occasionally inaccurate and a bit limited that sometimes obscure meaning. |
• Inconsistent evidence of awareness and sensitivity to situations (e.g., provide accounts for opinions, but do not necessarily handle the disagreement situation properly). | • Some evidence of engaging with the conversation, but not consistent (e.g., literally read the role-play card), • A next turn does not sometimes show an understanding of a previous turn. |
• Some turns are • Sometimes |
| 1 | • Delivery is choppy, fragmented, and minimal (due to a lack of language competence). | • Expressions sound • Linguistic expressions are inaccurate and quite limited that obscure meaning |
• Little evidence of situational sensitivity (e.g., absence of providing accounts for disagreement in particular, handle disagreement awkwardly) | • Noticeable absence of discourse markers • Evidence of not achieving a shared understanding |
• Noticeably abrupt overlap or no pauses for disagreement and refusal • |
Appendix C: Transcription symbols
: Lengthening of the preceding sound
- Abrupt cutoff
(.) Very short untimed pause
>word< Speech delivery that is quicker than the surrounding talk
<word> Speech delivery that is slower than the surrounding talk
w(h)ord A parenthesized h within a word indicates breathiness as in laughter
[ Point of overlap onset
= No gap between adjacent utterances
CAPITALS Especially loud sounds relative to surrounding talk
° ° Utterances between degree signs are noticeably quieter than surrounding talk
(3.5) Intervals between utterances (in seconds)
Acknowledgements
I wish to express my sincere appreciation to Dr. Gabriele Kasper for her insightful advice and comments on an earlier draft of this article. Special thanks go to Rue Burch for his helpful comments and suggestions on the article. I am also grateful to three anonymous reviewers and the editor, Luke Harding, for their constructive suggestions. I am responsible for all the errors that may remain.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Educational Testing Service (ETS) under a Committee of Examiners and the Test of English as a Foreign Language research grant. ETS does not discount or endorse the methodology, results, implications, or opinions presented by the researcher.
