Abstract
Introduction
Translanguaging is defined as “the ability of multilingual speakers to shuttle between languages, treating the diverse languages that form their repertoire as an integrated system” (Canagarajah, 2011: 401). Different from the concepts of traditional bilingualism and code-switching that treat two languages as separate linguistic systems with distinct features, translanguaging is used to highlight the transformation of various semiotic resources into “dynamic, mobile resources” (García and Li, 2014: 18) in response to different sociolinguistic contexts. Translanguaging as a pedagogical practice was originally viewed to involve modalities of listening, speaking, reading, and writing, as Li (2018: 10) illustrated when he discussed the relationship between translanguaging and multimodality. Recent theorizing of translanguaging has also included multimodal social semiotics, such as gestures and visual cues that often accompany linguistic signs, into an individual’s semiotic repertoire (García and Li, 2014). In other words, multilingual speakers’ employment of not just language but also other semiotic resources is deemed indispensable in the process of translanguaging. Analysis of multimodal translanguaging will provide an important interface of translanguaging research and research into multimodalities in the fields of education and communication.
The multimodal essence of translanguaging entails using instruments that allow researchers to annotate multimodalities involved in translanguaging. Annotation of multimodalities can delineate the orchestration of various semiotic resources, which makes the meaning-making process in certain contexts visible and more researchable. This is pivotal for pushing the field forward by moving from generic descriptions of translanguaging to unveiling how translanguaging intersemiotically unfolds. This line of research can also shed light on pedagogical practice in terms of adopting multimodal translanguaging to optimize pedagogical effects. One such instrument for annotating multimodalities is ELAN (EUDICO Linguistic Annotator). ELAN is a versatile tool for media data analysis developed at the Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands (Tacchetti, 2018), which is available free online (https://archive.mpi.nl/tla/elan). ELAN has been utilized to analyze language, gesture, and various types of video and/or audio recordings in pedagogical contexts. It is typically appropriate for analyzing semiotic resources in the process of translanguaging due to its various functions for transcribing, annotating, and calculating annotation statistics (presented below). This technology review gives an overview of the features of ELAN and its potential uses, followed by our reflection on its use in the multimodal discourse analysis of English language teaching videos collected from the Internet.
Overview
The ELAN is one of the most widely used annotation tools that satisfy the purposes of annotating, analyzing, and documenting multimodal data. It can be applied in mainly three computer operating systems: Windows, macOS, and Linux. While other annotation tools such as Multimodal Analysis Video (MMAV) are also available, ELAN has its advantages, for instance, being free and recognizing more formats of files such as Common Media Files, Moving Picture Experts Group (MPEG), Waveform Audio File Format (WAVE), and QuickTime files (see Figure 1). One possible limitation of this software is that its seemingly sophisticated design may discourage potential users.
Recognizable formats of files for ELAN 5.8 (2019).
Two steps are needed before making annotations in ELAN. The first step is to import a media file into ELAN and then process the file in line with different working modes such as the annotation mode, transcription mode, and segmentation mode. Each mode has its specific function as described in its name. For example, the transcription mode is for transcribing spoken language into written form and the segmentation mode is for dividing a video/audio into several sections. Teachers can use the transcription mode to assess students’ language use during classroom interaction, and the segmentation mode enables teachers to reflect on their use of multimodal translanguaging section by section, which is beneficial to their lesson planning and facilitative of the teaching–learning process.
Annotations in ELAN are conducted at tiers. Different tiers can be set up for the analysis of different modalities. For instance, when annotating a video clip, researchers can set up a verbiage tier, a facial expressions tier, a gaze tier, and a gestures tier to capture simultaneously the details of each modality in the clip (see Figure 2). Annotators can add, change, and delete tiers, and import tiers from other project files. When a limited number of annotation values can be predefined, these values can be created as a controlled vocabulary (CV) to ease the annotation task (see Figure 3). A CV can be imported and exported as a file (.ecv) (Cassidy and Schmidt, 2017). An annotation project (i.e., the annotation of a media file) is stored in an .eaf file, and the structure of tiers contained therein can be imported into other annotation projects conducted by other annotators.
An example of adding tiers.
An example of editing controlled vocabulary (CV).
Upon completing the annotation, researchers can obtain various types of information about annotations by consulting annotation statistics in ELAN. Examples of such information are the number of occurrences and frequency of certain values, average duration (i.e., the duration of an annotated value divided by the number of its occurrences), and time ratio (i.e., the duration of an annotated value divided by the observation period). These statistics are useful for further statistical analysis.
Reflection
We conducted a multimodal discourse analysis of translanguaging in a large number of video clips obtained from online open English courses. Here we take one video clip as an example, in which an English teacher taught English adopting both English and Chinese, accompanied by various body movements (see Figure 4). We first set up a CV to assign values to different tiers. For instance, we assigned five values (i.e., five types of hand gestures) to the gestures tier: iconic, metaphoric, deictic, beat, and performative. In an instance, the teacher said, “I did very, very well on my test, I killed my test I did amazing on my test”, and then knocked on the wooden desk, which was annotated as a metaphoric gesture. This is an instance of translanguaging in which the gesture implied the meaning of “hoping nothing bad would happen,” although this meaning was not expressed verbally. The use of translanguaging here may be underlain by the fact that in many cultures, the gesture of knocking on wood has its superstitious origin associated with hopes for good fortune and avoiding bad luck.
An example of translanguaging analysis in language teaching in ELAN 5.8.
This can be seen as a good example for discerning and annotating multimodal semiotics in translanguaging. In the same vein, language teachers can record the teaching and learning in class and then use ELAN to analyze students’ translanguaging features. Through a fine-grained analysis of the students’ translanguaging process, teachers can identify at which points students seem to transform the entirety of their semiotic resources into an integrated system and the reasons or motives (e.g., linguistic, psychological, or societal) underlying their translanguaging in class.
Conclusion
The ELAN is an annotation tool specifically designed for handling media data that are rapidly expanding and ubiquitous nowadays. It can greatly facilitate the annotation and analysis of multimodal discourses in educational contexts in general and in translanguaging in particular. Using this tool, researchers can “anatomize” and visualize the multimodal elements embedded in the translanguaging process, and the annotation statistics obtained are readily usable for further analysis. All versions of ELAN are available online and comprehensive user manuals are provided online (see: https://archive.mpi.nl/tla/elan/documentation).
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the China National Social Science Fund (Grant Number: 19BYY197).
