Insertion or voice-off in rendition of graphic codes: an experiment in Persian dubbing

1. Introduction

Audiovisual products can comprise a wide spectrum of visual codes ranging from actors’ facial expression and use of lighting to verbal information in written forms. Verbal information in written forms on screen is one of the four channels in audiovisual translation (AVT) with which translators need to engage. The concept is referred to in freshly yet variously coined terms and expressions such as ‘verbal visual’ (VV) (Chiaro, 2009; Delabastita, 1989; Díaz Cintas and Remael, 2007; Gottlieb, 1998; Sokoli, 2006; Zabalbeascoa, 2008) and includes subcategories such as ‘authorial titling’ (Pérez-González, 2013, 2014), ‘impact captioning’ (O’Hagan and Sasamoto, 2016; Sasamoto, 2014), ‘integrated subtitles’ (Fox, 2016), ‘on-screen text’ (Dwyer, 2015), ‘beyond screen text messaging’ (Zhou, 2014), ‘unspoken words’ (O’Sullivan, 2013), and ‘graphemes’ (Josephy-Hernández, 2017) which come from different methods that the graphic codes may use. Despite the sense implied in each of these categories, in this study, the term ‘graphic codes’, borrowed from Chaume (2004, 2012), is used for representing any text on screen such as emails, signs, newspaper headlines, SMS and internet texts.

Considering the increasing use of graphic codes in audiovisual products, especially feature films, they recently attracted the attention of AVT scholars. In recent years, graphic codes have been examined from a number of aspects such as: verbal information in written forms in film translation (Chiaro, 2009; Delabastita, 1989; O’Sullivan, 2013; Zabalbeascoa, 2008), in their non-translational function (Pérez-González, 2013, 2014; Sasamoto, 2014; Sasamoto et al., 2017), in the context of audio description (AD) (Orero, 2011), and by means of eye tracking (Dwyer, 2015; Redmond et al., 2015). A remaining gap in the full depiction of the image of this new phenomenon in AVT is a comparison of the common methods of the rendition of graphic codes in dubbing.

In the case of graphic codes, three methods are used for their rendition in dubbing (Chaume, 2004): audio-rendition of the target graphic codes (TGCs) with an actor’s voice-off and inserting the voice-off when the close shot is on the original graphic codes (OGCs); subtitling the OGCs; and editing and inserting new TGCs replacing the original when the OGCs have been deleted from the film.

The current study basically draws on descriptive translation studies (Toury, 2012) as the theoretical framework, which values detailed description of translational phenomena in specific contexts, locales and speech communities that can eventually lead to the discovery of translational universals or culture-specific translation norms. The discovery of either of these two can have pedagogical and professional implications which will prove useful both at a theoretical and practical level.

Based on the overall purpose of this study which is to fill in an existing gap in the literature and to compare the two common methods of the rendition of graphic codes in dubbing, namely voice-off and insertion, within an experimental design, the research takes advantage of the availability of the two official Persian dubbed versions of the Sherlock TV series, each of which uses one of the said methods in transferring the content of the frequent graphic codes. The comparison mostly focused on the comprehension and information recall of the messages in question.

Additionally, since these two methods have both advantages and disadvantages, the study also aims to analyse the views and comments of the viewers. With regard to the goals of the study, the following questions were designed and addressed.

2. Review of the Key Related Literature

Inspired by the terminology of film semiotics pertaining to the study of numerous meaning codes in cinema, Chaume (2004, 2012) advocates the use of the term ‘graphic codes’ for verbal visual information on screen. Within a filmmaking framework, Pérez-González (2013: 14) refers to graphic codes (authorial titling in his terms) as information that is added as part of the co-creational process; ‘titles without a translational function’ and simply serving as ‘an additional resource conveying the diegetic or extra-diegetic information’. On the contrary, taking them as translational and translatable, Chaume (2004) proposes three methods for the rendition of such graphic codes in different AVT modes (see Table 1).

OPEN IN VIEWER

Table 1.

Methods used in transfer of graphic codes (Chaume, 2004: 21).

Mode	Intertitles	Titles	Texts	Subtitles
Dubbing	(a) Voice-off (b) Subtitled (c) Insertion of new text in target language	(a) Voice-off (b) Subtitled (c) Insertion of new text in target language	(a) Voice-off (b) Subtitled (c) Insertion of new text in target language	Subtitled
Subtitling	Subtitled	Subtitled	Subtitled	Subtitled

Graphic codes, in the form of intralingual subtitling, have been empirically investigated from a number of aspects. Using a Tobii X-120 remote eye tracker, Redmond et al. (2015) find that the participants scan static graphic codes (e.g. shop and street signs, writings on a door) at first glance. The participants also track these kinds of texts that border the frame across its top and right-hand sides, again searching for information. Similarly, in their case study of animated feature film Up (dir. Pete Docter and Bob Peterson, 2009), Batty et al. (2016) report a high degree of attentional synchrony as for graphic codes. Sasamoto (2014) discusses the way viewers interpret programs with graphic codes and the role of these codes in the interpretation process. In her conceptual study, she argues, in line with Pérez-González (2013), that the interpretation process of graphic codes, enabling TV producers to create a sense of mutuality with viewers, depends on the context in which graphic codes are used; for example, in drama these codes facilitate telling the story (Sasamoto, 2014: 10). However, she believes that more research on this topic needs to be undertaken, especially in the area of reception (p. 11), so that the association between interpretation of graphic codes and enhanced viewing experience is better clarified. In a multimodal analytic study, Sasamoto et al. (2017) find that graphic codes affect viewers’ interpretations of a Japanese TV program.

Some studies have also investigated the translational aspect of graphic codes in AVT. Orero (2011) analyses the AD of spoken, tactile and written language in the Singaporean drama film Be with Me (dir. Eric Khoo, 2005). She classifies the film’s graphic codes into several categories as in the following: SMS language, internet chat language, logos on screen, handwritten languages, text written on a computer screen, text written using a manual typewriter, and other texts which appear on screen such as signs, names of the buildings, among others. She stresses the significant role of graphic codes in the film and concludes that far too little attention has been paid to the rendition of graphic codes in AD (p. 243). Exploring the fan-subbing practice in an Iranian context, Khoshsaligheh and Fazeli Haghpanah (2016) provide a few examples of translated graphic codes. Researching The Simpsons, Rybnickova (2016) explores the rendition of graphic codes in dubbing into Czech. She provides an overview of translation strategies used for rendering the graphic codes in the Czech dubbing and compares the strategies over time to see the changes that have taken place: for example, the amount of transferred visual information, the use of subtitling or the combination of verbal audio channel and subtitling. In an attempt to provide answers to the research questions regarding the rendition of graphic codes posed by Chaume (2012: 175), Mehdizadkhani and Khoshsaligheh (2017) categorize the moving graphic codes – pop-up written information on screen – into SMS language, internet searches and the characters’ mood and thoughts in Sherlock. They find that the BBC Persian TV network rendered all of the graphic codes with the insertion method; however, five scenes of graphic codes remained untranslated in the voice-off method used by the Iranian TV network.

3. Method

Experimental research in AVT is a commonly-used methodology. The focus, however, has almost always been on subtitling (e.g. d’Ydewalle and Van de Poel, 1999; Koolstra and Beentjes, 1999; Orrego-Carmona, 2015; Perego, Del Missier et al., 2010; Perego, Laskowska et al., 2016). A few studies have also taken advantage of experimental design in comparing and contrasting dubbing versus subtitling. Wissmath et al. (2009) explored the effects of the two modes with regard to presence, transportation, flow and enjoyment. Perego, Del Missier and Bottiroli (2015) and Matamala et al. (2017) compared subtitling and dubbing in terms of visual and verbal aspects of performance and age in Italian and Spanish contexts, respectively.

Drawing on the precedence of experimental research in AVT, the current study was designed as experimental research with a qualitative supplementary phase. In the preliminary phase, the participants were screened and selected for the experiment. In the second phase, a two-group design with inversion (Shadish et al., 2002) was used to compare the efficiency of the rendition of graphic codes through voice-off and insertion of the new text in the dubbed versions in terms of viewers’ comprehension and information recall. However, as correctly noted by Ross and Morrison (2004: 1023), an important factor in this type of design ‘is the use of pretesting or analysis of prior achievement to establish group equivalence’. So, one viewing session (see pre-check phase) was arranged to make two balanced groups for the experimental phase.

Additionally, given the advantages and disadvantages of both methods for the audience, the study also qualitatively analysed the views and comments of the viewers through in-depth guided-interview technique in the final phase.

3.2. Experimental phase

Within the experimental design, the two equivalent groups, namely groups A and B, were asked to watch two 10-minute clips (i.e. clip 1 and clip 2), selected from the first season of Sherlock, a 2010 British television series created by Paul McGuigan and Euros Lyn, in the two rendition methods (see Table 2). It should be mentioned that the participants reported that they had not previously watched or learned about this TV series. After viewing the two clips in both dubbing methods, the two groups filled out an originally designed test consisting of 18 closed-ended items, aiming at information recall (lexical and number items, n = 8, see the English version of the questions in Appendices 1 and 2) and comprehension (open and true or false questions, see the English version of the questions in Appendices 1 and 2, n = 10) of the content which directly depended on the rendition of graphic codes. It is worth mentioning that the first draft of the questions was subjected to the revision and comments of a few scholars in AVT studies. According to their suggestions, a number of questions were excluded or revised to measure comprehension and recalling information of the TV series better. To achieve face validity, a few of the participants were asked to review the questions to ensure appropriate readability and clarity.

OPEN IN VIEWER

Table 2.

Data collection steps in the experimental phase.

Clip 1	Clip 2
Group A (insertion)	Group B (insertion)
Group B (voice-off)	Group A (voice-off)

The reason for employing such experimental design was that the two variables of information recall and comprehension of the graphic codes were obtained by the sum score of both groups’ viewing in the two methods. In other words, for example, the variable of information recall was obtained by the sum score of group A watching clips 1 and 2 in the insertion and voice-off methods, respectively, and of group B watching clips 1 and 2 in the voice-off and insertion methods, respectively.

4. Results

The first two questions in this study sought to determine whether there was a significant difference between the two methods of voice-off and insertion when used in the Persian dubbed version in terms of the participants’ comprehension as well as their information recall of the film content shown in graphic codes. Based on the results of the pre-check phase, the 30 best achieving participants on the pre-check test (M = 17, SD = 1.14) were invited to the experimental study. The 10 women and 20 men (age range 18–30) were randomly assigned to two groups of 15 (i.e. groups A and B).

4.3. Reflections on translating graphic codes

The third research question sought the participants’ comments and opinions about the rendition of graphic codes. The participants emphasized that, unlike numerous films in which the graphic codes have been left untranslated, they have to be always rendered into the target language in order to avoid compromising the full comprehension of the entire film. They believed that graphic codes are indispensable components of a film and have to be considered as seriously as the dialogues of the characters.

4.3.1. Pros and cons of insertion

The participants believed that the rendition of graphic codes in the insertion method, used by the BBC network, was ‘a new way of translating graphic codes’. However, they mentioned that this method might not be suitable for viewers who have weak eye sight; therefore, this would constrain viewers to wear glasses for reading graphic codes. Probably, in those situations, someone is needed to read the TGCs for them. Moreover, they also noted that since illiterate people prefer dubbed films rather than subtitled ones, inserting the TGCs could ‘bother’ them.

They mentioned two advantages of the method: first, by using the insertion method, the dubbing studio could keep the style of OGCs in dubbing. Second, when there was a close shot on graphic codes and one of the characters of a film was talking with another character at the same time, inserting the TGCs could be the best choice; here, if the graphic codes were released in a voice-off method, the viewers would hear two voices coincidentally or what was known as dialogue interactions.

They also believed that the rendition of graphic codes in the insertion method might be associated with an increase in viewers. Not only could the participants make a connection between themselves and the film director, but they could also find that the dubbing studio cared about them and, far from being distracting, the viewers positively felt this method involved them in the story telling of Sherlock. They also maintained that graphic codes were rendered in an elegant design in this method. They particularly ‘loved’ the technique where graphic codes flash small messages on the screen.

However, a few of the participants found the insertion method ‘annoying’. They mentioned that graphic codes were really distracting from the actual video. They then questioned the use of graphic codes in films as they ruined the realism; they believed that a film had to play by its own rules. They could not understand how graphic codes might float beside a character and the character was blind to them.

4.3.2. Pros and cons of voice-off

A few of the participants believed that the rendition of graphic codes in the voice-off method allowed the dubbing team more freedom in changing stress, tone and accent according to the genre of films. Additionally, they noted that this method could help viewers to get back to the genre of the film. However, some of the participants questioned the situations in which there was a dialogue interaction in films. For example, in one of the scenes of Sherlock, Sherlock Holmes was texting a message and there was a close shot on the graphic codes. At the same time, Dr Watson was talking with Inspector Lestrade. As mentioned before, the voice-off method is the audio-rendition of the TGCs with an actor’s voice-off and inserting the voice-off when the close shot is on the OGCs. As the participants noted, it was obvious that if a dubbing studio were determined to render both graphic codes and dialogue of the actors, viewers would hear three voices at the same time; therefore, the Iranian TV network inevitably dubbed the dialogue of the characters and ignored the graphic codes. Hence, the participants suggested that in such scenes, in which both dialogue of characters and graphic codes were shown at the same time, the dubbing team could dub the dialogue and use the insertion method for rendition of the graphic codes.

A few of the participants suggested that it was also better to delete the TGCs in the voice-off method. They mentioned that the voice-off method might not work well with censored scenes of the graphic codes; for example, imagine situations when some viewers who have the source language knowledge of the film hear the voice-off rendition which is a censored translation of TGCs and see the OGCs at the same time. For example, in one of the scenes of the first episode of Sherlock, ‘A Study in Pink’, Sherlock found that the murdered woman was a ‘serial adulterer’. The phrase is considered to be a taboo phrase in Persian and needed to be deleted, euphemized or replaced with a non-taboo one. Interestingly, whereas the BBC Persian TV network euphemized the phrase with ‘She has many lovers’ (‘mashoughehaye ziadi dashte’), the Iranian TV network used a non-taboo phrase ‘complicated affair’ (‘ertebat ashofte’) with the voice-off narration.

Figure 1 shows the main themes that the interviewees mentioned.

OPEN IN VIEWER

Figure 1.

The main themes obtained from the in-depth interviews on the advantages and constraints of the two methods.

4.3.3. Hybridity?

The participants suggested that the use of both insertion and voice-off methods in the rendition of graphic codes in a dubbed film. For example, a few of them noted that graphic codes need categorizing according to their type. That is to say, the dubbing team suggested the use of different methods of translation for each type of graphic codes. For example, if there are two kinds of graphic codes such as headlines and internet language in a film, the dubbing team could use a different method of translation for each one.

In general, they all believed that if the dubbing team found that graphic codes might play an important role in a film, in addition to making technical issues for them, it is better to use other translators’ comments and suggestions to provide the best choices in translation methods such as voice-off and insertion according to the type of graphic codes and the genre of films which they are screening.

5. Discussion

The results of the quantitative study indicated that the participants comprehended and recalled graphic codes better in the insertion method. A possible explanation for this might be that graphic codes, rendered in the insertion method, remained on the screen for a fair amount of time giving the opportunity of re-reading (see Rayner, 1998; Szarkowska et al., 2013).

To date, no-one has conducted an experimental study to examine these two translation methods in terms of their general comprehension and information recall. However, if the rendition of graphic codes in insertion and voice-off could be considered as ‘normal’ subtitling and dubbing (see Koolstra et al., 2002), respectively, the findings are in line with d’Ydewalle and de Bruycker (2007), Perego, Del Missier et al. (2010), Perego, Del Missier and Bottiroli (2015) and Perego, Laskowska et al. (2016) who also find that the method of subtitling is effective in comprehension. However, in the context of Spanish, Matamala et al. (2017) report no significant difference in terms of comprehension between dubbing and subtitling. Our finding is also of some interest in that Iran is traditionally and predominantly a dubbing country and people are more involved with dubbing. However, this is in line with the results of the qualitative study indicating that a few of the participants who found the insertion method ‘annoying’ may be associated with the idea that people are interested in AVT modes to which they are accustomed (Koolstra et al., 2002). It was also confirmed in the experimental study by Di Giovanni (2012) which shows that the Italian viewers preferred documentaries to be voiced-over rather than simply subtitled because voice-over is the prevalent translation modality for documentaries in Italy and they are more comfortable with voice-over for documentaries. However, in their study, Szarkowska and Laskowska (2015) find that the Polish participants were no longer interested in voice-over and they were showing interest in other modalities such as subtitling. Moreover, the results of the study are in line with Perego, Laskowska et al. (2016: 222) who find that familiarity with AVT modes, particularly subtitling, may affect the way subtitles are seen.

Another theoretically interesting consideration relates to the positive effects of inserted graphic codes on the lexical aspects of performance. These results corroborate previous findings on the beneficial role of subtitles in lexical acquisition (Bisson et al., 2014; Van de Poel and d’Ydewalle, 2001; Kuppens, 2010; Matamala et al., 2017; Perego, Del Missier and Bottiroli, 2015; Sokoli, 2006).

The third research question was posed in order to gain the reflections and feedback of Persian dubbing viewers compared in terms of the use of insertion and voice-off methods in the rendition of graphic codes. Pérez-González (2013) points out that the graphic codes in Sherlock are integral parts of the drama. Although he considered the non-translational function of graphic codes, the results of this study indicated that, even in dubbing, they have the same role in this TV series. Additionally, the participants believed that a priority should be given to graphic codes in the same way as the characters’ dialogue. As they correctly mentioned, if a scene of graphic codes remained untranslated, it would compromise the understanding of a whole film. This finding is in line with the non-translational function of graphic codes in that, as Sasamoto (2014: 9) claims, ‘it would be difficult to follow the thread of the show’.

According to (Bagherzadeh, 2014), illiteracy threatens over nine million people in Iran Obviously, these people prefer to watch dubbed films and TV series. However, by rendering graphic codes in the insertion method, the dubbing studios or TV channels may dissatisfy this class of viewers. Probably, besides technical issues, in order to include this type of viewers, the Iranian TV network used the voice-off method in rendering the graphic codes. Moreover, the rendition of graphic codes in the insertion method could constrain viewers to wearing glasses in order to read graphic codes.

Some of the results vividly contradict each other. On the one hand, the participants mentioned that reading the inserted graphic codes was much more difficult than listening to the voiced-off ones. There could be several reasons to account for this. One of them is the difference between the two skills of reading and listening as reported in the literature (Díaz Cintas and Remael, 2007; Koolstra et al., 2002). Generally speaking, viewers can absorb speech more quickly and more easily than reading. However, Chiaro (2009: 151) points out that, since texts on screen are becoming ‘more and more readable and user-friendly’, this challenge should not be overstated. On the other hand, this supports the finding of the study considering an increase in viewers in the insertion method. Along the same lines, Sasamoto (2014: 9) claims that graphic codes in Sherlock resulted in the programme’s popularity. Similarly, from a creative filmmaking perspective, Pérez González (2013: 14) claims that graphic codes are appropriate for ‘foregrounding spectacularization … or facilitating the process of mutual recognition or affinity between the director … and the viewer’. Similarly, Zhou (2014) advocates the idea of using graphic codes on screen as they are both cheaper and more efficient because no more close-ups are needed. Moreover, he argues that when SMS texts are not shown as written texts on the screen, the ‘shot reverse shot’ (i.e. a film technique in which the character is shown reading the texts on the phone and then the close shot displays the phone on which the character is reading the text) of the character and the phone is slow and visually boring, and the phone has to be onscreen long enough for the audiences to read. In other words, the inserted graphic codes allow a filmmaker to combine action and reaction in the same frame and they give viewers an uninterrupted view of the actor’s performance.

A disadvantage of adding graphic codes as floating texts on the screen instead of showing the close-ups may lead to the transgression of the ‘fourth wall’. The notion of the ‘fourth wall’ as an invisible, imaginary wall which sets the actors apart from the audience, is used both in the case of theatre and television (Auter and Davis, 1991); breaking this imagined wall simply indicates the existence of something unrealistic (i.e. floating texts in the air).

Furthermore, the results of the interviews indicated that the insertion method establishes extra rapport in dubbing: that is, between viewers and the dubbing studio in which the viewers might feel that by inserting the TGCs, the dubbing team have involved them in the story telling of Sherlock. The participants were also surprised by the creative innovation of the insertion method. Dwyer (2015: 115) also claims that the graphic codes in Sherlock are ‘fresh, creative and playful’. Interestingly, the results also showed that the participants were also affected by the elegant use of the insertion method. The font of the TGCs has stayed consistent for each season of the TV series. Moreover, the colour is white instead of different colours for different characters. Although the use of different colours for different characters in the rendition of graphic codes may be distracting for viewers with normal hearing, for d/Deaf and hard of hearing viewers it is actually quite useful as the diversity of colours helps them identify speakers (Neves, 2009).

As mentioned in the results section, one interesting finding was that, if a dubbing studio or TV channel needs to censor scenes that include graphic codes, the insertion method may be more applicable than voice-off as viewers who have some knowledge of English hear the voice-off rendition, which is the censored translation of the TGCs, and see the OGCs at the same time. This result can be analysed against the fact that dubbing has traditionally been used by governments to censor content. This act may also decrease the ‘dubbing effect’ as mentioned by Braun and Orero (2010) and may lead to funny situations as stated by the participants. In contrast, this advantage of insertion in the rendition of censored graphic codes improves the ‘alienation effect’ of the dubbed TV series as noted by Kress and Van Leeuwen (2006). In order to describe this effect for the non-translational function of graphic codes, Pérez-González (2013: 17–18) cites that the use of graphic codes creates ‘an “alienation effect”, to make audiences more aware that they [are] watching a fiction and invite them to reflect on its content’. In contrast, the results of the study indicated that the use of the voice-off method also makes viewers aware that they are watching an animation, a drama or horror film in that it allows the dubbing team more freedom in changing stress, tone and accent.

Furthermore, the concepts of ‘dubbing effect’ and ‘alienation effect’ may be related to visibility and invisibility of the translator (Venuti, 1995) in the rendition of graphic codes, as mentioned by the participants. This finding supports the fact that, when the method of insertion or voice-off increases the ‘dubbing effect’ and therefore the ‘alienation effect’, the film translator or the dubbing studio becomes invisible in the process of dubbing. On the contrary, when the methods decrease these effects – for example, consider the voice-off method where viewers can see the OGCs and hear the censored translation of the TGCs simultaneously – the film translator or the dubbing studio becomes visible in the process of dubbing. This result is of some interest considering that the choice to dub is mainly defended with the argument that viewers cannot access the original content of films or TV programmes.

The findings of this study also showed that the voice-off method may not be applicable when graphic codes interact with film dialogues (Chaume, 2012). Here, the dubbing studio or TV channel inevitably dub the dialogue of the characters and ignore the graphic codes. In contrast, since the dialogue interaction does not affect the insertion method, the viewers can see the TGCs and also hear the dialogues at the same time without any constraints.

6. Conclusion

The goal of this study was twofold. First, it aimed to compare test scores of participants who watched the rendition of graphic codes in voice-off vs those who watched them translated by insertion method, with respect to their comprehension and information recall. Second, the study attempted to collect and analyse the views and comments of avid viewers through in-depth interviews. The findings of the second phase showed that the participants comprehended and recalled the information of the translated graphic codes better when they were in the form of the insertion method rather than a voice-off. However, the findings cannot be widely generalized since the study is limited by a relatively small number of participants and a convenience sampling technique. Despite the limitations, the study assumes significance as it is a unique mixed-methods study that has explored the translational function of graphic codes in AVT.

With regard to the results on the comments and feedback of the Persian dubbing viewers in terms of the use of insertion and voice-off methods in the rendition of graphic codes, the findings revealed that graphic codes have a highly significant role in an accurate understanding of the course of a dubbed film. Moreover, since these two methods have both advantages and disadvantages for the audience of dubbed films, it is concluded that, in order to increase the ‘dubbing effect’ and therefore their invisibility in the process of dubbing, translators and dubbing studios need to adopt the most applicable method with respect to the genre of films, dialogue interaction, different types of graphic codes (e.g. SMS texts, internet language, emails, etc.), target language cultural references, illiteracy and vision of the viewers.

There is abundant room for further progress in determining whether comprehension and information recall of graphic codes in the insertion and voice-off methods are affected by variables of gender, level of education and age. Additionally, having highlighted the insertion and voice-off techniques as the two different methods in the rendition of graphic codes in dubbing, further research is recommended using eye-tracking devices in exploring these techniques.

Finally, it is hoped that this preliminary inquiry using an experimental design and supplementary interview data into the rendition of graphic codes in dubbing kindles more interest in investigating this phenomenon in relation to other relevant variables and using more generalizable findings.