Capturing Image–Text Modal Relations in Mobile Instant Messaging Through WhatsApp Diaries

Abstract

Polymedia chat applications such as WhatsApp are facilitating ‘mixed-media’ relationships (Parks, 2017, Communication Research, 4, 505) as they have penetrated our everyday mediated interactions. This has led to calls for a deeper probe into the interconnections of the varied modes. After defining its vital term ‘mode’, the current study focuses on two widely used modes—image and text—in everyday mediated interactions via chat applications. In order to study their interconnections, the study adopts Martinec’s (2005) image–text relation systems. The findings indicate two extreme range points of image–text interconnections in mediated chat app-based interactions. At one extreme point, images and texts seem to be repeating each other’s messages, and at another point, they seem to complement one another. The point where images and texts complement one another highlights the role of images, and the point seems to play evocative and facilitative roles.

Keywords

Contemporary communication cultural studies digital media technoculture

Introduction

In 2010, when smartphones were still evolving into their current form, Schroeder (2010) proposed the concept of ‘multimodal connectedness’, noting that the usage of mobile phones is not in isolation from other information and communication technologies (ICTs). Years before Schroeder (2010) proposed this concept, theories such as channel complementary theory (CCT) (Dutta-Bergman, 2004), media multiplexity theory (MMT; Haythornthwaite, 2005) and modality switching had already hinted towards the interconnections between and among various modes and media.

The chat applications of modern smartphone devices are highly multimodal. A typical chat app offers affordances to create multiple message forms ranging from text messages to moving and still images, audio clips, various documents and more. The everyday usage of varied modes has given rise to what Parks (2017) calls ‘mixed-media relationships’. Our daily mediated chat interactions are a continuous stream of text messages, images, audio and video. As scholars are now increasingly calling for a deeper probe into the interconnections among various modes and media used together (Baym, 2009; Caughlin & Wang, 2019; Parks, 2017), polymedia devices and, by extension, chat applications, as argued in the next section, offer opportunities to explore these interconnections more closely.

The earlier studies on multimodality, such as CCT (Dutta-Bergman, 2004), MMT (Haythornthwaite, 2005), and modality switching, have primarily categorised modes into various media tools (old and new) and face-to-face (ftf) interaction. These media tools were not often fully unpacked into their primary constituent forms in this categorisation. For example, channel complementary studies worked with channels such as telephones and the internet. Media multiplexity studies (Haythornthwaite, 2005) use media tools: email, fax, phone call and video call. However, in the face of greater integration of these new media tools afforded by the current smartphone device and their chat applications, there is a need to look at their interconnections in their more elemental forms, namely images and texts. Parks (2017) also sharpens the focus on the primary forms of modes in mediated interactions by defining modes as ‘basic message form’.

So, the current study primarily extends this work in two ways. First, instead of focusing on media tools, it focuses on message forms and defines modes in their more elemental forms, namely images and texts. Second, unlike the previous studies, which have confirmed mainly that media tools are used together but have not explored how they interconnect and relate to one another (Parks, 2017), the present study looks at modal interconnections by exploring the relations between two primary message forms—text and images—and how users bring them together in their daily mediated interactions. Even many studies on texting and images have remained mainly independent (Jones, 2005, p. 1). As Jones (2005) points out, ‘questions surrounding how people choose and use these different modes strategically in the course of real-time interaction, however, have yet to be fully addressed’. Apart from text, the selection of images matters in the usage of chat applications such as WhatsApp, as images (moving or still) form a large part of messages being sent (Business Today, n.d.).

This brings us to the larger question the article aims to answer: Are the continuous streams of texts and images interconnected? If yes, then in what manner and how can such interconnectedness develop our understanding of the concept of multimodal interconnectedness and mixed-media relationships?

It adopts the image–text relations systems proposed by Martinec (2005) to explore this interconnectedness. It also addresses the ethical challenges associated with WhatsApp chat content by using what is called ‘Democratic Research Process’ (DRP). The findings indicate a range within which images and texts interconnect in our chat streams. At one extreme, the two are entirely independent and do not affect one another at all, and at another, the two are highly interdependent, so much so that images seem to shape texts. At this range point of image–text integration, images seem to play two dominant roles—evocative and facilitative. In their evocative role, images trigger emotional responses expressed through texts. The data captured five types of evocations—happiness, romantic passion, fond memories of shared past and feelings of recent or current lived experiences. Apart from evoking an emotional response, images also elicit textual responses, which are task-oriented and do not appear evocative. These images facilitate tasks primarily in the following manner—to arrive at a consensus or decision by making a selection to extend support or help.

Chap Application and Its Integrated Polymedia Environment

The theory of polymedia treats a media environment as integrated, where users use multiple features on offer to fulfil their needs (Madianou, 2020). While Madianou (2020) strictly talked in terms of smartphone device as a whole, interestingly, a typical chat app on a smartphone device today offers as many features and functionalities as that of the whole device ranging from capturing and sharing both moving and still images, audio clips, text messages, sharing of documents and more. Thus, while being a part of the device, a chat app still offers a polymedia-integrated environment.

Madianou (2020) calls for a more holistic understanding of media and cultural assemblages of polymedia, internet-enabled, constantly connected smartphone devices. Apart from media and cultural assemblages, this article argues that these devices, through their chat applications, afford spatial assemblage and integration between physical and virtual, which needs to be closely inspected. The polymedia chat app environment facilitates this spatial integration with ubiquitous connectivity, portability and fluidity. The polymedia chat app allows for the capture and movement of elements from the physical space to the virtual space in various forms and modes. Ubiquitous, constant connectivity afforded by polymedia chat apps facilitates more effortless movement of these elements from physical space to virtual, whereas fluidity allows for their easy capture and storage. Thus, the polymedia environment of a chat app with ubiquitous connectivity, portability and fluidity helps create a highly spatially integrated environment.

In order to explore this spatially integrated environment, the current article uses the WhatsApp log diary method. The everyday penetration of popular chat apps like WhatsApp into our daily lives and its ubiquitous, constant connectivity have opened avenues for scrutinising the mundane chats inaccessible earlier. Considering WhatsApp’s deep penetration in India (Diwakar, 2016; Shah & Tewari, 2021; Statista, n.d.) and its possible similarity and closeness with ftf interactions happening in physical space (Ramirez & Broneck, 2009), it can be an essential key to exploring how users bring mediated and non-mediated spaces together. Therefore, the study has used WhatsApp chat streams to study integrated communication. It has penetrated the everyday and mundane and has been found similar to ftf in employing relationship-maintenance strategies (Ramirez & Broneck, 2009).

While WhatsApp is multimodal, past studies have explored the channel in a monomodal manner, focusing exclusively on text mode (König, 2019) or visual mode Gursky et al. (2022). A study does focus on multiple modes but adopts a comparative rather than integrative approach to explore WhatsApp multimodality. It compares whether video mode is more potent than texting for spreading fake news (Sundar et al., 2021). The current study explores whether using varied modes is integrated or isolated. In other words, textual and visual modes are used together in mediated interactions.

WhatsApp Diary: A Tool to Collect Multimodal Text

The mobile device is at the forefront of contemporary media convergence. They facilitate macro-level convergences of technological systems to more micro-level convergences of everyday and mundane, such as television, photography, social networks, news dissemination and consumption (Fortunati & Bakardjieva, 2020). They are carrying forward the march towards digitalisation, which created a new language of a combination of ‘words, images and sounds’ (Fortunati & Bakardjieva, 2020, p. 82). I argue that such multimodality (Martinec, 2005) is facilitated by chat apps such as WhatsApp. A typical chat app on a smartphone device today offers affordances to create multiple message forms ranging from text messages to moving and still images, audio clips, a variety of documents and more.

Additionally, the everyday penetration of popular chat apps like WhatsApp into our daily lives has opened avenues for scrutinising the mundane chats inaccessible earlier. Increasingly, qualitative studies are exploring their ethnographic potential (see Kaufmann & Peil, 2019; Staudacher & Kaiser-Grolimund, 2016). Ramirez & Broneck’s (2009) study on IM use among US university students confirmed its penetration in everyday interactions (p. 308), along with other ‘communication channels’. Their study indicates that ftf and IM are not very dissimilar when it comes to maintaining relations across relation types (p. 310). They point out that using IM is also a common relationship maintenance strategy, such as ‘sharing tasks’ or ‘engaging’ in joint activities, which have been conventionally associated with ftf or physical proximity.

India is allegedly the world’s leader in mobile instant messaging use, with just 36% of smartphone market penetration in 2018 (Statista, n.d.). This kind of embeddedness of the mobile chat application WhatsApp in everyday life in India and its affordance to allow the creation of multimodal text provide a unique opportunity to capture and explore multimodal text and its interconnections. The current study focuses on the opportunities provided by India’s most expansive multimodal chat app WhatsApp and on how users interconnect the two modes, image and text, to hold meaningful interactions and navigate social relations. The current article, therefore, develops and uses the WhatsApp log diary method.

RQ 1: Is the WhatsApp-based diary method suitable for capturing multimodal chat content?

Conceptualising Mode and Adopting Image–Text Relation Systems to Study Modal Interconnections

Modes are forms of meaning-making (Kress, 2009, p. 84). These are the primary forms of message in which meaning is encoded, for example, speech, written text, still image, moving image and touch, and then circulated through a channel (Parks, 2017). While speech and writing have traditionally been the dominant modes of communication, the availability of digital media has facilitated the use of multiple modes such as writing, moving and still images, speech, and so on (pp. 96–97). This has led to a call for a multimodal approach to communication (Kress, 2009, p. 93). Calling ‘multimodal production’ an ‘ubiquitous fact’ of contemporary communication, Kress (2009, p. 102) calls for an urgent need to develop methods for their analysis.

A multimodal approach to communication looks at how meaning is realised when shaped within a multimodal assemblage (Kress, 2009, p. 93). For example, what meanings are created when users bring together images and texts in the multimodal ecosystem of WhatsApp? Martinec’s (2005, p. 2) conceptualisation of ‘multimodal texts’ as a meaningful interaction between text and images to create a more or less coherent whole provides essential directions to the study. According to Martinec (2005, p. 2), multimodal texts are created when text and images interact meaningfully to create a more or less coherent whole. In order to explore this meaningful interaction between modes, they propose an image–text relations system. The present study adopts this framework to closely analyse the interconnections of images and texts in our everyday interactions via chat applications. This framework provides a generalised semantic system of image–text relations that would map out how images and text interact (p. 2). Among the various systems proposed within the image–text relations system, the system of equal status combination has been considered the most suited to studying new media (p. 7). As mentioned in the introduction, the current study aims to bring out the implicit interconnections between the two modes of image and text. For this purpose, it adopts the system of equal status relations and the subsystems of image–text, that is, independent and complementary. Under an equal status relation system, images and texts come together in equal positions to create meaning (p. 7). They can repeat each other’s meaning and be independent, or they can complement each other to make a coherent whole. So, the article aims to answer the question:

RQ 2: How do textual and visual messages via multimodal chat application connect (or do not connect) with one another?

Image–Text Relation Systems: Image–Text Complementary and Image–Text Independent

As mentioned in the previous sections, among the various subsystems proposed by Martinec, the authors find the complementary and independent subsystems of image–text complementary and the most relevant to its study. Martinec (2005, p. 7) also asserts/finds that these subsystems are suitable for studying new media (p. 7). This subsystem is well suited for the current study as it focuses on the influence that text and image exert on one another. The interdependence of images or texts in the WhatsApp message stream helps differentiate whether an image–text relation is independent or complementary (Martinec, 2005, p. 8). According to this subsystem, image and text can be either completely independent of each other or complementary. Images or texts are considered independent relations if they do not modify the other. However, their relationship is complementary when images or texts influence, affect or shape each other. During data analysis, this subsystem was used to bring forth the underlying interconnections or their lack in the descriptions provided by the participants in their usage of texts and images together (for details, see the section ‘Findings’).

While adopting Martinec’s (2005) systems of image and text relations, it is essential to be mindful of the differences in the data. While Martinec (2005) is working with captioned images available in publicly available documents such as advertisements, the current study is working with private WhatsApp chat streams containing descriptions of a stream of texts and images. The captioned images used by Martinec (2005) are meant for public viewings, such as print adverts. The image–text combo in print adverts is self-explanatory. The matter is meant for everyone to see and is therefore created in a manner that people understand without any explanation. However, the image–text combo in a private WhatsApp chat is neither accessible nor meant for public viewing and is, therefore, not always created in a manner everyone understands. A researcher depends on participants to provide valuable comments on their multimodal text, which can help interpret how the image and text used in a WhatsApp chat stream relate to one another. The autoethnographic diary method provided participants with descriptions of their multimodal text and handled ethical issues with private WhatsApp chat content (for details, see the next section).

The descriptive commentaries provided by participants help inspect the image–text relations they have formed while using images and text together in the chat streams. These descriptive commentaries have been used for coding (see thick data in the section ‘Findings’). Participants’ descriptions become a vital element without which the underlying connections between images and texts may not become apparent to the outside researcher. So, while Martinec (2005) uses an image and a caption (typically from an advert), the current study has used the description of the image, such as ‘pic of a suit’, ‘pic of a cake’, along with the description of the text contents in order to establish the underlying connections.

RQ 3: Is the framework of image–text relations systems suitable for exploring the interconnections between textual–visual messaging via WhatsApp?

Addressing the Ethical Issues of Mobile Methods

When a study uses mobile phones for the purpose of data collection, it is categorised as mobile methods. Mobile devices are increasingly becoming popular data collection tools as they allow for in situ data collection (Boase & Humphery, 2018). They bring ‘temporal closeness of the self-report with the phenomenon itself’, which ‘enhances validity’ (Boase & Humphery, 2018, p. 155). The current study also uses WhatsApp chat log diaries, which excavate a chat stream from the original source as closely as possible. However, mobile methods are also considered hyper-intrusive and pose a threat to the privacy of the participants (Boase & Humphery, 2018). Their ethical issues stem from the unrestricted movement of data from participants’ devices to the researcher (Bouwman et al., 2013). There is no filtering or control over data that is transferred as part of data collection.

This ethical issue can be addressed by adopting what is called a ‘DRP’ (Philips & Zavros, 2012, p. 61; Singh Apte & Upadhyay, 2022) that empowers the participants. It essentially means involving participants in the research process. In the current study, the participants were allowed to code their own chat entries. They were asked to keep the chat stream in front and make entries about their chat content. This way, they are at liberty to select the chat content they want to share. In automated mobile data collection methods, the data typically moves unrestricted from a participant’s device to the researcher (Bouwman et al., 2013). However, involving participants in the coding of their own chat content ensures that the participants remain in control of chat content and provide only the coded data that they deem fit.

Furthermore, during the orientation session, participants were informed that their coded chat content would be displayed in the published study and, therefore, they should not report any content that they deemed private. They were also asked not to reveal the identities of their chat partners.

Method

Procedure

Data Collection

Participants were asked to keep their WhatsApp open while making entries in their diaries. They were asked to select any three chat streams of 1 day and code the content of the chats by reporting the topics. Apart from topics, they were also asked to report the media used in the chat stream. A well-structured diary was provided to the participants so that the data were collected uniformly.

Data Analysis

The chat streams recorded in participants’ diaries were moved to an excel file for more accessible coding. The excel row carried one chat stream, and this became the unit of analysis. The coding method of incident to incident (Charmaz, 2006, p. 47) was adopted. This helped in identifying and comparing similar incidents. Image–text relations emerged as an interesting category for further exploration. The image–text relations systems proposed by Martinec (2005) were adopted to analyse this category.

Participants’ Response Rate

Eighty-two participants expressed interest and registered for the study. Out of 82 participants, 63 completed the study; 3 did not have WhatsApp numbers; 4 were unwilling to use computers and 12 did not come for the study. Out of 63 participants, 27 completed all 6 days of the study; 22 completed 5 days; 9 completed 3 to 4 days, and 5 completed only 1 to 2 days.

Participants’ Profile and Sampling Method

The profiling of participants was done based on gender, English language proficiency (as English is their second language), and knowledge of using a computer. There were 20 female participants (1 participant was non-responsive in the profiling survey) and 42 male participants. Four participants reported Gujarati (a vernacular language in India) as their medium of instruction in school, while 58 participants reported having studied in an English-medium school. Fifty-six reported having received formal training in the use of a computer.

Sampling Method

Similar to other qualitative methods, solicited diary methods typically adopt purposive sampling rather than random sampling, which is representative of a population (Hyers, 2018, p. 79). Therefore, the current study adopted a convenience sampling method, and first-year undergraduate students enrolled in the communication class in a Mumbai suburban college were solicited for the study.

Participants’ Motivation

Participants’ motivation and commitment are considered essential factors for the success of a diary study (Alaszewski, 2006; Nezlek, 2012). Incentivising the participants keep them motivated for the study. Monetary incentives and non-monetary incentives, such as extra credit, are common for studies involving students in the west.

This study adopted the incentive of grace attendance in a staggered manner. The researcher believed that a staggered incentive mode might work better in diary-based research methods wherein the participants must provide data/make entries, and participate every day rather than just once. The participants were made to see their earned incentives at the end of each day. While no separate data were collected to monitor the benefits of such staggered-based incentives, the observation that the participants waited, enquired, and checked their grace attendance record every day after making their entries reflects that they were not oblivious to it. Daily incentives for daily diary entries perhaps helped to keep the respondents hooked.

On the last day of the study, a question was added to the structured diary to determine their motivations for participating. The vital motivational factors were identified based on the diary entries using quantitative descriptive analysis. Not surprisingly, grace attendance emerged as the prime motivational factor (29%), followed closely by the newness of the activity (27%). It is noteworthy that enroling students in a research study is not a common practice at the college. Other motivational factors were finding the activity interesting (22%) and fun (18.6%). Other minor motivational factors were curiosity about the study, love for WhatsApp, and following their friends/peers.

Results

The first question aimed to explore whether mimicking the WhatsApp chat stream could be used as a basis to develop a data collection protocol for WhatsApp diaries. The current study uses a chat stream-based framework in which the data collection is based on a diary structure that mimics the chat stream structure of WhatsApp. In the current study, participants were asked to report an entire chat stream of one day with an individual or a group, irrespective of time lags and changes in topics. Thus, the entire chat stream of one day becomes the basis of the data collection protocol. Mimicking a chat stream as the basis for developing a diary protocol has several benefits. First, it keeps the collected data closest to how interactions occur in the natural online world. Second, it allows for collecting multiple media-based chats, which is now common for all online interactions. Third, this is how data is stored in applications, making it easy for participants to make entries. As the below findings indicate, participants in their entries provided rich details of which mode was used for which purposes.

The second question aimed to explore how textual and visual messages via multimodal chat applications connect (or do not) with one another. The findings reveal two extreme range points of image–text interconnections in mediated chat app-based interactions. At one extreme point, images and texts seem to be repeating each other’s messages, and at another point, they seem to complement one another. The point where images and texts complement one another highlights the role of images, and they seem to play evocative and facilitative roles.

Images play an evocative role by triggering emotional responses expressed through texts. The data captured five types of evocations—happiness, romantic passion, fond memories of shared past and feelings of recent or current lived experiences (see below for thick data). The role of emotions in interpersonal relationships is well established (Knapp & Daly, 2002), and they are known to be influential in all stages of relationships—initiation, maintenance and termination (Knapp & Daly, 2002).

Image–Text Complement

Evocative Role of Images

As mentioned earlier, images that trigger evocative responses seem to have an evocative quality.

Following are the instances from the data that reveal various evocative acts.

An instance of an image evoking happiness and fun:

‘As I made cake yesterday, I shared a picture of that with her, to which she said, “It seems to be yummy.”

Today is her “Birthday” I said in a funny manner (sic) that “Yaya, this cake is only for your birthday, do not worry.”’

Analytic text: In the above excerpt, the image of a cake was shared, and fun textual exchanges followed this image posting. As the picture is the centrepiece of the textual exchanges, that is, all the fun textual exchange is around the picture of the cake, it can be concluded that it is the image that evokes the emotion of happiness and fun.

An instance of an image evoking romantic passion:

‘He shared picture of call log he also shared picture of his currrnt crush he was telling me how much he liked his new crush he described me how his crush looked’.

Analytic text: In the above excerpt, an image of an individual is categorised as ‘crush’, indicating that the chatter’s romantic interest is the person in the picture. The text messages around ‘how much he likes his new crush’ and a description of ‘how his crush is looking’ indicate the centredness of the image, surrounded by passionate textual messaging around it.

Instances of images become a trigger for talking about the shared past:

In the below instances, images trigger talking about the shared past and possible bonding over it.

‘The fun we had on our friend’s last birthday all those memories. And some PJ’s Nothing just memes and sending photos of friend’s last birthday.’

We are starting chats after a long time and we are discussing (sic) about the use of like app and remembering our memories of junior college, discuss about our parents and planning about the movie and hang out. The all media is about our pics of junior college classroom And snaps and canteen pics and the last pic is for good night greeting.

‘Picture is about our memories. The joke on one of our past friends Shared memories A lots of memories’

Analytic text: In the above excerpted images, some images evoke and become a trigger for eliciting fond past shared memories (see the above instances) and sharing of life experiences (see ‘slice of life’ instances). Again, while the images act as triggers/sources, it is through texting that the provocative act of reliving past shared memories or sharing life experiences is carried out.

Instances of images become a trigger for talking about lived experiences and slices of life:

In the below instances, images become a trigger for talking about lived experiences of the recent past or ongoing ones and possibly bonding over them.

‘Good morning messages, normal text, pics of class because she wanted to see my classroom; reason she wanted to see was that she stays in Surat, talking about lecture then we were talking about my classes, test. Then ask about the whole day what she did, normal routine like morning breakfast then tp then lunch then tuitions’.

‘I told him about function I attended Shared pictures which I clicked Talking about food and the dishes…Media exchanged was of food & various places which my friend visited at Lonavala’.

‘Pics hotel where we are staying at’

‘I saw her status she updated me that she went to DY Patil ground for the concert of Eminay. So the chat was mainly about this topic only how she got a concert pass and at what time did it started and ended. Did she enjoy being there? Was there a crowd’ (abridged)

…another pic was of a person from a group doing a facial for a party, which became the reason for the conversation

Analytic text: In the above excerpts, we see the textual interactions around images that have captured activities, events and slices of daily routine. In the last excerpt, we see the visual from a status that has triggered the conversation.

Facilitative Role of Images

Unlike the first category, images elicit task-oriented textual responses that do not appear evocative. Such images seem to facilitate tasks primarily in the following manner—to arrive at a consensus or decision by selecting and extending support or help.

In the below instances, images are used to provide support and make decisions or selections.

Instances of images as facilitators for arriving at a consensus or decision by making a selection:

‘The media exchange was about the snooker table, and we have to decide which table to select’.

Analytic text: The images of tables help the selection process.

‘The media content was about the photo of a gun to purchase in PUBG game and to contribute the money. The topic was to play PUBG in a group and to create room in PUBG to have fun, customize the game, purchase the pro-gun, and have fun with friends’.

Analytic text: The images of guns facilitate the selection of a gun.

‘The media is about the index page of account book And discussing the which topic we are going to study’.

Analytic text: It is the images of the index page that facilitate the selection of topics

Media exchanged ‘Photo of the suit’… ‘Asking how the suit was being made for sister’s wedding… Asking which shoes to wear, Replying that the colour of shoes and belt should match…

Analytic text: The images of the ‘suit’ facilitate the selection.

Instances of images as a facilitator to extend support or help

‘On Saturday, the last message from a parent was, please send me the photo of the Marathi assignment and other projects to be done in the project book.

So the first four messages were a photo of the assignment by the PTA member.

Then she said thank you.’

Image and Text Independent

‘Image and text independent’ occurs when both come together on equal footing and do not modify each other; in other words, they do not affect each other in any way. For instance, the below-given descriptions of participants do not indicate any interaction or conversation around the images or pictures shared. Unlike the previous system of image–text complementary, where the exchange of images and textual interaction either accomplishes a task or evokes emotions, in the below data, the textual messages and images/media exist in parallel, often repeating each other.

‘Australia won the match election dates and playing on turf’ (entry under message exchange)

‘On yesterdays match Ind vs Aus PUBG video and match scoreboard’ (entry under media exchange)

Analytic text: In the above entry, the textual message that Australia won the match was repeated through the image of the match scoreboard.

‘Birthday wishes to friend

Cake cutting pictures, videos & gif’.

Anniversary wishes, social messages, good morning texts, good night texts, etc. (entry under message exchange)

Jainism videos and anniversary wishes. (entry under media exchange)

Woman’s day greetings, good morning messages. (entry under message exchange)

Woman’s day greetings. (entry under media exchange)

Analytic text: Similarly, above the textual birthday wishes, anniversary wishes were repeated in various visual forms.

The third question aimed to explore whether the framework of image–text relations systems is suitable for exploring the interconnections between textual–visual messaging via WhatsApp. As mentioned earlier, the current study adopts the system of image–text relations to gain insight into how images and texts come together in our everyday mediated interactions. The subsystems of image–text independent and image–text complement help to explain the two major types of image–text relations noted in the data. Image and text are entirely independent of each other in the first relation type, and in the second relation type, they seem to complement each other.

These findings concur with the image relations systems of image–text independent and image–text complimentary. The analysed data indicate that texts become a vehicle/carrier for evocative or facilitative acts, while images act as a source or trigger. Images trigger responses that are expressed textually through messaging, and these textual responses seem to define the nature of the image. An image can be considered evocative if it triggers an emotional response; it can be considered facilitative if it triggers the facilitation of task completion. Thus, conforming to the image–text relation system of complementarity, these findings show that images and texts do not always work in isolation but complement each other. The analysed data also point towards the other system of image–text relations, that is, image–text independent, where they work independently and do not modify or influence one another. The below two sections provide details of each type of relation.

Discussion

Multimodality has penetrated our everyday mediated interactions with the broader adoption of multimodal smartphone chat applications such as WhatsApp. Our relations have increasingly become ‛mixed media’ (Parks, 2017), and scholars have made calls to look more closely at the interconnections of the varied modes used in everyday interactions (Baym, 2009; Caughlin & Wang, 2019; Parks, 2017). Prior research on multimodality has defined ‘modes’ very broadly by equating them to multiple media tools. These studies have also assessed the use of these media tools independently. The current article aims to bridge this gap by focusing on ‘modes’ as message forms and looking closely at their interconnections. By narrowly defining its key term ‘mode’, it focuses on the elemental and basic level at which modes interconnect. The current study focuses on two modes—image and text. In order to study their interconnections, the study adopts Martinec’s (2005) image–text relation systems. According to this system, images and texts either complement one another or remain independent of one another. These image–text independent and image–text complement systems helped unravel the two extreme range points of image–text interconnections in mediated chat app-based interactions. At one extreme point, images and texts seem to be repeating each other’s messages, and at another point, they seem to complement one another. The point where images and texts complement one another highlights the role of images. They seem to play evocative and facilitative roles. They evoke textual responses of happiness, romantic passion, fond memories of shared past, feelings of recent or current lived experiences, or slice of life. They also facilitate arriving at consensus or decision-making and extending support or help. These roles also seem to act as cement that interconnects images and texts, as these roles become the common ground upon which images and texts seem to come together.

These findings indicate that modes do not always act independently in a multi-modal environment, such as WhatsApp. They connect in various ways apart from acting independently. The analysed data indicate that texts become a vehicle/carrier for evocative or facilitative acts, while images act as a source or trigger. Images trigger responses that are expressed textually through messaging, and these textual responses seem to define the nature of the image. An image can be considered evocative if it triggers an emotional response; it can be considered facilitative if it triggers the facilitation of task completion. Thus, conforming to the image–text relation system of complementarity, these findings show that images and texts do not always work in isolation but complement each other. The analysed data also point towards the other system of image–text relations, that is, image–text independent, where they work independently and do not modify or influence one another.

Another aspect of the study is using the smartphone chat application WhatsApp for ethnographic methods. The penetration of smartphones in everyday lives and their immense computing powers have ushered in what is being called ‘the computational turn’ in the social sciences (Ørmen & Thorhauge, 2015). An essential feature of a smartphone that enables it to act as a research tool is its ability to generate log data. A log is ‘a collection of event records’ (Chuvakin et al., 2012). Such data collection can be performed by operating systems, servers and user applications (Ørmen & Thorhauge, 2015).

In contrast, the device or an application is in use, various meta-information about its usage, such as the time of sending and receiving messages, various counts of the messages sent and received, and so on, are/can be automatically captured and stored by smartphones and their applications. This provides an opportunity for ‘unobtrusive’ rich data collection. Such an automated manner of data collection has the advantage of being unaffected by various problems commonly associated with respondents during data collection, such as poor recall, biases, and so on.

The current study used the log data features available on WhatsApp. WhatsApp archives all user-generated data and allows users to access past chats sent and received individually and in groups. Asking participants to keep their chat content in front while making diary entries helped participants to overcome recall issues or biases. Further, it allowed them the freedom to select which data from their chat stream they wished to share (Singh Apte & Upadhyay, 2022). This is in contrast to mobile automated data collection methods, where the data directly travels unfiltered from a participant’s device to the researcher (Bouwman et al., 2013).

Conclusion

The current study contributes to the existing literature in two primary ways. First, it makes a methodological contribution by showing that it is possible to collect chat content of a private messaging application in an ethical manner by adopting what is called a ‘DRP’. In this manner of data collection, the participants are in complete control of their chat content. In mobile automated data collection methods, the data directly travels from a participant’s device to the researcher in an unfiltered manner (Bouwman et al., 2013). However, with the use of a diary, participants have complete control of their data, as they have complete freedom to choose which data (in this case, chat stream) they want to share. Keeping their chat streams in front, they are at liberty to select the chat content they want to share.

The second contribution that the article makes is theoretical. It aims to advance our understanding of ‘mixed media relationship’ (Parks, 2017) by showing that the most commonly used modes of our mediated interactions, that is, textual and visual modes interconnect with one another. Understanding their interconnections matters, as textual and visual modes are the most commonly used modes in mediated interactions, and they have been mainly studied in isolation.

The study has the limitation of sample size. As the study is of a qualitative nature, its sample size is small. Future studies can be conducted in the form of surveys with larger and more varied sample sizes to improve the internal and external validity of the study.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The researcher thanks National Communication Association (NCA) for funding the research project. The authors are indebted to Mobile Communication Interest Group (MCIG), International Communication Association (ICA) for supporting the study with ‘emerging scholar grant award of $500’.

References

Alaszewski

(2006). Using diaries for social research. Sage Publications.

Baym

N. K.

(2009). A call for grounding in the face of blurred boundaries. Journal of Computer-Mediated Communication, 14(3), 720–723. 10.1111/j.1083-6101.2009.01461

Boase

, & Humphreys

(2018). READ//Mobile methods: Explorations, innovations, and reflections. Mobile Media & Communication. 10.1177/2050157918764215

Bouwman

, de Reuver

, Heerschap

, & Verkasalo

(2013). READ opportunities and problems with automated data collection via smartphones. Mobile Media & Communication, 1(1), 63–68. 10.1177/2050157912464492

Business Today. (n.d.). WhatsApp users share 55 billion texts, 4.5 billion photos, and 1 billion videos daily. https://www.businesstoday.in/technology/news/story/whatsapp-users-share-texts-photos-videos-daily-85247-2017-07-27

Caughlin

J. P.

, & Wang

(2019). Relationship maintenance in the age of technology. In Ogolsky

B. G.

& Monk

J. K.

(Eds.), Relationship maintenance: Theory, process, and context (pp. 304–322). Cambridge University Press. 10.1017/9781108304320.016

Chuvakin

, Schmidt

, & Phillips

(2012). Logging and log management: The authoritative guide to understanding the concepts surrounding logging and log management. Newnes.

Charmaz

(2006). Constructing grounded theory: A practical guide through qualitative analysis (1st ed.). Sage Publications.

Diwakar

(2016). It’s complicated: The construction of Indian middle-class teens in social media. Journal of Creative Communications, 11(2), 161–182. 10.1177/0973258616644813

10.

Dutta-Bergman

M. J.

(2004). Interpersonal communication after 9/11 via telephone and internet: A theory of channel complementarity. New Media & Society, 6(5), 659–673. 10.1177/146144804047086

11.

Fortunati

, & Bakardjieva

(2020). Mobile convergence. In The Oxford handbook of mobile communication and society. Oxford University Press.

12.

Gursky

, Riedl

M. J.

, Joseff

, & Woolley

(2022). Chat apps and cascade logic: A multi-platform perspective on India, Mexico, and the United States. Social Media + Society, 8(2). 10.1177/20563051221094773

13.

Haythornthwaite

(2005). Social networks and internet connectivity effects. Information, Communication & Society, 8(2), 125–147. 10.1080/13691180500146185

14.

Hyers

L. L.

(2018). Diary methods. Oxford University Press.

15.

Jones

R. H.

(2005). You show me yours, I’ll show you mine: The negotiation of shifts from textual to visual modes in computer-mediated interaction among gay men. Visual Communication, 4(1), 69–92. 10.1177/1470357205048938

16.

Kaufmann

, & Peil

(2019). The mobile instant messaging interview (MIMI): Using WhatsApp to enhance self-reporting and explore media usage in situ. Mobile Media & Communication. 10.1177/2050157919852392

17.

Knapp

M. L.

, & Daly

J. A.

(2002). Handbook of interpersonal communication. Sage Publications.

18.

König

(2019). Sequential patterns in SMS and WhatsApp dialogues: Practices for coordinating actions and managing topics. Discourse & Communication, 13(6), 612–629. 10.1177/1750481319868853

19.

Kress

(2009). Multimodality: A social semiotic approach to contemporary communication. Routledge.

20.

Madianou

(2020). Polymedia and mobile communication. In The Oxford handbook of mobile communication and society. Oxford University Press.

21.

Martinec

(2005). A system for image-text relations in new (and old) media. Visual Communication, 4(3), 337–371. 10.1177/1470357205055928

22.

Nezlek

J. B.

(2012). Diary methods. Sage Publications.

23.

Ørmen

, & Thorhauge

A. M.

(2015). Smartphone log data in a qualitative perspective. Mobile Media & Communication, 3(3), 335–350. 10.1177/2050157914565845

24.

Parks

M. R.

(2017). Embracing the challenges and opportunities of mixed-media relationships. Human Communication Research, 4, 505. 10.1111/hcre.12125

25.

Ramirez

, & Broneck

(2009). ‘IM me’: Instant messaging as relational maintenance and everyday communication. Journal of Social and Personal Relationships, 26(2–3), 291–314. 10.1177/0265407509106719

26.

Schroeder

(2010). Mobile phones and the inexorable advance of multimodal connectedness. New Media & Society, 12(1), 75–90. 10.1177/1461444809355114

27.

Shah

, & Tewari

(2021). Mapping emoji usage among youth. Journal of Creative Communications, 16(1), 113–125. 10.1177/0973258620982541

28.

Singh Apte

, & Upadhyay

A. K.

(2022). Cross-directional hybrid space: A non-locative approach to mobile social media studies. Social Media + Society, 8(2). 10.1177/20563051221098342

29.

Statista. (n.d.). Smartphone penetration in India 2014–2019. https://www.statista.com/statistics/257048/smartphone-user-penetration-in-india/

30.

Staudacher

, & Kaiser-Grolimund

(2016). WhatsApp in ethnographic research: Methodological reflections on new edges of the field. Basel Papers on Political Transformations, 10, 29–40.

31.

Sundar

S. S.

, Molina

M. D.

, & Cho

(2021). Seeing is believing: Is video modality more powerful in spreading fake news via online messaging apps? Journal of Computer-Mediated Communication, 26(6), 301–319. 10.1093/jcmc/zmab010