Ideology through sentiment analysis: A changing perspective on Russia and Islam in NYT

Abstract

This article continues the line of research that combines the paradigm of Critical Discourse Analysis (CDA) with quantitative methods. We propose that Linguistic Inquiry and Word Count (LIWC), a software for automated text analysis widely used in social sciences, can enrich the CDA toolkit. The methodological advantage of LIWC is that its semantic categorization was developed and validated independently, which addresses the concerns about subjectivity. In two case studies we use LIWC to analyze the construction and representation of the ‘Other’ in mass media. Study 1 focuses on the representation of Russia in The New York Times (NYT) before and after its annexation of Crimea; Study 2 analyzes the change in sentiment toward Islam in NYT before and after 9/11. We find that in both cases the change in attitude is driven by an increase in negative emotion words rather than by a decrease in positive words.

Keywords

9/11 and Islam critical discourse analysis mass media positive and negative emotions quantitative methods representation of the ‘Other’Russia’s annexation of Crimea sentiment analysis

Introduction

The past two decades have seen an increase in the number of studies that combine Critical Discourse Analysis (CDA) with quantitative methods for data analysis (Baker, 2006; Baker et al., 2008, 2013; Orpin, 2005; Prentice, 2010). This tendency can be viewed as a reflection of a more general trend within the humanities and social sciences made possible thanks to technological advancements, the availability of large datasets, and tools that can analyze them (Iliev et al., 2015). This article continues the line of research that uses quantitative methods for data analysis to address questions of interest to CDA. Specifically, we propose that the methodological toolkit of CDA can be enriched by Linguistic Inquiry and Word Count (LIWC), an automated text analysis software developed by social psychologists (Pennebaker et al., 2003, 2007a). Since its introduction in the 1990s, LIWC has been extensively used in social sciences (Back et al., 2011; Fornaciari et al., 2014; Pennebaker and King, 1999). In this article we show how it can be applied to the analysis of texts within CDA and thus can complement traditional corpus linguistics methods.

To demonstrate the applicability of LIWC to discourse studies, we chose a classic topic within CDA – the construction and representation of the ‘Other’ in mass media (Le, 2002, 2006 and Macgilchrist, 2011 on Russia; Baker et al., 2013 and Richardson, 2004 on Muslims in the British press; Wodak, 1996 on immigrants; Wodak and Busch, 2004: 112–114 for an overview). In two independent studies we analyze how Russia and Islam are represented in The New York Times (NYT). In accordance with the discourse-historical approach (Reisigl and Wodak, 2009; Wodak, 1996), we take into account the historical context in which the articles were produced and analyze how the representation of Russia and Islam changes diachronically. We are specifically interested in how certain political events trigger the change in attitudes toward these actors.

In Study 1, we analyze how Russia is represented in NYT articles before and after its annexation of Crimea in the spring of 2014. This event triggered an international outcry, accompanied by a series of economic sanctions aimed to punish Russia for its violation of Ukraine’s territorial integrity.¹ It also significantly worsened the relations between the West and Russia, which led many political commentators to describe the resulting situation as a new Cold War, as the following headlines demonstrate: Cold war 2.0: how Russia and the west reheated a historic struggle (The Guardian, 24 October 2016),² The new ideology of the new Cold War (NYT, 1 August 2016),³ and Russians conduct nuclear-bomb survival drills as Cold War heats up (The Wall Street Journal, 25 October 2016).⁴ While Russia’s annexation of Crimea has had profound political consequences, this event and its effect on Russia’s representation in the Western media, have not yet been studied from the CDA perspective.

In Study 2, we analyze the representation of Islam in NYT articles before and after 9/11. The terrorist attacks on 9/11 changed the Western perspective on Islam, and the discourse correlates of this process have been extensively studied from the CDA perspective (cf. Alazzany and Wong, 2014; Hogan, 2006; Jahedi and Abdullah, 2012; Manan, 2009; Salama, 2011). However, while most of the previous studies discuss the representation of Islam in the media in the aftermath of 9/11, our study focuses on the perspective change by analyzing how Islam is represented before and after 9/11 and comparing the two contexts (cf. also Baker et al., 2013).

In both studies we measure the change in attitudes toward the relevant social actors, Russia and Islam, by performing sentiment analysis using LIWC. We specifically compare how negative and positive emotion words are distributed in NYT articles. The comparative analysis allows us not only to detect and quantify the change of attitudes but also to answer the question of whether a negative stance toward a particular social actor is due to (1) a decrease in positive emotion words compared to the norm, (2) an increase in negative emotion words compared to the norm, or (3) both. Thus, on the theoretical level we contribute to a better understanding of the construction and representation of the ‘Other’ in the media and of the effect of salient political events on this process. On the methodological side, our study complements quantitative research within CDA by introducing a novel methodological tool.

This article is structured as follows. In the following section, we provide a background on CDA with special attention to the evolution of its research methods, after which Linguistic Inquiry and Word Count (LIWC), a software for automated text analysis, is introduced and its applications in social sciences are discussed. The next two sections are case studies that analyze the representation of Russia and Islam, respectively, in NYT. Finally, we conclude the article with a summary of the findings and a discussion of its methodological and theoretical implications.

CDA: Assumptions and evolution of research methodology

CDA investigates how power and ideology are interwoven in language use and how power relations are enacted in discursive practices (Fairclough, 2010, 2015; Fairclough and Wodak, 1997; Fowler et al., 1979; Hodge and Kress, 1993; Van Dijk, 1993; Wodak and Meyer, 2009). CDA encompasses a variety of approaches that draw on social semiotics (Hodge and Kress, 1993; Van Leeuwen, 1995), neo-Marxist theory (Fairclough, 2010, 2015), critical theory (Reisigl and Wodak, 2009; Wodak and Meyer, 2009), and socio-cognitive theory (Van Dijk, 2001). The common perspective shared by CDA scholars is that the critical study of language should be applied to raise readers’ awareness of implicit ideological messages encoded in language use.

Central to CDA is the concept of ideology, which is intrinsically linked to the discursive construction of common sense or ‘naturalization’ of particular worldviews through language (Fowler, 1991, 1996; Van Leeuwen, 1995; Wodak, 2006). Fairclough (2015: 32) discusses ideology as representations of the world that sustain social and power relations and the institutions in which they operate. For Van Dijk (1995: 17–33), ideologies constitute a system of mental representations that organize sets of beliefs into social attitudes that are enacted in communicative events. Since powerful groups can access and control knowledge production, their ideologies usually dominate public discourse. As such, ideologies of the privileged become ingrained in habitual social interaction and language use, and thus taken for granted as ‘natural’ or ‘common sense’ knowledge. CDA scholars have demonstrated how hegemonic ideologies are transmitted in society by showing the ways in which ideological views are ‘naturalized’ through particular linguistic forms (Fowler, 1991; Hodge and Kress, 1993; Van Dijk, 1989, 1991).

From its inception, one of the main domains for CDA studies has been media discourse. Drawing on qualitative analyses of selected texts, CDA researchers have demonstrated how certain linguistic features such as voice and nominalization encode implicit ideological messages (Fairclough, 2015; Hodge and Kress, 1993; Van Dijk, 1995; Van Leeuwen, 1995). These authors have argued that the media plays a key role in sustaining social hierarchies, as groups characterized by ethnicity, gender or class are represented in ways that perpetuate their positions in society (Fairclough, 1995; Fowler, 1991; Van Dijk, 1987, 1989, 1991; Wodak, 1996). These studies offered important insights into the ways in which attitudes and prejudices toward different social groups are shaped into being by the language choice in the media.

While early CDA studies have contributed significantly to our understanding of how ideology and power relations are exercised in mass media discourse, the methodology used in this work was criticized in the 1990s. Much of the criticism pertains to the corpus size, criteria for data selection, and analytical rigor. For example, Stubbs (1997) observes that analyses based on small data samples cannot be replicated by other researchers (cf. also Fowler, 1996: 3–14). Regarding data selection, Widdowson (1995), Stubbs (1997), and Koller and Mautner (2004) draw attention to CDA researchers’ lack of systematicity in selecting texts and subjectivity in their choice of textual features for the analysis. A common criticism voiced in the literature is that analysts often transition directly from description to interpretation without explaining why some features are chosen instead of others (Breeze, 2011), or that interpretations are derived subjectively instead of being based on systematic examination of linguistic data (Orpin, 2005; Prentice, 2010; Stubbs, 1997; Verschueren, 2001). With respect to analytical rigor, early CDA studies are often criticized for the absence of the quantitative component. Without statistical analysis, it is difficult to evaluate the findings or to assess their generalizability and validity (cf. Wodak and Meyer, 2009). One of the most salient features of early CDA studies is the absence of control or reference corpus that provides ‘comparison with norms in the language’ (Stubbs, 1997: 107). A control corpus serves as a baseline against which the texts selected for data analysis are evaluated, and is crucial for assessing the findings.

Some of the issues discussed so far have been addressed in corpus-assisted discourse studies, which incorporated corpus linguistics methodology and (semi)automated text analysis techniques (Baker, 2006; Baker et al., 2008; Krishnamurty, 1996; Prentice et al., 2012; Taylor, 2014). For example, Krishnamurty’s (1996) study of the near-synonyms ethnic, racial, and tribal demonstrated the value of combining qualitative analysis of small text samples with automated analysis of large corpora. Comparing frequent collocations for each adjective, the author found a host of negative semantic associations (e.g. violence, killings, discrimination) commonly used to characterize particular nationalities (e.g. Africans, Eastern Europeans). This co-occurrence of nationalities with lexical items that express negative emotion revealed how minority groups were construed from Eurocentric notions of the ‘Other’. Krishnamurty’s (1996) study showed the importance of comparing language found in a specialized corpus (news articles) with control corpora (dictionary entries and COBUILD’s Bank of English) to investigate how ethnocentric views are reflected in language use.

More recently, Prentice (2010) demonstrated how an automated corpus analysis combined with a CDA framework can address the questions of subjectivity and reliability raised by the critics of early CDA. Working within the discourse-historical framework, the author applied automated semantic tagging to identify key themes, discursive strategies and their linguistic realizations in texts that address the Scottish Independence debate. Prentice (2010) compared a 3.3 million word corpus compiled from an online pro-independence discussion forum with a reference corpus of 7.9 million words containing British newspaper articles on Scottish Independence. Using the UCREL Semantic Analysis System, the author obtained a list of ‘key semantic tags’ (concepts and themes) for the specialized corpus, which allowed her to identify the discursive and argumentative strategies used by forum participants in their construction of national identity. Prentice (2010) showed that nationalist Scots articulated a cultural identity based on outgroup differences from the English and in-group similarities with all Scots. Importantly, her analysis served to demystify stereotypes of Scottish nationalists as hateful toward the English.

The application of corpus linguistics to CDA research has proven highly effective in addressing some of the criticism directed at early CDA studies (Baker, 2006; Hunston, 2002; Koller and Mautner, 2004; Orpin, 2005; Prentice, 2010; Stubbs and Gerbig, 1993). Yet corpus linguistics methods have their limitations. One such limitation pertains to the fact that it is difficult to completely eliminate the subjective component from the analysis. Prentice (2010) observes that some CDA studies that employ corpus linguistics methods ‘can be criticized on the grounds of their subjective application of a coding system that has been created to fit the data under investigation’ (pp. 407–408). Take, for example, the analysis of stance. In order to establish whether a specific social actor or group is represented in a positive or negative light, the analyst needs to decide which words to classify as positive and which words as negative. But this classification is often based on a subjective decision. For example, Jahedi and Abdullah’s (2012) study on the representation of Iran in NYT articles following the 9/11 attacks showed that Iran often occurs in contexts in which the words for violence, extremism, threat, and terror dominate the discourse (see Jahedi and Abdullah, 2012: 66, for more details). The authors concluded that these discursive strategies serve to represent the Iranian state as ‘the Other’. While Jahedi and Abdullah’s (2012) study contributed to our understanding of the effects of 9/11, their analysis was based on a subjective evaluation of contexts and thus can be criticized for the lack of objectivity. Objective measures are needed to ensure that (1) all relevant words for positive/negative emotions are taken into consideration and the valence of the text is established based on their combined effects, and (2) words that are not truly positive/negative are not included in the analysis. By using independently developed measures, such as the semantic tagger in Prentice (2010), we can begin to address this issue. LIWC, the software used in this article to analyze sentiment, is a step in this direction. LIWC semantic categories, including positive and negative emotions, meet the objectivity criterion since they have been developed independently and validated in a number of studies. In what follows, first we summarize the properties of LIWC, and then present the two case studies on the representation of Russia and Islam in NYT.

Linguistic Inquiry and Word Count

LIWC is an automated text analysis software developed by social psychologists as a tool for quantitative text analysis (Pennebaker et al., 2007a; Tausczik and Pennebaker, 2010). Since its introduction, it has been applied in a wide range of contexts in the humanities and social sciences, ranging from studies on how language relates to gender and age (Pennebaker and King, 1999) to detecting deception in Yelp reviews (Li et al., 2014). The fundamental idea behind LIWC is that linguistic choices can reveal speakers’ attitudes, emotional states, personality traits, and the way people perceive social reality. In this respect, the underlying assumptions of LIWC are similar to those of CDA, which makes its application to CDA research only natural.

The software has two main components: (1) a word processor that is an executable file used for word count and semantic tagging, and (2) a default built-in dictionary of words (agony) and lemmas (enrag*). A single file or multiple files can be accessed by the processor which scans the text(s) in search of words (target words) that match the dictionary entries (dictionary words). The matches are analyzed along several dimensions, including their syntactic information, such as part of speech, and are assigned to semantic categories, such as positive and negative emotions, cognitive processes, and biological processes. For example, the word laughed would be classified as belonging to the categories happiness, positive emotion, affect, verb, and past tense. The 2007 version has a total of 64 categories.

Similar to other automated text analysis software, such as the semantic tagger used in Prentice’s (2010) study, LIWC is an objective tool of text analysis. Semantic categories were created a priori, independently of the goal of the current research, and were validated in a number of studies. During the early stage of its development, between 1992 and 1994, an initial list of words for each category was compiled and evaluated by human judges, who were asked to decide on the adequacy of the words selected for each cat-egory. The process was repeated until a level of 93–100% agreement was reached (Pennebaker and King, 1999; Pennebaker et al., 2007b).

One of the widely used features of LIWC is its ability to detect words for positive and negative emotions in texts, which is a foundation of sentiment analysis. The analysis of emotion words can be used to identify people’s reaction toward a traumatic event (Back et al., 2011), to reconstruct a politician’s profile (Tumasjan et al., 2010), to study the relation between age and language (Pennebaker and Stone, 2003), and to identify deception (Li et al., 2014; Newman et al., 2003). For example, Back et al. (2011) used LIWC to analyze the content of 422,502 pager messages on 11 September 2011. The analysis of emotive words based on LIWC categories showed that sadness was not the primary emotion and that there was an increase in anger. In another study, Tumasjan et al. (2010) used LIWC to analyze political sentiment on Twitter before the federal elections in Germany in 2009. For each leading candidate they created a multidimensional profile based on LIWC categories. They analyzed over 100,000 tweets and found that overall positive emotions were more frequent than negative emotions. The only deviations from this pattern were observed for the leaders of the liberal and the socialist party. The authors concluded that ‘the sentiment embedded in tweets does reflect nuanced differences between the politicians’ (Tumasjan et al., 2010: 182). In another study, Pennebaker and Stone (2003) showed how LIWC can be used to study language and age. The authors analyzed writing samples from over 3000 individuals between 8 and 85 years of age and discovered that changes in lexicon are related to aging. Among other features, they found that the use of negative affect words and self-references decreased with aging, which breaks stereotypes about the elderly as being lonely, bitter, and socially withdrawn.

Sentiment analysis can also be used to detect deception. For example, Newman et al. (2003) showed that LIWC can correctly distinguish between liars and truth-tellers. They found that liars use more negative emotion words compared to those who tell the truth. This effect can be explained by the fact that guilt and tension give rise to negative emotions that are then expressed in language. In addition, liars’ stories are usually less complex and are characterized by a decrease in self-reference as well as reference to others, linguistic features for which LIWC has dedicated categories. Interestingly, in a comprehensive study on deception in reviews, Li et al. (2014) found that deceptive reviews have a higher number of sentiment words and that this holds for both positive and negative fake reviews.

As this discussion shows, LIWC has been used extensively by social scientists. We believe that studies within CDA can benefit from its application as well.⁵ LIWC has a number of advantages over more traditional corpus linguistics methods. First, it can pick up positive/negative emotion words that are not adjacent to the target word or node. Second, it allows for an analysis of large datasets, thus addressing the concern about sample sizes. Finally, it addresses the subjectivity concern raised in the previous work. In what follows, we present the results of two case studies that use LIWC to analyze sentiments toward Russia and Islam in NYT articles.

Study 1: Sentiment toward Russia in NYT before and after its annexation of Crimea

Method

In this study we were interested in whether the perception of Russia changed in the aftermath of its annexation of Crimea. To address this question, we searched the NYT database for all articles that contain the word Russia. We collected articles published 3 months before Russia’s annexation of Crimea, from 18 December 2013 to 18 March 2014, and compared them to the articles published 3 months after this event, from 21 March 2014 to 21 June 2014. We refer to these two corpora as the Russia Before corpus and the Russia After corpus, respectively. The Russia Before corpus consisted of 1208 articles, and the Russia After corpus consisted of 808 articles. Following the standard practice in social sciences, we also assembled two control corpora. For the control corpora we randomly selected articles in the NYT database that did not contain the word Russia and that were published within the same time frames as the articles in the Russia Before and the Russia After corpora. The control corpora were matched for the number of articles with the Russia corpora. Table 1 summarizes the data.

Table 1.

Data summary for Study 1.

Corpus name	Search criteria for articles	Publication period	Number of articles
Russia Before	Contain the word Russia	18 December 2013 to 18 March 2014	1208
Russia After	Contain the word Russia	21 March 2014 to 21 June 2014	808
Control Before	Do not contain the word Russia	18 December 2013 to 18 March 2014	1208
Control After	Do not contain the word Russia	21 March 2014 to 21 June 2014	808

Design and procedure

All articles were downloaded in batches of 50. The Russia Before corpus and the Control Before corpus consisted of 24 files each, and the Russia After corpus and Control After corpus consisted of 16 files each. These files were used as input to the LIWC 2007 software. The program returned the proportion of words in each of the LIWC categories. Our analysis focused on the positive and negative affect categories.

Results and discussion

The comparison of the Russia Before and the Russia After corpora showed that there was a change in sentiment. Two trends were observed. First, there was a decrease in positivity following Russia’s annexation of Crimea. There were more words for positive emotions in the Russia Before corpus (M = 2.38, standard deviation (SD) = 0.21, N = 24) compared to the Russia After corpus (M = 2.25, SD = 0.15, N = 16). The differences were statistically reliable according to a two-sample t-test (t(38) = 2.12, p = 0.039, two-tailed). Second, there was an increase in negativity. The articles published after Russia’s annexation of Crimea had more negative emotion words (M = 1.84, SD = 0.16, N = 16) compared to the articles published before this event (M = 1.66, SD = 0.17, N = 24). The differences were statistically significant (t(38) = 3.3, p = 0.002, two-tailed). While these measures allowed us to establish how positive and negative emotion words change over time, they did not provide information about the frequency of positive words compared to the frequency of negative words in texts. To check what kinds of emotions prevail, that is, whether the overall valence of the texts is positive or negative, we created a measure of valence by subtracting negative means from positive means. Positive numbers indicate that positive emotions prevail, and negative numbers indicate that the overall valence of the text is negative. Our results showed that in the Russia Before corpus there were more positive emotion words than negative emotion words. The same holds for the Russia After corpus. Importantly, however, the articles published after Russia’s annexation of Crimea, while positive overall, had a decreased positivity; that is, the differences between positive and negative emotions indices were smaller (M = 0.41, SD = 0.27, N = 16) compared to the articles published before this event (M = 0.72, SD = 0.31, N = 24). The results of an unpaired t-test showed that the differences were statistically reliable (t(38) = 3.27, p = 0.002, two-tailed).

Can we attribute the observed change in sentiment to the changing attitude toward Russia in the aftermath of its annexation of Crimea? To answer this question, we need to rule out the alternative explanation, according to which the observed change simply reflects a general trend in language and is not restricted to articles that mention Russia. To address this issue, we analyzed the distribution of positive and negative emotion words in our control corpora. The analysis of positive words showed that there was a decrease in positivity in neutral texts: there were more positive words in the Control Before corpus (M = 2.4, SD = 0.2, N = 24) compared to the Control After corpus (M = 2.28, SD = 0.17, N = 16). The differences between the two corpora were statistically significant (t(38) = 2.07, p = 0.05). There was also a slight increase in negativity in the articles published after Russia’s annexation of Crimea: the Control Before corpus had fewer negative words (M = 1.4, SD = 0.20, N = 24) compared to the Control After corpus (M = 1.45, SD = 0.23, N = 16). In this case, however, the differences between the two corpora were not statistically significant (t(38) = 0.60, p = 0.55, two-tailed). Finally, we looked at how the overall text valence changed in the control corpora. As previously discussed, this measure was derived by subtracting negative emotion means from positive emotion means. We found that in both corpora, Control Before and Control After, the frequency of positive words was higher than the frequency of negative words. However, the two corpora differed in that there was a decrease in positivity: the Control Before corpus overall was more positive (M = 1, SD = 0.30, N = 24) compared to the Control After corpus (M = 0.83, SD = 0.26, N = 16). The differences are marginally significant (t(38) = 1.89, p = 0.07, two-tailed). The results of these analyses are shown in Figure 1.

Figure 1.

Sentiment in NYT articles that mention Russia (upper panel) and neutral NYT articles (lower panel). After Russia’s annexation of Crimea, there is a significant increase in negativity in the Russia corpus but not in the control corpus.

The comparison of the articles that mention Russia to neutral articles published during the same time period demonstrated that both corpora showed a decrease in positivity. Therefore, this effect was not associated with the mention of Russia in the target corpora but was a reflection of a more general trend in language. However, the increase in negativity observed in our Russia After corpus was not found in the corresponding general corpus, and thus could not be explained as a reflection of a more general trend. Therefore, we can conclude that the observed increase in negativity in the articles that mention Russia reflects a change in attitudes toward Russia in the aftermath of its annexation of Crimea.

An important finding that emerges from this study is that the change of sentiment toward Russia is driven by the increase in negativity as opposed to the decrease in positivity. To check whether the same effect is observed in other contexts, we conducted a second study in which we analyzed the representation of Islam in NYT articles before and after 9/11.

Study 2: Sentiment toward Islam in NYT before and after 9/11

In this study we were interested in whether the results from Study 1 can be generalized to other cases. Specifically, is the change of perception toward a particular social actor in the aftermath of a salient negative event driven by an increase in negativity? To address this question, we analyzed the representation of Islam in NYT before and after 9/11. Before we discuss the design, it is important to compare our methodology to that employed by Baker et al. (2013) in their study on the representation of Islam in the British press. While both studies employ a diachronic approach and perform quantitative data analysis, there are important differences between the two. First, there is a difference in the granularity of the temporal frames analyzed. Baker et al.’s (2013) study spans a larger period of time, from 1998 to 2009.⁶ Each year in their study is treated as a discrete unit of analysis. Therefore, their approach allows for an analysis of a change over a longer period of time. Since we were interested in an immediate effect of 9/11 on discourse, we focused on a narrower window and compared the representation of Islam 3 months before and 3 months after 9/11.⁷ Second, in their study, Baker et al. (2013) performed keyness analysis for each year (top 100 words that have the greatest log-likelihoods), as well as the change of topics. The analysis revealed that top keywords often referred to conflicts that involved Muslims and that the proportion of conflict tokens in 2001 was above average, which can be explained as an effect of 9/11. Our approach is different in that we analyze the distribution of emotion words that are topic-independent, the assumption being that texts that do not overtly mention emotionally charged issues or events still express positive or negative emotion. Finally, the keywords in Baker et al.’s (2013) study that are closest to the categories of positive and negative emotion words were Hate and tolerance keywords, such as alienate, hate, tolerant, and multiculturalism. The authors observed that between 2001 and 2003 only a few of these keywords could be found in the corpus and that their frequency increased in the following years. Baker et al.’s (2013) Hate and tolerance keywords collapse both positive and negative emotion words. Our study is different in that we are interested in comparing and contrasting the words for positive and negative emotions.

Method

For the purpose of this study, we searched the NYT database for articles that mention the words Islam, Islamic, or Muslim. Since we were interested in whether a specific event, the terrorist attacks on 9/11, affected how Islam was represented in mass media, we constructed two corpora. The Islam Before corpus consisted of all articles that mention any of the target words and were published 3 months before the September 11 attacks, from 10 June 2001 to 10 September 2001. This corpus consisted of 384 articles. The Islam After corpus consisted of the articles that mention any of the target words and were published 3 months after the terrorist attacks, from 12 September 2001 to 12 December 2001. This corpus consisted of 1684 articles. These numbers indicate that there was a fourfold increase in the number of articles that mention Islam after 9/11. This tendency is not unique to NYT and has also been observed in the British press (Baker et al., 2013).

As in the previous study, we also constructed two control corpora. These corpora consisted of NYT articles that do not mention Islam, Islamic, or Muslim and that were published within the same time frames as the two Islam corpora. The articles in the control corpora were randomly selected from the NYT database. As in the previous study, we matched the number of articles in the respective Islam corpora and the control corpora. Table 2 summarizes the data.

Table 2.

Data summary for Study 2.

Corpus name	Search criteria for articles	Publication period	Number of articles
Islam Before	Contain Islam words	10 June 2001 to 10 September 2001	384
Islam After	Contain Islam words	12 September 2001 to 12 December 2001	1684
Control Before	Do not contain Islam words	10 June 2001 to 10 September 2001	384
Control After	Do not contain Islam words	12 September 2001 to 12 December 2001	1684

Design and procedure

The NYT articles were downloaded in batches of 50 articles. The Islam Before corpus and the Control Before corpus consisted of eight text files each. The Islam After corpus and the Control After corpus consisted of 34 text files each. These files were used as input to the LIWC 2007 software. As the output, LIWC returned a file with numeric values corresponding to each category, such as positive and negative emotions scores.

Results and discussion

First, we compared the distribution of positive emotion words in the Islam Before and the Islam After corpora. We found that the number of positive words was approximately the same in the Islam Before corpus (M = 1.95, SD = 0.13, N = 8) and in the Islam After corpus (M = 2.03, SD = 1.9, N = 34). The differences between the two corpora were not statistically reliable on the unpaired t-test (t(40) = 1.1, p = 0.28, two-tailed). This means that the articles that mention the words Islam, Islamic, and Muslim have the same level of positivity before and after 9/11. Next, we compared the distribution of negative emotion words in our Islam corpora. We found that there was an increase in negative words after 9/11. The Islam Before corpus had fewer negative emotion words (M = 2.06, SD = 0.23, N = 8) compared to the Islam After corpus (M = 2.71, SD = 0.37, N = 34). The differences were statistically reliable on an unpaired t-test (t(40) = 4.73, p < 0.0001, two-tailed). Finally, we checked the overall valence of the two Islam corpora. As in the previous study, this measure was derived by subtracting negative emotion means from positive emotion means. Positive numbers indicate that there are more positive emotion words in a given text compared to the negative emotion words, and negative numbers signal negative overall valence. Unlike the previous study, we found that even before the terrorist attacks on 9/11, the overall valence of the texts in which Islam words appear was predominantly negative (M = −0.11, SD = 0.26, N = 8) and that the negativity increased even further after the terrorist attacks (M = −0.69, SD = 0.42, N = 34). The difference was statistically reliable on an unpaired t-test (t(40) = 3.67, p < 0.001, two-tailed). This is similar to what Baker et al. (2013) observed for the representation of Islam in the British press, eventhough they did not analyze negative words per se but token words for conflict.

In order to establish whether the change in sentiment reflects a change in attitudes toward Islam instead of being a reflection of a general trend in language, we analyzed how the words for positive and negative emotions were distributed in our control corpora. First, as far as positivity goes, we found that the texts published before 9/11 were less positive (M = 2.3, SD = 0.12, N = 8) than the texts published after the terrorist attacks (M = 2.45, SD = 0.25, N = 34). The differences were marginally significant on an unpaired t-test (t(40) = 1.71, p = 0.09, two-tailed). Second, with respect to the distribution of negative words, we found that there was an increase in negativity after 9/11: the Control Before corpus had fewer negative emotion words (M = 1.52, SD = 0.31, N = 8) compared to the Control After corpus (M = 1.64, SD = 0.25, N = 34). The differences were not statistically significant (t(40) = 1.14, p = 0.26, two-tailed). Finally, we checked the overall valence of the texts in our control corpora. We found that unlike in the Islam corpora, the predominant emotion in our control corpora was positive. Positive words were more frequent than negative words, and there was a slight increase in positivity after 9/11: the Control Before corpus was slightly less positive (M = 0.77, SD = 0.34, N = 8) compared to the Control After corpus (M = 0.81, SD = 0.38, N = 34). But this slight change toward positivity in the control corpus was not statistically significant (t(40) = 0.28, p = 0.78, two-tailed). The results are shown in Figure 2.

Figure 2.

Sentiment in NYT articles that mention Islam, Islamic, and Muslim (upper panel) and neutral articles (lower panel). There is a sharp increase in negativity in contexts that mention Islam after the terrorist attacks on 9/11.

Our results show that the change in attitudes toward Islam is driven primarily by the increase in negativity and cannot be attributed to the decrease in positive emotion words. This confirms the finding from Study 1. One interesting difference that emerges from these case studies is that unlike Russia, Islam appears in predominantly negative contexts even before the terrorist attacks on 9/11. This negativity only increases after the attacks.

General discussion and conclusion

In the two studies we analyzed how the representation of Russia and Islam, two social actors that are often construed as the ‘Other’ in the Western media (Le, 2002, 2006 and Macgilchrist, 2011 on Russia; Baker et al., 2013 and Richardson, 2004 on Islam), changed after a salient negative event. In both cases the target events under consideration had a profound effect on the relation between the West and the social actors under discussion. We found that the change in attitudes was reflected in mass media language and that the overall trend was the increase in negativity as opposed to the decrease in positivity. In Study 1, we found that NYT articles on Russia published after its annexation of Crimea were characterized by a higher number of negative emotion words compared to the articles on neutral topics from the same period. In Study 2, we found that after the terrorist attacks on 9/11 there was a sharp increase in negativity in articles that mention Islam. One difference with the study on Russia was that negative words in our Islam corpora were more frequent than positive words even before the target event. This is surprising since language in general tends to be positive (Boucher and Osgood, 1969; Matlin and Stang, 1978). However, this finding in fact supports earlier, pre-9/11 work on the representation of Islam in the media which showed that Muslims were often represented in a negative light (Said, 1997).

While our findings about the increase in negativity are in accord with previous CDA literature on the representation of the ‘Other’ (cf. Alazzany and Wong, 2014; Hogan, 2006; Jahedi and Abdullah, 2012; Manan, 2009; Salama, 2011 on the representation of Islam in mass media post 9/11), our study crucially differs from the previous work in the methodology that we use. First, our analysis is diachronic, which allows us to show how the attitudes toward Russia and Islam change over time. Thus, our study is closest in spirit to Baker et al.’s (2013) analysis of Islam in the British press. However, as we discussed in the previous section, there are significant differences between the two. Second, our study uses control corpora. The analysis of control corpora allows us to show that the observed change in attitudes, the increase in negativity specifically, is not a language general phenomenon and can be attributed to Russia and Islam. Finally, by using LIWC, which has a priori defined categories for sentiments, we are able to address the issue of subjectivity raised by Prentice (2010) and others.

In terms of its theoretical contribution, our study sheds light on the mechanisms by which ideologies are construed and reproduced in discourse. A common assumption within the CDA framework is that ‘outgroups will be described in neutral or negative words’ (Van Dijk, 1998: 270). While most work within the CDA tradition focuses on the negative representation of the ‘Other’, in our study we also analyze the relation between negative and positive emotion words. The important finding from both studies is that the change of attitudes is driven by an increase in negativity rather than by a decrease in positivity. Why should this be the case? One possible explanation for this trend is what psychologists call the ‘negativity bias’, that is, the finding that bad emotions, experiences, and events often evoke a stronger cognitive response compared to the strength of response that positive events of the same magnitude would evoke (Baumeister et al., 2001; Rozin and Royzman, 2001). For example, people’s bad behaviors are remembered better than good ones (Skowronski and Carlston, 1987, cited in Baumeister et al., 2001: 343), and the same holds for social groups: ‘unfavorable characteristics once acquired as part of a stereotype may be difficult to lose’ (Baumeister et al., 2001: 344). Stereotypes are usually based on ‘illusory correlations’, that is, false assumptions that membership in a particular social group (immigrants) is associated with a specific type of negative trait or behavior (extremism or drug abuse). Once we have an instance of ‘bad’ behavior, such as Russia’s annexation of Crimea, there will be a tendency to link the social actors responsible for this event to other instances of bad behavior. From this perspective, the increase in the number of negative emotion words in the contexts that mention Russia and Islam in the aftermath of salient negative events might be an instantiation of a more general cognitive trend, the negativity bias.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author biographies

Anastasia Smirnova, PhD, is an Assistant Professor of Linguistics at San Francisco State University. Her research focuses on cognitive and social aspects of language. Her publications address the questions about language ideology, how language affects stereotypes in situations of ethnic conflicts, and what properties of language reveal about the speaker and the society. To address these questions she employs a variety of empirical methods, ranging from behavioral experiments to corpus studies.

Helena Laranetto is a postgraduate student in Linguistics and Research Assistant at San Francisco State University. Her research interests are in the fields of cognitive linguistics, critical discourse analysis, and rhetoric with a particular focus on legitimization and discursive authority in government communication.

Nicholas Kolenda is an undergraduate student at the University of Michigan who is dual-majoring in Political Science and Near-Eastern Studies with an emphasis on Arabic and Islam. Nicholas’ research interests have largely consisted of studying and observing discourse within society with a specific interest in the politicization of language. Nicholas is also a former contributor to Dr Juan Cole’s blog ‘Informed Comment’.

References

Alazzany

MAOA

Wong

(2014) Islam and Muslims in the New York Times: Two versions, two camps. Pertanika Journal of Social Sciences & Humanities 22: 33–58.

Back

Küfner

Egloff

(2011) ‘Automatic or the people?’ Anger on September 11, 2001, and lessons learned for the analysis of large digital datasets. Psychological Science 22(6): 837–838.

Baker

(2006) Using Corpora in Discourse Analysis. London: Continuum.

Baker

Gabrielatos

Khosravinik

. (2008) A useful methodological synergy? Combining critical discourse analysis and corpus linguistics to examine discourses of refugees and asylum seekers in the UK press. Discourse & Society 19(3): 273–306.

Baker

Gabrielatos

McEnery

(2013) Discourse Analysis and Media Attitudes: The Representation of Islam in the British Press. Cambridge: Cambridge University Press.

Baumeister

Bratslavsky

Finkenauer

. (2001) Bad is stronger than good. Review of General Psychology 5(4): 323–370.

Boucher

Osgood

(1969) The Pollyanna hypothesis. Journal of Verbal Learning and Verbal Behavior 8(1): 1–8.

Breeze

(2011) Critical discourse analysis and its critics. Pragmatics 21(4): 493–525.

Fairclough

(1995) Media Discourse. London: Arnold.

10.

Fairclough

(2010) Critical Discourse Analysis: The Critical Study of Language. London; New York: Routledge.

11.

Fairclough

(2015) Language and Power. New York: Routledge.

12.

Fairclough

Wodak

(1997) Critical discourse analysis. In: Van Dijk

(ed.) Discourse as Social Interaction. London: SAGE, pp. 258–284.

13.

Fornaciari

Ranganathan

Venkatakrishnan

(2014) Sensitivity of information disclosed in Amazon reviews. In: Proceedings of the 8th Annual International Conference on Digital Government Research: Bridging Disciplines & Domains (ed. Berntzen

Smedberg

Tangian

), Barcelona, 23–27 March, pp. 1–7. Red Hook, NY: Curran Associates.

14.

Fowler

(1991) Language in the News: Discourse and Ideology in the Press. London: Routledge.

15.

Fowler

(1996) On critical linguistics. In: Caldas-Coulthard

Coulthard

(eds) Texts and Practices: Readings in Critical Discourse Analysis. New York: Routledge, pp. 3–14.

16.

Fowler

Hodge

Kress

. (1979) Language and Control. London: Routledge & Kegan Paul.

17.

Hodge

Kress

(1993) Language as Ideology. London: Routledge.

18.

Hogan

(2006) Letters to the editor in the ‘war on terror’: A cross-national study. Mass Communication and Society 9(1): 63–83.

19.

Hunston

(2002) Corpora in Applied Linguistics. Cambridge: Cambridge University Press.

20.

Iliev

Dehghani

Sagi

(2015) Automated text analysis in psychology: Methods, applications, and future developments. Language and Cognition 7(2): 265–290.

21.

Jahedi

Abdullah

(2012) Post-September 11 discourse: The case of Iran in The New York Times. International Journal of English Linguistics 2(1): 59–70.

22.

Koller

Mautner

(2004) Computer applications in critical discourse analysis. In: Coffin

Hewings

O’Halloran

(eds) Applying English Grammar: Corpus and Functional Approaches. London: Arnold, pp. 216–228.

23.

Krishnamurty

(1996) Ethnic, racial and tribal: On critical linguistics. In: Caldas-Coulthard

Coulthard

(eds) Texts and Practices: Readings in Critical Discourse Analysis. New York: Routledge, pp. 129–149.

24.

(2002) Human rights discourse and international relations: Le Monde’s editorials on Russia. Discourse & Society 13(3): 373–408.

25.

(2006) The Spiral of Anti-Other Rhetoric: Discourses of Identity and the International Media Echo. Amsterdam; Philadelphia, PA: John Benjamins.

26.

Ott

Cardie

. (2014) Towards a general rule for identifying deceptive opinion spam. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, MD: Association for Computational Linguistics, 23–25 June, pp. 1566–1576.

27.

Macgilchrist

(2011) Journalism and the Political: Discursive Tensions in News Coverage of Russia. Amsterdam; Philadelphia, PA: John Benjamins.

28.

Manan

(2009) Through Western eyes: Covering Islam after September 11. In: Abdullah

Abdullah

Hoon

(eds) Critical Perspectives on Language and Discourse in the New World Order. Newcastle: Cambridge Scholars Publishing, pp. 111–132.

29.

Matlin

Stang

(1978) The Pollyanna Principle. Cambridge: Schenkman.

30.

Newman

Pennebaker

Berry

. (2003) Lying words: Predicting deception from linguistic styles. Personality and Social Psychology Bulletin 29(5): 665–675.

31.

Orpin

(2005) Corpus linguistics and critical discourse analysis: Examining the ideology of sleaze. International Journal of Corpus Linguistics 10(1): 37–61.

32.

Pennebaker

King

(1999) Linguistic styles: Language use as an individual difference. Journal of Personality and Social Psychology 77(6): 1296–1312.

33.

Pennebaker

Stone

(2003) Words of wisdom: Language use over the life span. Journal of Personality and Social Psychology 85(2): 291–301.

34.

Pennebaker

Booth

Francis

(2007a) Linguistic Inquiry and Word Count: LIWC [Computer software]. Austin, TX: LIWC.net.

35.

Pennebaker

Chung

Ireland

. (2007b) The Development and Psychometric Properties of LIWC 2007 [LIWC manual]. Austin, TX: LIWC.net.

36.

Pennebaker

Mehl

Niederhoffer

(2003) Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology 54(1): 547–577.

37.

Prentice

(2010) Using automated semantic tagging in Critical Discourse Analysis: A case study on Scottish independence from a Scottish nationalist perspective. Discourse & Society 21(4): 405–437.

38.

Prentice

Rayson

Taylor

(2012) The language of Islamic extremism: Towards an automated identification of beliefs, motivations and justifications. International Journal of Corpus Linguistics 17(2): 259–286.

39.

Reisigl

Wodak

(2009) The discourse-historical approach (DHA). In: Wodak

Meyer

(eds) Methods of Critical Discourse Analysis. London: SAGE, pp. 87–121.

40.

Richardson

(2004) (Mis)representing Islam: The Racism and Rhetoric of British Broadsheet Newspapers. Amsterdam; Philadelphia, PA: John Benjamins.

41.

Rozin

Royzman

(2001) Negativity bias, negativity dominance, and contagion. Personality and Social Psychology Review 5(4): 296–320.

42.

Said

(1997) Covering Islam: How the Media and the Experts Determine How We See the Rest of the World. New York: Random House.

43.

Salama

(2011) Ideological collocation and the recontexualization of Wahhabi-Saudi Islam post-9/11: A synergy of corpus linguistics and critical discourse analysis. Discourse & Society 22(3): 315–342.

44.

Skowronski

Carlston

(1987) Social judgment and social memory: The role of cue diagnosticity in negativity, positivity, and extremity biases. Journal of Personality and Social Psychology 52(4): 689–699.

45.

Stubbs

(1997) Whorf’s children: Critical comments on critical discourse analysis (CDA). British Studies in Applied Linguistics 12: 100–116.

46.

Stubbs

Gerbig

(1993) Human and inhuman geography: On the computer-assisted analysis of linguistic texts. In: Hoey

(ed.) Data, Description, Discourse. London: HarperCollins, pp. 64–85.

47.

Tausczik

Pennebaker

(2010) The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology 29(1): 24–54.

48.

Taylor

(2014) Investigating the representation of migrants in the UK and Italian press: A cross-linguistic corpus-assisted discourse analysis. International Journal of Corpus Linguistics 19(3): 368–400.

49.

Tumasjan

Sprenger

Sandner

. (2010) Predicting elections with Twitter: What 140 characters reveal about political sentiment. In: Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media, Washington, DC, 23–26 May, pp. 178–185. Menlo Park, CA: AAAI Press.

50.

Van Dijk

(1987) Communicating Racism: Ethnic Prejudice in Thought and Talk. Newbury Park, CA: SAGE.

51.

Van Dijk

(1989) Mediating racism: The role of the media in the reproduction of racism. In: Wodak

(ed.) Language, Power, and Ideology. Amsterdam; Philadelphia, PA: John Benjamins, pp. 199–226.

52.

Van Dijk

(1991) Racism and the Press: Critical Studies in Racism and Migration. London; New York: Routledge.

53.

Van Dijk

(1993) Principles of critical discourse analysis. Discourse & Society 4(2): 249–283.

54.

Van Dijk

(1995) Discourse analysis and ideology analysis. In: Schaffner

Wenden

(eds) Language and Pace. Aldershot: Dartmouth Publishing, pp. 17–33.

55.

Van Dijk

(1998) Ideology: A Multidisciplinary Approach. London: SAGE.

56.

Van Dijk

(2001) Critical discourse analysis. In: Schiffrin

Tannen

(eds) The Handbook of Discourse Analysis. Malden, MA; Oxford: Blackwell, pp. 352–371.

57.

Van Leeuwen

(1995) Representing social action. Discourse & Society 6(1): 81–106.

58.

Verschueren

(2001) Predicaments of criticism. Critique of Anthropology 21(1): 59–81.

59.

Widdowson

(1995) Discourse analysis: A critical view. Language and Literature 4(3): 157–172.

60.

Wodak

(1996) The genesis of racist discourse in Austria since 1989. In: Caldas-Coulthard

Coulthard

(eds) Texts and Practices: Readings in Critical Discourse Analysis. London: Routledge, pp. 107–128.

61.

Wodak

(2006) Critical linguistics and critical discourse analysis. In: Ostman

Verschueren

(eds) Handbook of Pragmatics. Philadelphia, PA; Amsterdam: John Benjamins, pp. 1–25.

62.

Wodak

Busch

(2004) Approaches to media texts. In: Downing

McQuail

Schlesinger

. (eds) The SAGE Handbook of Media Studies. Thousand Oaks, CA: SAGE, pp. 105–122.

63.

Wodak

Meyer

(2009) Critical discourse analysis: History, agenda, theory, and methodology. In: Wodak

Meyer

(eds) Methods of Critical Discourse Analysis. London: SAGE, pp. 1–32.