جمال_خاشقجي# #JamalKhashoggi: Unraveling multilingual Twitter sentiment dynamics in a longitudinal comparative analysis of tweets in Arabic and English

Abstract

In 2018, Jamal Khashoggi, a Saudi journalist who wrote for the Washington Post, was last seen alive entering the Saudi Consulate in Istanbul. Confirmed news of his murder ignited a heated and polarized public debate on Twitter. We use agenda melding as a theoretical lemma and argue that Twitter sentiment flourishes within multilingual, ad hoc public spheres contributing to an emotional agenda. We examined the Twitter sentiment from 2018 to 2021 by looking at the most popular hashtags used in both the Arabic-and-English language spheres. The daily sentiment analysis of 3,278,464 tweets revealed that both languages had a predominantly negative sentiment; however, the English sphere was more extreme in their emotional expression. An additional analysis of external media URLs found in a subsample of tweets highlighted distinct references to media discourse, emphasizing an East–West divide. Implications for global communication are discussed.

Keywords

#JamalKhashoggi ad hoc public spheres agenda melding multilingual sentiment analysis Twitter

Jamal Khashoggi, a Saudi journalist who wrote for the Washington Post, was last seen alive entering the Saudi Consulate in Istanbul, Turkey on 2 October 2018. It was later revealed that a 15-hitman squad, tortured, killed, and dismembered Khashoggi’s body inside the Consulate (Kirkpatrick and Gall, 2018; Zarocostas, 2018). Once his death was confirmed, the media and the Turkish government immediately pointed to Saudi Crown Prince Mohamed Bin Salman (MBS), accusing him of ordering the killing and violating a number of international and human rights laws (Milanovic, 2020). Khashoggi’s death went from a Saudi political matter to an international political event, mostly driven by social media, such as Twitter (#JamalKhashoggi; جمال_خاشقجي#; note the reading direction from right to left), and it became a focal point for both public outrage against and support for Saudi Arabia and the Crown Prince.

Khashoggi’s murder ignited a polarized debate in both legacy and social media. An analysis of the Washington Post (in English) and Al Jazeera (in Arabic) showed that the news outlets politicized the issue and shifted their reporting style from objective coverage to murder accusations toward Saudi Arabia (AlMomani and Atiyyat, 2020). On Twitter, a polarized public debate flared up between regional coalitions of activists and Egyptian members of the Muslim Brotherhood exiled in Turkey who opposed Saudi Arabia’s leadership against a group of mostly Saudi accounts and media outlets who defended the Kingdom (Abrahams and Leber, 2021). The public discourse on social media in the days following the murder revolved largely around the then-trending Arabic hashtag (جمال_خاشقجي#). There is, to date, only scarce evidence of the sentiment in polarized debates in Arabic, let alone in comparison with a broader debate in other languages, or the dynamics of such debates.

Khashoggi’s murder case was a trigger event (Birkland, 1998) that brought Saudi Arabia’s leadership into the spotlight and to the top of international news agendas, with Western news media blaming Saudi Arabia’s leadership, and Middle Eastern news media defending Saudi Arabia’s leadership. Controversial mediated conflicts are prone to instrumental actualization (Kepplinger et al., 1991), especially if the events revolving around the conflict support multiple viewpoints; then, the reporting can be seen as instrumental for the cause of one side and evokes reactions from the other side. Similarly, agenda dynamics were traditionally investigated within the same cultural environment and for broader societal issues (but see Vliegenthart and Walgrave, 2008), thus largely disregarding social media sentiment that can influence public agendas via public expressions of emotions to news.

To integrate social media sentiment into agenda-setting, we use agenda melding as a conceptual lemma that describes the process of combining “agendas from various sources, including other people, to create pictures of the world that fit our experiences and preferences” (McCombs et al., 2014: 794). Emotions can mediate agenda-setting (Miller, 2007), which still needs more conceptual integration as a relatively new phenomenon (see Weimann and Brosius, 2017). Particularly multilingual, cross-cultural perspectives are lacking—a research gap that we aim to narrow in this article. In order to do so, we compared the daily sentiment in 3,278,464 unique tweets around Khashoggi’s murder from 2018 to 2021 using popular hashtags in both Arabic and English. The hashtag approach provided a way to examine different digital public communities shaped by language. We also included references to external media URLs mentioned in the tweets and compared their sentiment to the sentiment in the tweets.

The public’s emotions: merging intermedia agenda-setting and emotions

International intermedia agenda-setting

Western, educated, industrialized, rich, and democratic (WEIRD) countries are the dominant agenda-setters for international news (Chang, 1998; Galtung, 1971), while peripheral countries (Galtung, 1971) are rarely mentioned in international news coverage unless a disaster, conflict, or an exceptional event occurs to make them internationally newsworthy, although the coverage is often negative (Ali Mohammed and Mccombs, 2021; Golan, 2006). Expanding on second-level agenda-setting, where affective news attributes and tone are considered as crucial as object salience (McCombs et al., 1997), Golan (2006) argued that the intermedia agenda-setting process is the source of newsworthiness for international events. Rather than how media shapes the public’s agenda, intermedia agenda-setting focuses on the influence media agendas have on each other, highlighting how media agendas are being shaped (Roberts and Mccombs, 1994). Guo and Vargo (2017) examined the countries’ intermedia agenda-setting power by looking at emerging online-only media (e.g. BuzzFeed) and traditional media (e.g. newspapers). While the United States dominated the world news and had the most intermedia agenda-setting power, Saudi Arabia ranked only second to the United States in terms of intermedia agenda-setting power. Ali Mohammed and McCombs (2021) explored the Western media’s (the New York Times and The Guardian) interaction with Egyptian news media (Al-Ahram, English) regarding a murdered Italian student in Egypt and showed that there are constant intermedia agenda-setting dynamics and dependencies, suggesting that no single agenda can lead all other agendas all the time.

The reporting practices of media organizations in elite countries, specifically the United States, are not limited by geographical distance or language barriers (unlike local media in less-developed countries), which explains why elite media are likely to have a broader coverage of world news but with a higher level of conformism and reproduction (Grasland, 2020). For instance, studies on framing of the Syrian crisis suggest that the news coverage tends to focus exclusively on conflict and war frames (Chuang and Roemer, 2013), which has led the American public to view the Syrian refugees as a threat to the American identity (Jahng and Doshi, 2021). This emphasizes the importance of the media in assigning importance to certain topics and in shaping how the public perceives these topics.

Adding emotions to agenda-setting

Miller (2007) highlighted the importance of the public’s emotional reactions to an issue, which reflects how audiences attribute issue importance. Negative emotional reactions signal an important problem that needs to be addressed. The advent of social media accelerated things: emotional framing was largely invisible in news media due to journalistic ideals of objectivity, and now has spillover effects on public expressions of emotions via social media (Wahl-Jorgensen, 2019), where negative sentiments used by the mainstream media stir up negative emotions reflected on Twitter (Nikolayenko, 2019). However, the reverse is also true in that emotional expression on social media spill over into mainstream media’s reporting (Chadwick, 2013). Similarly, Russell Neuman et al. (2014) examined the framing of 29 political issues in traditional and social media. Their findings highlighted a dynamic relationship between traditional media agendas and Twitter public agendas, instead of a one-way pattern. Ceron et al. (2016) took this line of research one step further and examined the second-level agenda-setting by looking at the Twitter user’s sentiment around two Italian political debates. The findings indicated that traditional media remains the first-level agenda-setter; however, they did not find any reverse effect. In terms of sentiment, Twitter conversation had a more negative sentiment than traditional media and media slant and Twitter sentiment were completely uncorrelated. Exploring this dynamic in a controversial context, Qin (2015) examined the case of Edward Snowden by looking at both legacy media and the public discourse on Twitter. Findings revealed that the Twitter discourse framed Snowden more positively and highlighted the differences between the framing of a global issue in legacy news media and on Twitter. Thus, although legacy and social media follow a similar agenda (Bang et al., 2021), there seem to be tectonic news shifts going on with Twitter facilitating the creation of counter-public spheres (Thorsen and Sreedharan, 2019), where users build their own agendas (O’Boyle, 2019) and thereby contribute to the global agenda-setting of news media (O’Boyle and Pardun, 2021). However, the reasons behind such differences remain largely unexplained.

For example, AlMomani and Atiyyat (2020) showed in a content analysis of the Washington Post, an intermedia agenda-setter (Livingston and Asmolov, 2010), that the discourse became politicized and shifted from objective news coverage to a more accusatory tone toward Saudi Arabia’s leadership. Furthermore, El-Falaky (2019) specifically looked at the journalists’ linguistic choices in the online versions of four newspapers—Arab News (Saudi Arabia), Hurreyat Daily News (Turkey), the New York Times (United States), and the Tehran Times (Iran) and found that headlines were strategically framed to encourage the reader to adopt the opinions preferred by the national policies of their countries.

Twitter as a public sphere with language-parallel political discourses

Globalization coupled with social network sites have shifted the discussions/debate from a national to a global domain, resulting in what is referred to as “global civil society.” In such society, the movement of public opinion is reflected through the spread of information in a diversified media system, and of “the emergence of spontaneous, ad hoc mobilization using horizontal, autonomous networks of communication” (Castells, 2008: 86). In particular, Twitter allows users to create their own hashtags in response to a real-time political situation instantly with no delays. The instantaneous formation of a hashtag community, and thus a public sphere, in reaction to a political event is what Bruns and Burgess (2011) term an ad hoc public sphere. The dynamic nature of conversation within the ad hoc public sphere can provide insight into not only how individuals within a hashtag community interpret and react to certain events but also how the event is being understood differently between hashtag communities.

In the context of a global and conflictual event, language differences can be a barrier to a global “opinion crossroads.” As hashtag communities are often shaped by language, such communities can largely co-exist in parallel with no way for discussions to merge. For instance, Bodrunova et al. (2018) investigated the role language played in the globality of the ad hoc Twitter discussion revolving around the “polar” #JeSuisCharlie and #JeNeSuisPasCharlie hashtags after the killings of Charlie Hebdo. Despite the presence of over 30 languages in these public spheres, the debate remained largely Euro-Atlantic, highlighting a prevalent echo chambers that represented neither the conflict sides nor the “clash of civilizations” (Bodrunova et al., 2018). Interestingly, their findings also indicated that despite the fact that English-language users can bridge conversations, they simultaneously excluded non-English-speaking users from the discussion. Similarly, Hopke (2015) examined the transnational, anti-fracking movement by focusing on the #globalfrackdown hashtag and looking at multilingual Twitter users who communicated in both English and Spanish. The analysis revealed a high level of consistency between tweets in both languages; however, only the use of multilingual hashtags in Spanish-speaking tweets encouraged a crossflow of information between languages and the sharing of information beyond their linguistic sphere. In another study, Öztürk and Ayvaz (2018) examined the discussion revolving around the Syrian refugees in Turkey in two distinct communities shaped by language. Their analysis revealed that not only did the content of their tweets focus on different aspects of the issue but that their sentiment toward the issue also differed.

In a similar event to Khashoggi’s murder, Kovács et al. (2021) examined the European countries’ Twitter conversation revolving around the controversial murder case of Ján Kuciak, a Slovakian journalist investigating corruption in Slovakia in 2018. Following the #AllforJan hashtag and other key words, the study translated the non-English tweets into English, clustered the countries based on the patterns of twitting activity and then investigated the sentiment of the conversation overtime. Their analysis revealed that the activity peaks in the clusters differed based on the proximity of a given country to Slovakia (i.e. neighboring countries) and whether a similar case (a journalist murder case) took place at a given country. Moreover, upon investigating the Twitter sentiment over time, they found that the sentiment became increasingly positive, suggesting that users supported and approved of the claims about the connections between the Slovakian government and the mafia and its political consequences. Thus, public Twitter spheres that are shaped by language provide insight into how different communities react to the same event, and how their understanding of the event might shift and change over time, which is expressed via the user’s sentiment. We hypothesize:

H1. The sentiment of tweets about the Khashoggi murder case will become more positive over time.

Explaining differences in sentiment dynamics on Twitter through language

Language-shaped hashtag communities have different sentiment dynamics. Naskar et al. (2020) analyzed the emotional expression dynamic of Twitter users in English. Their findings indicated that 63% of users changed their opinion over time. Interestingly, terrorism and terror-related topics were the only topics that maintained both negative emotions and opinions in the English tweets over time. This suggests that when the English-speaking community reacts to news related to the Middle East, often covered through a war or conflict frame, negative sentiments are likely to dominate. To our knowledge, Arabic Twitter sentiment over time has not been extensively examined. Jamal et al. (2015) looked at Arabic tweets from seven different countries between 1 January 2012 and 3 December 2013—a period of instability in the region—and their findings indicated a deep-rooted mistrust toward the United States, where the negative sentiment was present regardless of what the United States does (Romney et al., 2021). This mistrust is intensified by the conspiracy theories that are embedded in the Middle East and reflected in the long-grown paranoia associated with nontransparent sources of information, such as the elite news organizations that often report negatively on the region (Zonis and Joseph, 1994). Against this backdrop and given the nature and the proximity of the event to the Arabic users, we hypothesize:

H2. The sentiment of tweets about the Khashoggi murder case in the English language will be less negative than the sentiment of tweets in Arabic.

Based on the intermedia agenda-setting and emotional news framing rationale where more personalized and emotional news receives a greater share of attention (see Beckett and Deuze, 2016), we assume that more emotional news is more likely to be shared on Twitter as it will potentially evoke stronger emotional reactions (i.e. tweets with a more positive or negative sentiment). Given the scarce research in this specific area, we ask the following research question (RQ):

RQ1. To what extent do the tweets about the Khashoggi murder case contain external references with polarized (i.e. positive or negative) sentiment?

The present study examines Twitter data from 1 October 2018 to 5 October 2021, using the most popular hashtags around the Khashoggi case. The daily sentiment of N = 3,278,464 unique tweets were analyzed in both languages. This study helps to fill multiple gaps in news research by highlighting the importance of emotional expression on social media not only within a digital community shaped by a common language but also between different digital communities separated by language.

Method

Event data

Khashoggi’s disappearance and its aftermath were closely followed by the media. The following timeline highlights some of the most significant dates based on CNN Arabic (2018) and CNN English (CNN, n.d.) articles (Figure 1). The CNN Arabic article was translated to English by the first author, who is fluent in Arabic.

Figure 1.

Timeline and aftermath of the Khashoggi murder case.

Twitter data collection

We used the Academic Research Product Track of Twitter API v2 to harvest the data. This track grants access to the entire Twitter archive instead of the last 6–9 days provided by the standard Twitter API. We harvested all tweets between 1 October 2018 00:00 AM until 5 October 2021 23:59 PM, starting a day before Khashoggi’s murder on 2 October 2018 and ending just after the third-year anniversary of this event, which gives us access to all the developments in the case overtime. The Twitter database queries contained Khashoggi’s name, both with and without a hashtag preceding it. Additional exploratory analyses revealed the most relevant Khashoggi-related hashtags on Twitter.¹ The final query included the following search terms: Khashoggi, #khashoggi, jamalkhashoggi, #jamalkhashoggi, #jamal_khashoggi, jamal_khashoggi, and their respective translations in Arabic (خاشقجي_جمال #,خاشقجي_جمال #,خاشقجي,خاشقجي).

For each tweet, we collected tweet-level data (e.g. tweet text; number of likes, replies, retweets; URLs and hashtags used), account-level data (e.g. account ID; number of followers; age of the account; self-specified location of the account), and metadata (e.g. tweet language; source [i.e. through which channel the tweet was sent]; conversation ID; geolocation of the tweet). A full list of all API endpoints is provided at the end of the manuscript.² We requested only English and Arabic tweets from the API and purposefully excluded retweets (i.e. only original content or quoted retweets were harvested). For each external webpage linked in the tweet, we scraped the actual URL as well as the title of the webpage and the webpage description provided by the Twitter API.

The harvesting was done in Python 3.9.7 for Mac. The Twarc library (version 2.3.10) was used to communicate with the Twitter API and to retrieve each tweet as a JSON object (Twarc, 2021). Tweets were collected in batches of 100, and the relevant variables (see the endpoints in the Note 2) were parsed into different data formats and stored in a locally hosted MySQL database using the SQLAlchemy library (Bayer, 2012). We started the data collection on 7 October 2021, and tweets were collected in reverse chronological order. This resulted in a total sample of N = 3,280,983 tweets.

Data cleaning and pre-processing

We removed 2127 tweets in languages other than English or Arabic, and removed all duplicates, wrongly parsed tweets, and tweets with no actual content. We pre-processed the raw textual content of each tweet by removing all @-mentions, hashtags, numbers, URLs, emoticons, punctuation, and special characters. Stop word removal was done with the Natural Language Processing package tidytext in R (Silge and Robinson, 2016). This pre-processing protocol resulted in a new variable containing only the core message of each tweet. This was used for all further analyses. The urltools library in R was used to pre-process the URL data. This package extracts the domain name, subdomain, extension, and other relevant web address information from a given URL.

Data analysis

We applied a three-step approach. First, sentiment analysis (Rastogi and Bansal, 2021) was used to gauge public attitudes in the context of Khashoggi’s murder. In order to do so, we used the NRC word-emotion lexicon, which exists both in English and Arabic (Mohammad and Turney, 2013). This approach enabled us to assess the text polarity (i.e. positive vs negative sentiment) for each tweet. Second, we computed a polarity index with the following formula:

P o l a r i t y = \frac{(P o s i t i v e - N e g a t i v e)}{(P o s i t i v e + N e g a t i v e)}

The polarity index ranges from −1 (exclusively negative) to +1 (exclusively positive). This allows us to explore the polarity in the Twitter discourse over time. Third, we aggregated the polarity scores on three different levels: First, we grouped the data per day. This enabled us to analyze how the discourse around the Khashoggi murder case evolved over time in the Arabic and English Twitter spheres (H1). Second, with reference to our second hypothesis (H2), we aggregated the sentiment on the level of the hashtags. This was done to study differences in polarity in the different hashtags as proxies for ad hoc publics (see Bruns and Burgess, 2011). Finally, in line with previous research (Kursuncu et al., 2018; Wang et al., 2011), we summarized polarity scores on domain level of the referenced domain names within the tweets to answer our research question (RQ1).

Results

The final sample consisted of n = 3,278,464 unique tweets in Arabic (n = 1,583,777) and English (n = 1,694,687). In both language spheres, tweet volumes peak in late October 2018 and then decrease steadily, even as significant events continue to influence them regularly (see Figure 2).

Figure 2.

Daily tweet volume timeline with annotated remarkable dates and events.

The daily proportions of positive and negative sentiments fluctuated differently in both language spheres. As demonstrated by the area plots in Figure 3, the proportion of total positivity and negativity remained stable in the Arabic-language sphere, while the proportion of positivity gradually increased in the English-language sphere. A linear regression analysis in which we controlled for the daily proportion of negativity, R² = .031, F (2,1098) = 18.45, p < .001, corroborates that English-language tweets tended to be more positive over the course of time (β = .177, p < .001). This trend, however, was not significant for the proportion of negativity in the English-language sphere (β = –.013, p = .682) nor for the sentiments in the Arabic-language sphere (β_negative = –.051, p = .078; β_positive = .051, p = .080). Therefore, our first hypothesis (H1) was confirmed for the English-language Twitter sphere only.

Figure 3.

Evolution over time of the average daily proportions of positivity and negativity of tweets in Arabic and English.

Figure 4 shows the daily average polarity scores. Overall, both Twitter spheres were predominantly negative. However, and in contrast to H2, the Arabic discourse was much more nuanced than the English discourse, where the peaks in the Arabic sphere were less dramatic than in the English sphere.

Figure 4.

Daily average polarity score of tweets in Arabic and English.

This suggests that the English discourse is more extreme in expressing sentiments on both positive- and negative-laden days. A two-sample t-test confirmed that the overall average daily polarity score in the English Twitter-sphere (M_English = –.106, SD_English = .123) was statistically significantly more negative than in the Arabic sphere (M_Arabic = –.039, SD_Arabic = .080), t = 15.499, df = 1930.8, p < .001. Thus, we reject H2.

Additionally, there are several observations worth mentioning: First, the year 2021 was, overall, more positive in both Twitter spheres, yet this is particularly true for the Arabic Twitter discourse. Our data show that 45% of the days investigated for 2021 (until October) were predominantly positive, which is significantly more than in 2020 (18.6%), 2019 (19.7%), and 2018 (34.8%) (Cramer’s V = .238, χ² = 71.394, p < .001). Second, we observed that the most negative day in the subsample was 3 April 2021 (polarity score = –.63). This was arguably related to an attempted coup d’état in Jordan. Third, in the Arabic Twitter sphere, positive days often came in sequences of multiple days. For example, the periods around 30 September 2019, 24 November 2019, 21 May 2020, and early March 2021 stood out as we observed multiple consecutive days that were predominantly positive in the Arabic discourse. Positive days in the English Twitter discourse were more scattered over time. The positive sentiment peak was on 30 September 2019 (polarity score = 0.16), which coincided with the broadcast of the “60 minutes” interview where the Crown Prince took responsibility for what had happened to Khashoggi (CBS News, n.d.). Simultaneously, the human rights initiative “DAWN” (Democracy for the Arab World Now) was officially launched and was perceived as predominantly positive by the Arabic Twitter sphere.

Understanding hashtags as ad hoc public spheres, to further differentiate the findings for H2, we aggregated the polarity scores at the hashtag level. Figure 5 shows the results of this analysis (see also Tables S1 and S2 in the Appendix for additional information).

Figure 5.

Average polarity scores per hashtag.

There are five general trends that can be observed in the data. First, the most frequently used Arabic and English hashtags differ in their polarity. In line with our previous findings, the overall average polarity is significantly more negative for English hashtags than for Arabic ones, t = 4.768, p < .001. Additionally, among the top 30 hashtags in English, we found only three that appeared in slightly positive-toned tweets: #JusticeforJamal (.059), #Istanbul (.039), and #CIA (.006). In contrast, 12 out of the top 30 hashtags in the Arabic sample appeared in tweets with a net positive average sentiment. The single most positive hashtag was الله_ذمه_في_خاشقجي_جمال# [Jamal Khashoggi, has passed away], with an average polarity score of .364.

Second, we found that there was “hashtag hijacking” (Gilkerson and Berg, 2017; Sanderson et al., 2016; Xanthopoulos et al., 2016) in the English Twitter sphere, which describes the use of a popular yet otherwise unrelated hashtag only to boost the publicity of an issue. Out of the 30 most-used hashtags in the English sample, three strong negative hashtags referred to Fred Lumbuye and/or Uganda, but nevertheless also used a more popular hashtag related to the Khashoggi murder case. For example, #SearchforFredLumbuye appeared to be the most negative (–.318) and #UgandaIsBleeding ranked fifth (–.193). Fred Lumbuye is a Ugandan blogger, human rights activist, and critic of Ugandan President Museveni. He was imprisoned in Turkey in early August 2021 (DW, n.d.). While this event barely got any Western media attention, it was discussed on Twitter and brought to attention by including hashtags referring to Khashoggi’s case due to some case parallels.

Third, the Khashoggi discourse on Twitter is geo-political in nature. In both subsamples, a substantial number of the most prevalent hashtags refer to other countries and country leaders, such as #Yemen, #Iran, #Turkey/ تركيا# [Turkey], #Erdogan/ أردوغان# [Erdogan], #USA, #Trump/ ترامب# [Trump], قطر# [Qatar], and اسطنبول# [Istanbul]. What both samples have in common is an overall negative sentiment in tweets that mention Turkey and Erdogan, although the Arabic sphere is slightly less negative toward both.

Fourth, the two spheres are diametrically opposed when it comes to tweets about Saudi Arabia. The English discourse is rather negative toward everything Saudi-related (#Saudis, #KSA, #SaudiArabia, #Saudi), but this is not the case for the Arabic subsample. Hashtags referring to the country are rather neutral (e.g. السعودية# [Saudi Arabia] (.000)) or even positive (العظمى_السعودية# [Greater Saudi Arabia] (.107)). In fact, the second-most positive hashtag out of the top 30 Arabic hashtags was السعوديه_العدل_مملكه# [Kingdom_of_justice_Saudi Arabia] (.214).

Fifth, MBS, the Saudi Crown Prince, is mentioned mainly in negative tweets in both samples (مبس# [mbs] (–.110), #MBS (–.096), سلمان_بن # [bin_salman] (–.090), and سلمان_بن_محمد# [Mohammed bin Salman] (–.021)). This might be a direct result of the news about the intelligence reports that say MBS was personally involved in approving the operation to capture and kill Khashoggi. In fact, in the Arabic Twitter sphere, they even have a dedicated hashtag for this: خاشقجي_قاتل_مبس# (Mbs_killer_Khashoggi). Yet, the fact that this hashtag has a rather neutral polarity score (–.006) raises the question of to what extent an orchestrated effort was undertaken to neutralize this hashtag/ad hoc public.

Exploratory analysis: differences between the Arabic- and English-cited domain names

For the last part of our analysis, we aggregated the data on the level of the external domains that the tweets referred to. This allowed us to investigate to what extent references to different websites drove certain sentiments on Twitter (RQ1). Two important findings will be discussed here. First, the Khashoggi Twitter discourse is not strongly embedded within a broader media ecosystem. Only 0.15% (N = 4942) of the total sample of tweets referred to an external website. Arguably, most of the discourse on Twitter was not linked to external sources. Additionally, there was only little overlap between the top-ranked domains in both Twitter spheres. As can be seen in the bump chart in Figure 6, only seven domains were found in both Twitter spheres.

Figure 6.

Bump chart showing the overlap between the top 30 URL link domain names mentioned in tweets in Arabic or English.

These domains were YouTube, Twitter, CNN, the Washington Post, the New York Times, The Guardian, and the BBC. This suggests that these two Twitter-spheres are citing sources from two very distinct media discourses, with a clear East–West divide: Whereas tweets in the Arabic Twitter sphere predominantly referred to social media such as YouTube (n = 453), Twitter (n = 293), and Telegram (n = 75), and to mainstream Saudi news media (e.g. okaz, mwso3h), the English Twitter sphere was mostly geared toward Western legacy news media such as the New York Times (n = 365), the Washington Post (n = 365), The Guardian (n = 182), CNN (n = 134), BBC (n = 82), NBC News (n = 39), CNBC (n = 32), AP News (n = 38), and NPR (n = 32). Furthermore, we observed that the independent news organization Middle East Eye—a news outlet for which Jamal Khashoggi also wrote himself—was in the English top 30 and RT (previously Russia Today) was in the Arabic top 30.

Second, in both Twitter spheres, some domain names were cited in a positive context, and some in a negative context. In Figure 7, the average polarity scores are plotted for each of the top 30 domains.

Figure 7.

Polarity of URLs mentioned in tweets in the total sample: English and Arabic tweets.

The domain name in the English Twitter sphere that appeared in the most negative context was RawStory (–.391), an independent online tabloid, which was marked as one of the top 30 junk news sources by the Oxford Internet Institute (Marchal et al., 2018). Links to CNN were included mainly in positive tweets in both Twitter spheres. In fact, it was the most positive domain in the entire English top 30 (.499). The domain that was linked in the most positive Arabic tweets was mwso3h. Mwso3h is a platform that showcases paid content and advertisements to the Middle East and North Africa (MENA) region and Gulf Cooperation Council (GCC) countries. This platform is closely linked to “speakol,” a company that designs algorithms to display recommended articles and personalized ads to its users. Finally, the data show that RT, the Russian state-controlled news channel, appeared predominantly in positive Arabic tweets.

Discussion

The present study investigated the Twitter sentiment in different ad hoc public spheres manifested as hashtag communities separated by language around Khashoggi’s murder. A daily sentiment analysis of positive and negative emotions expressed in English and Arabic tweets revealed differences both within and between language-based, ad hoc public spheres. Overtime, the English sphere became more positive; however, this was not the case for the Arabic sphere. Furthermore, while the daily average polarity of tweets in both languages was predominantly negative, the Arabic discourse was more nuanced than the English discourse. We also examined the most popular media URLs that were mentioned in a subsample of all tweets (about 0.15%) and explored their sentiment.

Although the concept of a global public sphere is relatively new and remains widely disputed among scholars (Castells, 2008; Fisk, 2011; Fraser, 2007), the approach of this study sheds new light on this concept. First, previous work addressing the globality of language-based conflictual public discussion on Twitter did not find evidence of a truly global public sphere (Bodrunova et al., 2018). However, such analyses tend to focus on different languages within the same hashtag without considering the same hashtag but in different languages (i.e. جمال_خاشقجي# vs #JamalKhashoggi). Second, Bruns and Burgess’ (2011) notion of ad hoc public sphere does not differentiate between public spheres that are part of a dominant sphere and counter-public spheres (see, for example, Giglietto and Lee, 2017); some hashtags come with associated opinions, values, and affective positions such that they may influence the discourse within these hashtag ad hoc public spheres (e.g. #JeSuisCharlie vs #JeNeSuisPasCharlie). Thus, our approach to Khashoggi’s case using the same neutral hashtags, but in different languages, cannot determine whether an ad hoc public sphere can be considered a dominant sphere, counter-public sphere, or if these spheres reflect different cultural aspects of the same event. More refined research in this domain will be crucial to our understanding of global ad hoc public spheres moving forward.

Consistent with previous studies (Hunt and Gruszczynski, 2021), our analysis shows that Twitter activity spiked in reaction to certain events. However, the Arabic and English spheres did not always have matching activity spikes, implying that they may have been reacting to different events. Taking this finding a step further, the analysis of sentiment of these activity spikes revealed that: first, while there was an activity spike in the Arabic sphere around the time the US Intelligence Report implicated the Crown Prince for the killing of Khashoggi, this event evoked positive rather than negative sentiment. Although unexpected, we speculate that the Arabic Twitter users tried to counter the negative information from the report with positive sentiments (e.g. expressions of solidarity with the Saudi leadership), but it is unclear if this reaction was an orchestrated marketing effort from the Saudi leadership itself. Second, in the Arabic sphere, positive days often come in a row, which is different from the English sphere, where positive days were more scattered throughout time. This pattern might reflect the different conversation and sentiment dynamics between the users and the media, where highly anticipated events (e.g. such as the Crown Prince’s meeting with Khashoggi’s sons) typically generate positive sentiments that tend to spread over days before they decay (Ferrara and Yang, 2015).

We explored the most popular media URLs that were mentioned in a subsample of all tweets (about 0.15%). This exploratory analysis tapped into the role of external references in the construction of ad hoc public spheres and shed light on language-based differences in their use. First, the news media has often been credited for initiating the construction of ad hoc public spheres, such as when Twitter users react to a real-time political situation highlighted by the media (Bruns and Burgess, 2011). The limited mentions of popular media URLs suggest that Twitter users, in both languages, seem to be using the platform to share their thoughts and express their sentiments rather than referring to news stories found on external media. This behavior is consistent with Gruzd et al. (2011) and suggests that reactions to emotionally charged events on Twitter are, in general, more focused on emotional expressiveness rather than on sharing or referring to media stories.

Second, previous studies have indicated that elite media (such as CNN and the New York Times) published news that they deemed newsworthy on Twitter through their own Twitter profiles and their affiliate journalists (Groshek and Tandoc, 2017). The focus of these studies has been on the interactions between journalists and audiences on Twitter without any sentiment consideration, which limits our understanding of how Twitter is used. For instance, looking at the polarity of the tweets with external media references, our findings highlight that in both languages, the New York Times and CNN were predominantly found in positive tweets and Al Jazeera was found in negative tweets. While the polarity was the same in both language spheres, the reasons might be very different due to the outlet’s approach to the murder. For instance, CNN Arabic was a member of the “Defend Saudi” camp compared to CNN English, which was critical of Saudi’s leadership (Abrahams and Leber, 2021). Arguably, CNN portrayed a different reality tailored to its audience (based on language), which explains why it was referred to in positive tweets in both languages. Moreover, while Al Jazeera (Arabic and English) was found in both Arabic and English negative tweets, it could be for different reasons. Al Jazeera (both Arabic and English) is banned in Saudi Arabia and other Arab countries due to the political rivalry between Arab countries and Qatar, where Al Jazeera is based. Al Jazeera Arabic had a very politicized discourse when covering Khashoggi’s murder (AlMomani and Atiyyat, 2020). While the content of Al Jazeera’s news might explain the negative tweets (directed toward the news outlet itself), the English sphere’s reference to Al Jazeera in their negative tweets could be mirroring the sentiment of the media.

Limitations

Several limitations have to be kept in mind when interpreting our findings. First, although Twitter has established itself as a vital global avenue for public expression, studies have indicated that bots are used during Western political events to spread misinformation, diffuse fake news, and manipulate online discussions to redirect the political discourse and the public’s opinions. Second, Twitter users in the Middle East may not share the same freedom of expression that Western Twitter users have. The revolutions in the Middle East have reshaped how Twitter is used and perceived by the Arabic public as well as the Middle Eastern governments (Isani, 2021). Due to the fear of being monitored and tracked by governments on social media, Arabs tend to use different strategies such as self-censorship and sarcasm to indirectly express their opinions online (Salem, 2017). Third, the analysis and extraction of sentiment in tweets in the Arabic language is lagging behind. Most available resources and dictionaries for content analysis are based on Indo-European languages, which cannot be directly applied to Arabic due to its unique script and a variety of dialects (Isani, 2021). More specifically, social media sentiment analysis in Arabic remains problematic due to the lack of resources to accurately identify polarity modifiers, to detect and classify negation and sarcasm, and to determine the use of different dialects and spelling mistakes (Gamal et al., 2018) to name a few. We used NRC, a word-emotion lexicon that does not include other rule-based techniques; the NRC is the closest available resource that allows for a direct comparison between English and Arabic. Finally, Khashoggi’s murder case is just one example for what can be conceptualized as a “state vs state/community” or a “state vs individual” news scenario. Arguably, these scenarios can yield different news salience for different news organizations, particularly if journalists are involved. Future studies should look deeper into how such scenarios can be associated with more pronounced, heinous, and/or sensational news reporting, and henceforth with different cross-cultural sentimentality on social media across borders and hashtag communities.

Conclusion

This study highlights the importance of emotional expression on social media not only within a digital community shaped by a common language but also between different digital communities separated by language. The study’s conceptualization of Khashoggi’s murder as a global trigger event with conflictual aspects coupled with its approach to examining the hashtag ad hoc public spheres in English and Arabic shed light on the ad hoc public sphere’s formation and dynamic overtime. Furthermore, this study emphasizes the importance of shifting the analysis toward using shorter time lags when analyzing sentiment of a global trigger event to allow a closer look at the fast-paced emotional agenda dynamics between news media and public agendas in digital public spheres across different languages.

Supplemental Material

sj-docx-1-nms-10.1177_14614448221137016 – Supplemental material for جمال_خاشقجي# #JamalKhashoggi: Unraveling multilingual Twitter sentiment dynamics in a longitudinal comparative analysis of tweets in Arabic and English

Supplemental material, sj-docx-1-nms-10.1177_14614448221137016 for جمال_خاشقجي# #JamalKhashoggi: Unraveling multilingual Twitter sentiment dynamics in a longitudinal comparative analysis of tweets in Arabic and English by Nour Zeid, Thomas Frissen and Sebastian Scherr in New Media & Society

Footnotes

Authors’ note

Sebastian Sherr is also affiliated to University of Augsburg, Germany.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Nour Zeid

Thomas Frissen

Supplemental material

Supplemental material for this article is available online.

Notes

Author biographies

Nour Zeid (M.A., Texas A&M University) is a Ph.D. student at the Department of Communication & Journalism at Texas A&M University, USA. Her research revolves around media effects with an emphasis on persuasion and influence mechanisms, attitude formation, and attitude change in cross-cultural settings. Her interest spread across health and political domains with a particular interest in the MENA region.

Thomas Frissen is an Assistant Professor in Digital Technology and Society at the Faculty of Arts and Social Sciences, Maastricht University. He specializes in the web’s underbelly and particularly in gloomy digital experiences, cultures, and behaviors. His past research has focused on Salafi-jihadism, right-wing extremism, and communities of Non-Suicidal Self-Injurious behavior. Currently, he works on extreme memes and other forms of disinformation and their spread dynamics in the online information ecosystem. Thomas uses a blend of computational, quantitative, and qualitative social science research methods.

Sebastian Scherr (Ph.D., University of Munich) is a Full Professor and Chair of Digital Health Communication in the Department of Media, Knowledge, and Communication at the University of Augsburg, Germany. His research interests focus on individual and structural susceptibility factors for media effects in the domains of health and political communication, with a special emphasis on digital health, mental health, suicide prevention, and empirical methods.

References

Abrahams

Leber

(2021) Framing a murder: Twitter influencers and the Jamal Khashoggi incident. Mediterranean Politics 26(2): 247–259.

Ali Mohammed

Mccombs

(2021) Intermedia agenda setting or international news flow? Cross-lagged comparison of elite international newspapers. International Journal of Communication 15(2021): 3948–3969.

AlMomani

Atiyyat

(2020) التوظيف السياسي للخطاب الإعلامي في مقتل الصحفي جمال خاشقجي: المواقع الإخبارية لقناة الجزيرة وجريدة واشنطن بوست نموذجا [The politicization of media discourse in the murder of the journalist Jamal Khashoggi: the news websites of Aljazeera and the Washington Post as a model]. Dirasat: Educational Sciences 47(2): 38–52.

Bang

Lee

Rao

(2021) The Egyptian protest movement in the twittersphere: an investigation of dual sentiment pathways of communication. International Journal of Information Management 58: 102328.

Bayer

(2012) SQLAlchemy. In: Brown

Wilson

(eds) The Architecture of Open Source Applications Volume II: Structure, Scale, and a Few More Fearless Hacks, pp. 291–314. Available at: https://aosabook.org/en/sqlalchemy.html

Beckett

Deuze

(2016) On the role of emotion in the future of Journalism. Social Media + Society 2(3): 2056305116662395.

Birkland

(1998) Focusing events, mobilization, and agenda setting. Journal of Public Policy 18(1): 53–74.

Bodrunova

Smoliarova

Blekanov

, et al. (2018) A global public sphere of compassion? #JeSuisCharlie and #JeNeSuisPasCharlie on Twitter and their language boundaries. Monitoring of Public Opinion: Economic and Social Changes 1: 267–267.

Bruns

Burgess

(2011) The use of Twitter hashtags in the formation of ad hoc publics. Available at: https://asset-pdf.scinapse.io/prod/35899105/35899105.pdf

10.

Castells

(2008) The new public sphere: global civil society, communication networks, and global governance. The ANNALS of the American Academy of Political and Social Science 616(1): 78–93.

11.

CBS News (n.d.) Mohammad bin Salman denies ordering Khashoggi murder, but says he takes responsibility for it. CBS News. Available at: https://www.cbsnews.com/news/mohammad-bin-salman-denies-ordering-khashoggi-murder-but-says-he-takes-responsibility-for-it-60-minutes-2019-09-29/ (accessed 4 November 2021).

12.

Ceron

Curini

Iacus

(2016) First- and second-level agenda setting in the Twittersphere: an application to the Italian political debate. Journal of Information Technology & Politics 13(2): 159–174.

13.

Chadwick

(2013) The Hybrid Media System: Politics and Power. Oxford: Oxford University Press.

14.

Chang

T-K

(1998) All countries not created equal to be news: world system and international communication. Communication Research 25(5): 528–563.

15.

Chuang

Roemer

(2013) The immigrant Muslim American at the boundary of insider and outsider: representations of Faisal Shahzad as “Homegrown” terrorist. Journalism & Mass Communication Quarterly 90(1): 89–107.

16.

CNN (n.d.) Jamal Khashoggi fast facts. CNN. Available at: https://www.cnn.com/2019/01/10/middleeast/jamal-khashoggi-fast-facts/index.html (accessed 18 October 2021).

17.

CNN Arabic (2018) “اختفاء في عز الظهر”. . تسلسل زمني لتطورات قضية جمال خاشقجي أولا بأول. CNN Arabic, 19 October. Available at: https://arabic.cnn.com/middle-east/article/2018/10/19/khashoggi-saudi-turkey-crisis-timeline

18.

DW (n.d.) Ugandan blogger’s “life in peril” after arrest in Turkey. DW. Available at: https://www.dw.com/en/ugandan-bloggers-life-in-peril-after-arrest-in-turkey/a-58778226 (accessed 4 November 2021).

19.

El-Falaky

(2019) Murder in the consulate: the grammar of transitivity in the headlines of news reports about Jamal Khashoggi. Advances in Language and Literary Studies 10(3): 11–22.

20.

Ferrara

Yang

(2015) Quantifying the effect of sentiment on information diffusion in social media. PeerJ Computer Science 1: e26.

21.

Fisk

(2011) Global public sphere. In: Chatterjee

(ed.) Encyclopedia of Global Justice. Dordrecht: Springer, pp. 439–441.

22.

Fraser

(2007) Special section: transnational public sphere: transnationalizing the public sphere: on the legitimacy and efficacy of public opinion in a post-Westphalian world. Theory, Culture & Society 24(4): 7–30.

23.

Galtung

(1971) A structural theory of imperialism. Journal of Peace Research 8(2): 81–117.

24.

Gamal

Alfonse

El-Horbaty

E-SM

, et al. (2018) Opinion mining for Arabic dialects on Twitter. Egyptian Computer Science Journal 42(4): 52–61.

25.

Giglietto

Lee

(2017) A hashtag worth a thousand words: discursive strategies around #JeNeSuisPasCharlie after the 2015 Charlie Hebdo shooting. Social Media + Society 3(1): 2056305116686992.

26.

Gilkerson

Berg

(2017) Social media, hashtag hijacking, and the evolution of an activist group strategy. In: Jin

Austin

(eds) Social Media and Crisis Communication. New York: Routledge, pp. 141–155.

27.

Golan

(2006) Inter-media agenda setting and global news coverage: assessing the influence of the New York Times on three network television evening news programs. Journalism Studies 7(2): 323–333.

28.

Grasland

(2020) International news flow theory revisited through a space–time interaction model: application to a sample of 320,000 international news stories published through RSS flows by 31 daily newspapers in 2015. International Communication Gazette 82(3): 231–259.

29.

Groshek

Tandoc

(2017) The affordance effect: gatekeeping and (non)reciprocal journalism on Twitter. Computers in Human Behavior 66: 201–210.

30.

Gruzd

Doiron

Mai

(2011) Is happiness contagious online? A case of Twitter and the 2010 Winter Olympics. In: 2011 44th Hawaii international conference on system sciences, Kauai, HI, 4–7 January, pp. 1–9. New York: IEEE.

31.

Guo

Vargo

(2017) Global intermedia agenda setting: a Big Data analysis of international news flow. Journal of Communication 67(4): 499–520.

32.

Hopke

(2015) Hashtagging politics: transnational anti-fracking movement Twitter practices. Social Media + Society 1(2): 2056305115605521.

33.

Hunt

Gruszczynski

(2021) The influence of new and traditional media coverage on public attention to social movements: the case of the Dakota Access Pipeline protests. Information, Communication & Society 24(7): 1024–1040.

34.

Isani

(2021) Methodological problems of using Arabic-language Twitter as a gauge for Arab attitudes toward politics and society. Contemporary Review of the Middle East 8(1): 22–35.

35.

Jahng

Doshi

(2021) Securitising an international humanitarian crisis: news framing of the Syrian refugee crisis in the US. The Journal of International Communication 27(2): 283–299.

36.

Jamal

Keohane

Romney

, et al. (2015) Anti-Americanism and anti-interventionism in Arabic Twitter discourses. Perspectives on Politics 13(1): 55–73.

37.

Kepplinger

Brosius

H-B

Staab

(1991) Instrumental actualization: a theory of mediated conflicts. European Journal of Communication 6(3): 263–290.

38.

Kirkpatrick

Gall

(2018) Audio offers gruesome details of Jamal Khashoggi Killing, Turkish official says. The New York Times, 17 October. Available at: https://www.nytimes.com/2018/10/17/world/europe/turkey-saudi-khashoggi-dismember.html

39.

Kovács

Kovács-Győri

Resch

(2021) #AllforJan: how Twitter users in Europe reacted to the murder of Ján Kuciak—revealing spatiotemporal patterns through sentiment analysis and topic modeling. ISPRS International Journal of Geo-information 10(9): 585.

40.

Kursuncu

Gaur

Lokala

, et al. (2018) Predictive analysis on Twitter: techniques and applications. In: Agarwal

Dokoohaki

Tokdemir

(eds) Emerging Research Challenges and Opportunities in Computational Social Network Analysis and Mining. Springer International Publishing, pp. 67–104. Available at: https://doi.org/10.1007/978-3-319-94105-9_4

41.

Livingston

Asmolov

(2010) Networks and the future of foreign affairs reporting. Journalism Studies 11(5): 745–760.

42.

McCombs

Llamas

Lopez-Escobar

, et al. (1997) Candidate images in Spanish elections: second-level agenda-setting effects. Journalism & Mass Communication Quarterly 4: 703–717.

43.

McCombs

Shaw

Weaver

(2014) New directions in agenda-setting theory and research. Mass Communication & Society 17(6): 781–802.

44.

Marchal

Neudert

L-M

Kollanyi

, et al. (2018) Polarization, partisanship and junk news consumption on social media during the 2018 US midterm elections. Available at: https://demtech.oii.ox.ac.uk/wp-content/uploads/sites/127/2018/11/marchal_et_al.pdf

45.

Milanovic

(2020) The murder of Jamal Khashoggi: immunities, inviolability and the human right to life. Human Rights Law Review 20(1): 1–49.

46.

Miller

(2007) Examining the mediators of agenda setting: a new experimental paradigm reveals the role of emotions. Political Psychology 28(6): 689–717.

47.

Mohammad

Turney

(2013) Crowdsourcing a word-emotion association lexicon. Computational Intelligence 29(3): 436–465.

48.

Naskar

Singh

Kumar

, et al. (2020) Emotion dynamics of public opinions on Twitter. ACM Transactions on Information Systems 38(2): 1–24.

49.

Nikolayenko

(2019) Framing and counter-framing a Peace March in Russia: the use of Twitter during a hybrid war. Social Movement Studies 18(5): 602–621.

50.

O’Boyle

(2019) Twitter diplomacy between India and the United States: agenda-building analysis of tweets during presidential state visits. Global Media and Communication 15(1): 121–134.

51.

O’Boyle

Pardun

(2021) How Twitter drives the global news agenda: tweets from Brazil, Russia, India, China, the UK and US and online discourse about the 2016 US presidential election. Global Media and Communication 17(3): 363–384.

52.

Öztürk

Ayvaz

(2018) Sentiment analysis on Twitter: a text mining approach to the Syrian refugee crisis. Telematics and Informatics 35(1): 136–147.

53.

Qin

(2015) Hero on Twitter, traitor on news: how social media and legacy news frame Snowden. The International Journal of Press/Politics 20(2): 166–184.

54.

Rastogi

Bansal

(2021) Visualization of Twitter sentiments on Kashmir territorial conflict. Cybernetics and Systems 52(8): 642–669.

55.

Roberts

Mccombs

(1994) Agenda setting and political advertising: origins of the news agenda. Political Communication 11(3): 249–262.

56.

Romney

Jamal

Keohane

, et al. (2021) The enemy of my enemy is not my friend: Arabic Twitter sentiment toward ISIS and the United States. International Studies Quarterly 65(4): 1176–1184.

57.

Russell Neuman

Guggenheim

Mo Jang

, et al. (2014) The dynamics of public attention: agenda-setting theory meets Big Data. Journal of Communication 64(2): 193–214.

58.

Salem

(2017) Social media and the Internet of Things: towards data-driven policymaking in the Arab world (No. 7). Available at: https://mbrsgcdn.azureedge.net/cmsstorage/mbrsg/files/13/1383b88a-6eb9-476a-bae4-61903688099b.pdf

59.

Sanderson

Barnes

Williamson

, et al. (2016) “How could anyone have predicted that #AskJameis would go horribly wrong?” public relations, social media, and hashtag hijacking. Public Relations Review 42(1): 31–37.

60.

Silge

Robinson

(2016) tidytext: text mining and analysis using tidy data principles in R. JOSS 1(3): 37.

61.

Thorsen

Sreedharan

(2019) #EndMaleGuardianship: women’s rights, social media and the Arab public sphere. New Media & Society 21(5): 1121–1140.

62.

Twarc (2021) Twarc2. Available at: https://twarc-project.readthedocs.io/en/latest/twarc2_en_us/

63.

Vliegenthart

Walgrave

(2008) The contingency of intermedia agenda setting: a longitudinal study in Belgium. Journalism & Mass Communication Quarterly 85(4): 860–877.

64.

Wahl-Jorgensen

(2019) Questioning the ideal of the public sphere: the emotional turn. Social Media + Society 5(3): 2056305119852175.

65.

Wang

Wei

Liu

, et al. (2011) Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach. Proceedings of the 20th ACM International Conference on Information and Knowledge Management, 1031–1040. Availabel at: https://doi.org/10.1145/2063576.2063726

66.

Weimann

Brosius

H-B

(2017) Redirecting the agenda: agenda-setting in the online era. Agenda Setting Journal: Theory, Practice, Critique 1(1): 63–101.

67.

Xanthopoulos

Panagopoulos

Bakamitsos

, et al. (2016) Hashtag hijacking: what it is, why it happens and how to avoid it. Journal of Digital & Social Media Marketing 3(4): 353–362.

68.

Zarocostas

(2018) Physician alleged to have taken part in Khashoggi murder. The Lancet 392(10158): 1613.

69.

Zonis

Joseph

(1994) Conspiracy thinking in the Middle East. Political Psychology 15(3): 443–459.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB