“Bend the truth”: Benchmark dataset for fake news detection in Urdu language and its evaluation

1 Introduction

Even though the Urdu language has more than 100 million speakers across the world, it is a resource poor languages in the Natural Language Processing (NLP) domain both from the perspective of NLP tools inaccessibility as well as scarcity of labeled datasets  [1]. In this work, we dedicate our attention to assemble a plausible and credible source in the form of Urdu corpus for automatic fake news detection.

In digital media, the epidemic of fake news grows substantially when a change in public opinion is demanded during an important event. Hence, we need to tap natural language processing algorithms to design a system that can determine whether a source is trustworthy or politically inclined with or without human curation. For example, in January 2019 Google showed incorrect Pakistani rupee value against US dollar (exchange price of the dollar) in Pakistan  1 , the following day stock market in Pakistan was crashed because people started to sell their shares due to the dramatic decline of the stock exchange.

Fake news is painting significant challenges to branch out our society. The availability of information has raised the challenges associated with testing the trustworthiness of the data automatically. For this reason, it is necessary to build systems for controlling the amount of factually incorrect and misleading data on the Web. This deficit can be met by designing computational models for detecting fake news. In turn, this requires sufficient amount of labeled data to apply supervised machine learning approaches. As fake news dissemination can be cross-lingual, it is best to have datasets available in a wide variety of languages.

Therefore, we present a labeled dataset for fake news detection in Urdu language. It contains 500 labeled real news from legitimate news sources and 400 fake news in the corresponding topics: (i) Business, (ii) Health, (iii) Showbiz, (iv) Sports, and (v) Technology.

Additionally, we provide baseline classification methods for fake news detection on this dataset. There are three categories of fake news detection methods [3]: knowledge-based (attempt fact verification), context-based (analyze how the news disseminate in social networks), and style-based (analyze writing style). The problem with implementing the first two approaches for the Urdu language is the unavailability of the NLP tools required for their intermediate feature crafting. However, the style-based approach in its basic form is based on analyzing n-gram sequences.

The main contributions of this work are:

the first corpus for the Urdu language for research on automatic fake news detection containing real news extracted from various legitimate news agencies and fake news written by professional native Urdu-speaking journalists;

The rest of the paper is composed as follows. Section 2 overviews the state-of-the-art work on fake news detection and corpora for other languages. Section 3 describes the methodology we followed for building the corpus along with the annotation guidelines and the corpus statistics. Section 4 describes the classification approach for automatically detecting fake news. In Section  5 we analyze the experimental results. Subsequently, Section 6 present general conclusions and points to the permissible steps of future work.

2 Related work

In this section, we review the literature regarding the automatic analysis of fake news which has been a subject of particularly acute attention. The presence of fake news started with the invention of printing press back in 1439  2 . However, there are divergent opinions in defining the term “fake news.”

Definition 1. Fake News:Fake news is a news article that is intentionally and verifiably false [3].

In recent times, there are only two main directions of research to automatically classify fake news: on a conceptual and an operational level. On a conceptual level, fake news have been further divided into three categories [4]: hoaxes, i.e., posting factitious information using social networks alluding to certain news broadcast in its genuine form via reputable news websites; satire, i.e., news that imitate the real content of news with addition of untrue and sarcastic content; and serious fabrications, i.e., misleading news about a celebrity or an event that did not take place.

On an operational level, researchers [5] suggested different approaches, such as an inference task in a Markov random field (MRF) [6], fact-checking, and source-checking. Moreover, fake news detection and deception detection has been used in several studies as a data mining [3] to classify news pieces, posts, and online reviews in publicly available corpora [7, 8].

Fake news pieces contain dogmatic and seditious language to urge users to click on the link to read the full article (known as “clickbait”)  [9]. Thus, in the fake news detection task, the linguistic features have been used to capture the different writing styles in news content and sensational headlines [7]. Additionally, social networks, in particular Twitter posts associated with natural disasters have been used for the development of fake news detection model [10].

Linguistic-based features are derived from language to examine various aspects of a language at different levels such as characters, words, sentences, and documents as a whole. There are two primary types of features: common and domain-specific linguistic features. Common linguistic features contain two kind of features: (i) lexical features and (ii) syntactic features. Lexical features including character level and word-level features, such as a total number of words, an average number of characters per word, frequency of words present in the dataset, unique word count, frequency of function words and phrases, parts-of-speech (POS) tags, etc. Syntactic features include sentence-level features such as syntactic dependencies/constituents, clauses, and punctuation. Ivanov and Tutubalina [11] use syntactic clause features for user review analysis.

Domain-specific linguistic features, which are precisely aligned to news domain, are quoted words, external links, number of images, etc. [2]. Furthermore, to find out the deceptive cues in writing styles to flag fake news, features such as author lying-detection features and different types of new features can be created [12].

The reason to attain features is to outline the content of news items mathematically. Model-oriented fake news research opens the door to developing more robust models for fake news detection. Recent studies  [2, 13] have suggested different approaches by focusing on extracting several kinds of features and integrating them into supervised classification models such as logistic regression(LR), k-nearest neighbours (kNN), random forest (RF), and support vector machines (SVM), and after that choosing the classifier that outperform other machine learning algorithms.

A recent study suggested a multi-task ordinal regression framework that models the problem of trustworthiness and political ideology detection of entire news content jointly instead of analyzing each news article individually. Furthermore, this study also revealed that joint models over models that tackle the problems separately obtained significantly better results [14].

Additionally, fake news detection has been investigated as a stance detection problem rather than true/false classification [15, 16]. In particular, this approach was adopted in the Fake News Challenge project (FNC1) 3 which reduces problem to checking the relationship between the title and the body of the news: a) the title and the body are clearly related, b) no association between the title and the body, and c) a partial relationship.

The winning team (best performing system) achieved 82.02 accuracy score using machine learning and deep learning approaches [17]. For machines, 2-gram and 3-gram features with TF-IDF weighting scheme using Gradient-Boosted Decision tree were used. For deep learning, word level vectors using word2vec embeddings from Google News [18] were applied using a one-dimensional deep convolutional neural network (CNN) on the title and the body text.

News articles can be accumulated using different online sources, such as news agency homepages, search engines, and social networks. Despite this, manually checking the authenticity of a news article requires annotators with domain expertise who conduct deliberate analysis of claims. For fake news detection, datasets in English and Spanish are available. In English the datasets are available, such as BuzzFeedNews [2], BS Detector 4 , Liar [19], CREDBANK [20] and FakeNewsNet [21]. Likewise, Fake News Corpus Spanish [13] annotated for fake news detection in Spanish. A corpus of social network news feed paraphrases exist for the Russian language, yet it is not annotated for content authenticity [22]. However, to the best of our knowledge, there is still no such resource available in the Urdu language despite the tremendous advancement in research work for this language.

EMILLE 5 Project (Enabling Minority Language Engineering) was the first initiative to assemble a 67 million word corpus of South Asian languages [23]. The Urdu corpus which was collected within this project contained approximately 0.5M spoken Urdu words transcribed from transmissions of BBC Asian Network and BBC Radio. Subsequently, researchers started to make attempts to build resources for resource-poor language, such as the Urdu corpus for word sense disambiguation [24], the Urdu POS-tagged corpus [25], and the initiative of phonetically rich Urdu corpus for speech recognition [26]. However, these resources do not have annotation suitable for fake news detection.

3 The data: the first corpus of fake news in Urdu

In this section, we provide an overview of the data acquisition process as well as the corpus statistics. We assembled real news by crawling thousands of news articles from numerous reliable sources for the time frame from January 2018 to December 2018.

This corpus contain news from five domains: (i) Business, (ii) Health, (iii) Showbiz (entertainment), (iv) Sports, and (v) Technology. This selection of topics is in line with a similar dataset for English language [7] except for the educational domain which presented difficulties in obtaining.

Previous study [2, 27] has almost exclusively focused on providing a more detailed analysis of procedures on how two types of news (real and fake) are collected. It also discussed serious issues associated with fake news corpus. Moreover, some news corpora contain news articles which are a combination of real and fake information. As far as we know, no previous research has illuminated the rigorous criteria for fake news definition and categorization. Although researchers have examined different types of news in creating a corpus, some questions regarding the exact procedure of how they annotated the news pieces remain to be addressed. With this in mind, we introduced an alternative approach to data collection to address this limitation in fake news annotation and applied it to the Urdu language.

This “Bend The Truth” corpus is a unique, reasonably accurate, and reliable source of its kind in the Urdu language for this particular task. Urdu is a national language of Pakistan. This is a binary annotated corpus. The uniqueness about this corpus apart from its language, is that we availed professional journalist services to write fake news stories corresponding to the original real news, just as what takes place in the real life. News agencies used to crawl real news are mentioned in Table 1. The “Bend The Truth” Urdu corpus is publicly available to use for academic research  6 .

In this section, we describe a series of experiments on automatic fake news detection set as a binary classification problem (real or fake). We explore various combinations of feature sets, look at different feature value weighting schemes (scalers), and try out a number of classifiers. This is done to find a best performing baseline classifier for the assembled dataset.

5 Result analysis

This section presents the analysis of the experimental results and provides recommendations for the set of features, weighting schemes, and classifiers based on the best performing combinations of those.

We present the results as top 10 best performing experimental combinations by F1_Real score in Table 6, top 10 best performing experimental combinations by F1_Fake score in Table 7, and the experiments that are best performing by both F1_Fake score and F1_Real score, i.e., those experimental combinations that achieve highest performance both in detecting fake news as fake and real as real, in Table 8.

As it can be seen in the tables, all best performing experiments achieved performance well above the trivial single-class baseline of 0.55, which indicates that the task of fake news detection can be effectively addressed using n-gram features. The maximum F1_Fake score was achieved by AdaBoost (often called boosted decision tree) classifier which outperformed other classifiers with the peculiar combinations of character-word 2-grams and 1-grams (i.e., 2c-1w-0f) by providing 0.87 F1_Fake score on the test set.

Feature Combinations. Our experiments covered combinations of n-grams ranging in size from 1 to 6. We observed that performance decreases with the increase of n-grams size. Apparently, this is due to the exponential growth of the feature space dimension. For example, for a feature combination 4c-4w-4f, the total features are 22548, and for the feature combination 6c-6w-6f the total features are 37208. Our training dataset size, 638 instances, is not enough to train a classifier in such highly dimensional feature spaces. Therefore, we omit the results for n-grams sizes 4 and higher as they are inferior to the performances for smaller n-grams.

Figure  1 illustrates ROC-curves for the experimental combinations from Table 8.

OPEN IN VIEWER

Fig. 1

ROC curves for best performing experimental combinations.

Additionally, both Tables 6 and  7 contain feature sets that have only one type of n-gram, e.g., 2c-0w-0f includes only character bi-grams, two types, e.g., 2c-1w-0f and 2c-0w-1f, and all three n-gram types jointly. In addition, while character n-grams can lead to top results individually, there’s no feature combination where it would be absent. Hence, we conclude that character n-grams are the most descriptive features for fake news detection in Urdu. Further, as the tables show, their text representation can be enhanced by either of word n-grams or function word n-grams or both. it was also noticed during experiments that the classifiers without function words did not give better results compared with experiments using stop-words. Therefore, in the course of our experiment, stop-words played an important role.

Weighting Schemes. We also considered a number of weighting schemes for feature values. Figure 2 shows performance distribution for the weighting schemes. It can easily be noticed that certain weighting schemes are performing consistently better than the others. Others, such as TF (raw frequency) and TF-IDF are not showing good results. We observed that when using global weighting schemes such as TF-IDF and log-entropy, the classification performance decreases significantly in the majority of the experiments. Global weighting functions measure the importance of an n-gram across the entire collection of documents by reducing the weight of common features and highlighting the uncommon words. For the particular case of fake news detection, global weighting schemes majorly failed. It indicates that common words (character and function words n-grams) provide more relevant information to the classifier (see Tables 6 and 7).

OPEN IN VIEWER

Fig. 2

Weighting scheme performance in terms of F1-score distribution. Y-axes show experiment counts.

From these distributions as well as from the best performing experimental combinations presented in Tables 6 and  7, we conclude that binary and normalized frequency weighting schemes are most useful in the majority of cases. However, as it can be seen from those tables, other weighting schemes are involved in highly performing runs for some feature combinations.

Classifiers. In the Figure 3, the performance of different classifiers during Our experiments has shown. Tables 6 and 7) results have displayed that some classifiers did a splendid work in differentiating legitimate news content, however, some classifiers presented poor performance. Particularly, we observe that classifiers such as Adaboost indicated consistently good performance across domains. DecisionTreeClassifier with maximum depth equal to 1 was chosen as a AdaBoost base. Similarly, bootstrap equal to true was selected as a particular parameter for Random forest trees. Linear kernel was tested with SVM only, because for the sake of time and space and we saw that the linear was enough for the baseline experiments.

OPEN IN VIEWER

Fig. 3

Classifiers Performance.

Bayes based classifiers showed inferior results to AdaBoost, we haven’t performed much analysis on the feature distribution (which is actually our future work and will be done in an upcoming paper). To our understanding, these both classifiers are suitable for discrete data such as counts and we ran experiments on both binary features as well as counts to give best settings for each of the classifiers respectively. However, they still showed inferior results compared to AdaBoost in most of the experiments — beyond the paper scope

To gain further insights into the classes that are associated with fake and real content, we evaluate which classes show significant differences between the two groups of news.

The reflective observation from the experiments leads to the following conclusions:

Combinations of different n-gram types obtain better results instead of single n-gram type.

The paper concludes by arguing that the automatic detection of fake news is a promising area of research. In this research, a new resource for poor resource languages in the form of a dataset in Urdu language is presented and build a model that can correctly prognosticate the likelihood that a given news article is fake news. To our knowledge, this is the first corpus in the Urdu language for fake news detection, extracted from the internet and annotated manually containing real or fake news.

This is an essential contribution to the development of the Urdu corpus. Importantly, we provide statistics of the complete corpus, casts light on vocabulary size, vocabulary overlap and significant findings by the analysis. The present results confirm with the implementation of machine learning classifiers on lexical features BOW, n-grams (with n varying from 1 to 6), and in combination with n-grams methods. Overall, our results demonstrate the broad implication of the present research by obtaining promising results. On this basis, the main conclusion that can be drawn is that the addition of the new Urdu corpus is a particularly fruitful, reliable resource for the further development of fake news detection models.

In the future, we intend to explore whether the system can be adapted to other languages (it was trained exclusively on Urdu) and whether it can be trained to detect region-specific biases. We will also investigate new features to flag fake news.