YouTube based religious hate speech and extremism detection dataset with machine learning baselines

1 Introduction

Social media platforms have helped natural language processing researchers to tackle a variety of problems. The data provided by the platforms can serve to evaluate user-related issues online i.e sexism [10], fake news [5] and emotions [3]. While the users discuss different topics, religious discourse is also seen widely. Social media gives the authority to users to express themselves freely, however, many of the users take advantage of the freedom of speech and use the medium for abuse and hate speech.

Hate speech in the text can be identified as a deliberate attack, an attempt motivated by aspects of the group identity or directed towards a specific group of people [14]. The religious form of hate speech is very important to overcome as these platforms have high reachability and popularity and can help extremist organizations in planning and mobilizing events for extremism and protests  [30]. These websites are also used by extremist groups to spread their ideologies and agendas among their viewers [1]. Content related to hate speech can create more impact with time as it gets more viewership and attention [17]. Hence, early detection of religious forms of hate speech is of immense importance.

Social networking websites such as YouTube are sometimes left unchecked as they require an understanding of the context in the form of videos, text and audio.

Many research articles [13, 41] suggest a trend of extremist content (videos, text, images) are posted in the form of groups. The study [13] gave evidence of many Jihadi videos from Iraq and their links to users on Youtube. Their majority of viewers (under 35 years) belonged to the United States and outside the region of the Middle East and North Africa (MENA). A pattern of this behaviour is replicated in other extremist organizations over the years. Hence, it is important to find novel techniques to identify religious hate speech content through one of the most consumed content websites existing today.

In this research, our focus is on the comments that are posted on religious videos on YouTube. We aim to detect those videos and comments on YouTube that are promoting hatred among individuals and groups. Furthermore, we investigate whether comments are used for initiating hatred among people from a different religion. First, we extracted videos and embedded comments within the videos. After that, data mining techniques [25] were used on the extracted comments. We collected more than 100 YouTube videos and extracted comments from these videos. We discovered that most of the comments were written in English. Extracted comments were then distributed in positive and negative classes based on attributes and features that are basic for hate speech. The paper also discusses the defined rules for the distribution of comments into two classes. Finally, we used machine learning classifiers such as Support Vector Machines, Logistic Regression, and k-Nearest Neighbor to provide baseline results.

The remaining paper is organized as follows: Section 2 discusses the detection of hate speech in websites, newspapers and social networking sites. Section 3 methods discusses the methodology and section 4, highlights the performance of our proposed method. Finally, Section 5 conclusion concludes the findings of the paper.

2 Related work

Natural language processing researchers have tackled many challenging problems in text with the aid of statistical and deep learning methods such as sentiment analysis [21], emotion detection [3], human behavior detection [6], fake news detection [5], question answering [9], depression and threat detection [4, 28].

We have seen researchers in history targeting a religious aspect of hate speech. A study [2] introduced a religious hate speech dataset for Twitter in the Arabic language. The purpose of the study was to distinguish religious targeting from profanity. This proved challenging considering the Arabic morphology and limited resources. The authors used pre-trained word embeddings with Recurrent Neural Network (RNN) architecture and Gated Recurrent Units (GRU) to provide baseline scores. Similarly, another study [38] gathered data from Yahoo and American Jews Congress (AJC) and defined hate speech in their work. The authors used paragraph-level classification and manually annotated the dataset in seven categories. The baseline results were presented using SVM with a linear kernel function. They used tenfold cross-validation to obtain an overall accuracy of 96%, precision 59%, recall 68% and f1-score of 63% . The community has also worked in developing a method [22] for detecting hate speech against the black community. Motivated by work done in [38], a Twitter-based dataset was developed. The severity of arguments was judged by floating a questionnaire to students of different races. The dataset contained a training dataset of 24,582 tweets which was pre-processed and classified using a Naive Bayes (NB) classifier. The classifier showed an accuracy of 86% .

In the early days of hate speech detection on text, a study [36] composed a generic hate speech identification system to identify hate phrases from websites. Initially, they used the NB classifier for selecting the most discriminative features, following a Multinomial Updatable Naive Bayes (MNB) classifier was used to produce labelled sentences. Finally, the Decision Table/Naive Bayes hybrid (DTNB) classifier was used to obtain an accuracy of 68% by using tenfold cross-validation. Studies for identifying hate speech in the respective social settings using Facebook [35], Twitter [32] and Youtube [12, 34] were frequently seen. In one of the influential studies, authors presented a framework [12] for the identification of extremist videos on YouTube. They extracted syntactic, lexical, and content-specific features from user-generated data and used various feature-based classification techniques to classify videos. Another group of researchers [1] designed a crawler-based approach for retrieving YouTube user profiles promoting extremism and hate. However, our approach in this work is different as the related studies are more generic whereas our study has a distinct emphasis on the religious form of hate speech.

Over the years, several machine learning methods are reported for tweet classification of hate speech or antagonistic focusing racism, ethnicity and religion. A study [8] that observed the murder of a drummer, Lee Rigby in Woolwich, UK used a context-free lexical model with the Stanford Lexical Parser to conduct typed dependencies in phase one and Bag of words (BoW) as a feature in phase two in tweets. They experimented with the machine learning classifiers such as Decision Trees, Random Forest, SVM etc and achieved 95% f1-measure using tenfold cross-validation. Another study categorized opinions first and then found their sentiment polarity by using machine learning classifiers (SVM, NB and k-nearest neighbors (k-NN)) on tweets [42]. The study showed SVM to have superior results than NB and k-NN. Researchers have developed multiple individual models [7] for the classification of cyberhate for characteristics such as race, disability and sexual orientation. They used text parsing for extracting typed dependencies. Moreover, they developed a model for cyberhate to improve classification when multiple characteristics are attacked such as race and sexual orientation.

In the recent studies, we see the trend of methodologies being changed for machine learning and feature-based methods to pre-trained transformer models [15, 33], deep learning [11, 40] and graph-based approaches. Researchers [37] have identified hate in memes successfully using transformer models such as LXMERT, VILLA, ERNIE-Vil, Visual BERT, ViLBERT, VLP, UNITER. Deep learning algorithms such as Bi-LSTM and CNN have been successful in many other NLP applications. An approach [40] for hate speech using CNN and Bi-LSTM gave us micro-F1 scores of 80% and 69% . A GRU and RNN based approach was applied on Greek-Gazzetta-Corpus [39], whereas, a graph embedding approach [11] was also applied using whole graph embeddings and achieved an F1-score of 89.16% through Graph2vec.

3 Methodology

The proposed methodology as shown in Fig. 1, is composed of two phases (training and testing phase). Four tasks are performed in the training phase: videos and comments collection, feature representation, storing the dataset and training classifier. Whereas, five tasks are performed in the testing phase: videos and comments collection, feature representation, storing the dataset, hypothesis prediction and evaluation. Each task is further elaborated on next.

OPEN IN VIEWER

Fig. 1

Block diagram of proposed methodology.

For experiments, we used WEKA [18]. WEKA is a popular machine learning tool that offers techniques for tokenization, stop words removal, attribution selection, feature weighting, regression, classification, clustering and modelling algorithms. The dataset is first transformed into feature representation to train and evaluate SVM, LR and k-NN classifiers. The performance of three classifiers for five religions is shown in Tables 5, 6 and 7. The performance of SVM for five religions shows that the proposed method can classify tweets with a precision of 84.70% and recall of 89.60% . The highest obtained precision was for Islam and Christianity, 84.70% and 84.00% respectively. Islam and Christianity have the highest number of tweets as compared to other religions. That is why the precision is high for Islam and Christianity. In SVM performance, the recall was high than precision for Christianity. The same pattern is also found in the result of LR and k-NN (for Islam). For such cases, the recall was high as the classifier was able to correctly classify most of the positive tweets in the dataset. On the other hand, a low precision value indicates that SVM classified most of the noisy (irrelevant and neutral) tweets into positive classes.

Overall, SVM performed better for comments classification as compared to LR and k-NN due to the following reasons: (1) It has a robust internal over-fitting mechanism which enables it to perform better for high dimensional data [27], (2) It relies on a small number of examples, and (3) It can produce accurate results on a sound theoretical basis, even in cases when the input data are non-monotone and non-linearly separable.

For accuracy comparison of three classifiers, paired t-test (corrected) is performed in WEKA. Statistical paired t-test compares two datasets in which observations in one dataset can be paired with other dataset observations. The main objective of this test is to investigate the statistical evidence that the mean difference from paired observations from two datasets on a particular outcome is significantly different. More detail on paired t-test can be found in [19]. Classifiers comparison for 5 religions is shown in Table 8. LR is selected as baseline classifier and each classifier was run 10 times and obtained accuracy is the mean and the standard deviation in rackets of those 10 runs.

For three religions (Islam, Hinduism and Christianity), SVM performed better as compared to LR and k-NN. On average, SVM achieved a classification accuracy of 82.6%, LR 82.5% accuracy and k-NN 80.06% accuracy. Obtained results indicate that some differences in accuracy of SVM, LR and k-NN exist. However, differences in classifiers accuracy are not statistically significant. This research will be instrumental in monitoring the YouTube videos by eliminating hate speech using public sentiments. Public sentiments are a great source of information to detect hate speech. Moreover, we have seen exponential growth in video traffic recently [29], therefore detection of hate speech on YouTube using public sentiments is necessary.

Social Networks are most widely used to express one’s feelings and thoughts. Individuals of various religions, races, and nationalities express their right to speak freely about problems on social networking sites. However, many users use this free will for malicious purposes. They make individuals against each other. Religion is the most sensitive issue discussed worldwide on social networks. To avoid hate speech, regulating authorities of these social networks are required to find automated mechanisms to identify and remove such videos from YouTube. In this paper, we have selected the most sensitive issue “religious hate” and put up a methodology that can filter religious opinions and then can highlight hate speech from those religious views. We have used supervised learning algorithms SVM, LR and k-NN to show the effectiveness of the methodology. These algorithms were used to classify video comments. For comments classification, SVM performed better than LR and k-NN.