Hybrid approach for text categorization: A case study with Bangla news article

Abstract

The incredible expansion of online texts due to the Internet has intensified and revived the interest of sorting, managing and categorising the documents into their respective domains. This shows the pressing need for automatic text categorization system to assign a document into its appropriate domain. In this article, the focus is on showcasing the effectiveness of a hybrid approach that works elegantly by combining text-based and graph-based features. The hybrid approach was applied on 14,373 Bangla articles with 57,22,569 tokens collected from various online news corpora covering nine categories. This article also presents the individual application of both the features to explicate how they generally work. For classification purposes, the feature sets were passed through the Bayesian classification methods which yield satisfactory results with 98.73% accuracy for Naïve Bayes Multinomial (NBM). Also, to test the robustness and language independency of the system, the experiments were performed on two popular English datasets as well.

Keywords

Bangla texts graph-based feature hybrid feature Naïve Bayes multinomial text categorization text-based feature

1. Introduction

Text classification (TC) is evolving rapidly and is one of the major fields of research in Natural Language Processing (NLP). It is widely studied in the current era when numerous data are available online, in many languages around the world. It is a supervised categorization process, where text documents are automatically classified into predefined categories. To deal with the problem of manual indexing or sorting a large volume of increasing data in day-to-day life, there is a great necessity for developing an automatic text classification system. A TC can be of various types: binary (classifying the texts into one of the two domains), multi-class (classifying the texts into one domain and the possible number of domains in the set are more than two) and multi-labelled (classifying the texts into more than one domains) and can play a prime role in a wide array of areas including text information retrieval [1], sentiment analysis [2], word sense disambiguation [3], question-answering systems [4] as well as other applications involving text document organisation. Categorization often depends on a domain for which texts are predefined, and relationships are identified based on the terms, the text documents consist of.

Within the past several years, the Internet has begun to dominate the landscape of computing. The exponential growth of the Internet has led to a massive increase in digital text documents. There is a pressing need for efficient tools and software to help users searching for useful information. This has paved the way for text classification that assigns the documents to their respective text categories. In essence, text document classification or text categorization is one of the core technologies having a strong impact on the development of language engineering as well as having strong application relevance in the commercial world. It is associated with many challenges and difficulties relating to both the text and the content of the documents. First, it is observed that there are difficulties in dealing with the semantics and abstract concepts of natural language texts from a few sets of keywords. Second, high dimensionality and variable length of text documents, along with varied content and quality, create hurdles in achieving efficiency and accuracy of an automatic text classification system. Although a considerable amount of work is present in literature in English and other advanced languages, the case is not the same for Indian languages, especially for Bangla. One reason for this is the inherent complexity of Bangla which is accompanied by the unavailability of standard datasets. Bangla, a language of the Indo-European family, is the national language of Bangladesh and the state language of West Bengal, India. It is written using Bangla script, a unique writing system that is also used for the Assamese, another language of India. As far as the information on the Internet is available, it is the sixth most spoken language (population-wise) worldwide and third most spoken language in India [5]. This creates a pressing need to organise Bangla text documents effectively so that users can easily recognise required or related documents. Automatic Bangla text categorization system – that the present work desires to develop – will help the language to move little far towards the state of digitisation as well as the urgent needs of the Bangla text users who are waiting eagerly for a reliable system for retrieving useful information in this language will be addressed.

2. Our contributions

We proposed a tool that can classify Bangla text documents. The major contributions of the work are summarised as follows.

The Bangla text document classification has not been well studied (Section 3). In the proposed technique, we have combined the two features (text-based and graph-based features) for the extraction of features. Furthermore, we have implemented Naïve Bayes Multinomial – an extended version of standard Naïve Bayes algorithm – that has widely been studied for classification of English text documents but not for Bangla.

Since there are no publicly available datasets in Bangla, we developed our datasets from several online news corpora (see Section Data collection). The dataset will be provided for research purposes (upon the request).

Having an automatic Bangla text categorization tool can address the need for retrieving useful information from the digital Bangla text documents for Bangla text users.

The remaining part of this article is sketched in the following manner: Section 3 refers to the related works on text classification; in Section Proposed methodology, the proposed methodology obtained for attaining the task have been discussed; Section Result and analysis casts lights on the results achieved using the classifier being chosen along with the statistical significance test as well as comparison with existing methods, and finally Section Conclusion and future work draws conclusion along with the future scopes.

3. Related works

TC is elaborately researched and gained attention for several resourceful languages especially in English, Chinese, and Arabic; but very few works have been done for Indian languages especially in Bangla. Some of the recent works have been discussed. Parlak and Uysal [6] performed an exhaustive comparison for categorising the abstracts obtained from Turkish and English medical journals. They extracted the features using three methods: uni-gramme, bi-gramme and hybrid (combination of uni-gramme and bi-gramme). Then, they have used three distinct feature selection approaches and seven classification algorithms in their work. The experimental results show that the unigram approach along with distinguishing feature selector (DFS) and Multinomial Naïve Bayes (MNB) classifier performed better on both the datasets. Lee et al. [7] proposed a new approach capable of handling thematic flows of texts with Deep Neural Networks (DNNs) so that the converse knowledge and disperse presentations of texts are integrated using Rhetorical Structure Theory (RST)-based discourse analysis. They tested their experiments for two document-level text classification, sentiment analysis and sarcasm detection tasks. Malliaros and Skianis [8] introduced a graph-based model where each text was determined by a graph that depicts the relationships among the words. Their system is capable of capturing the relationships among the words that coexist in a text thus developing a feature set. The significance of a word in a text was specified using graph theoretic vertex centrality measure. They tested the experiments in standard datasets using some classification algorithms. Rousseau et al. [9] represents the documents as graph-of-terms where they have extracted relevant features corresponding to larger n-grammes using iterated sub-graph mining. Based on the idea of k-core, they have reduced the graph to its main core for generating the final feature vector. Experiments were performed on four standard datasets which shows better performances compared with the state of the art methods. Li et al. [10] worked with a combined feature pruning technique for reducing the feature set depending on rudimentary investigation of the existing works. Then the neural architecture was trained and used to categorised 9804 Chinese documents from 20 domains. Luo et al. [11] used N-gramme feature where N is uni, bi and the combination of the two for extraction of features. They have further proposed a method depending on feature extraction and reduction approaches to enhance the performance of their system. Wu et al. [12] proposed a hybrid system by generating a lexical stock of a certain dimension and then estimates each term based on 1-of-m encoding. Then the convolutional neural network (CNN) was used to obtain the morphological features of character vectors from each word, and then through large scale text material training the semantic feature of each word vectors are be obtained the semantic feature of each word vectors. Finally, the text classification is carried out with the SVM multiple classifier. Al-Tahrawi [13] used Alj-News corpus includes 1500 Arabic news documents uniformly partitioned into 5 distinct classes from which 240 documents were used for training and 60 documents for testing. The author used the chi-square method in their experiment for the categorization task. Ali and Ijaz [14] provided comparisons among various statistical measures for categorising Urdu documents using NB and SVM. They have applied language-dependent preprocessing techniques for developing the final feature vector. From the experiment, it can be observed that on increasing the value of n up to 3 the performance increases, which decreases for the further increase from 3 to 4.

Lack of research materials in TC for Indian languages especially for Bangla shows that there is a pressing need for more research in this domain. Gupta and Gupta [15] proposed a hybrid classification algorithm by combining NB and Ontology-based classification techniques. They executed their approach on 184 Punjabi news articles collected from 7 sub-classes of sports category. Out of all, 50 text documents were used for training and remaining for testing. Their study showed that Hybrid classification proved to be better than other approaches. ArunaDevi and Saveeth [16] proposed a C-feature extraction model containing a pair of terms for Tamil text classification. The automatic Marathi documents text categorization technique based on the user’s profile was presented by Patil and Bogiri [17]. The system uses the LINGO approach using the vector space model. They tested their approach to 200 articles from 20 domains. Their experiment shows the efficiency of the LINGO clustering algorithm for the classification of Marathi text documents. Patil and Game [18] applied four classifiers among which NB proved to be most efficient concerning time and accuracy. Islam et al. [19] used t_f-id_f and SVM in their experiment and obtained 92.57% accuracy for the categorization of articles from 12 domains. Alam and Islam [20] used t_f-id_f and neural network for categorization of Bangla text documents and obtained a precision of 0.96. Hassan et al. [21] proposed a text classification system based on term-graph model for categorising texts from 12 domains. They obtained an average accuracy of 90.99% using KNN algorithm.

4. Proposed methodology

This article aims to show how the combination of text-based and graph-based features can work wonders compared with the individual application of the mentioned features in classifying Bangla text documents collected from various web news corpora. The motive behind proposing these two features is exploring its use in the text classification task for Bangla as there are very few techniques available in the Bangla text categorization task. The outline of the present methodology is shown in Figure 1 that shows different stages being adopted for the categorization of Bangla news articles whose detailed descriptions are presented in the following sections. Also, the comparison between several classification algorithms has been provided to analyse the system’s performance.

Figure 1.

The outline of the proposed methodology.

4.1. Data collection

Data are the most crucial part of any experiment whose collection needs care and efficiency so that errors cannot be noticed in further processing. Since from a literature study, it can be seen that no domain-based Bangla texts data are available publicly except one or two Bangla corpus, which are not adequate to be taken into account for experimenting. So, we had to build our dataset for the experiment. A total of 14,373 texts were obtained for the present work from various online Bangla news corpora, webpages and magazines from nine domains as stated by leading newspapers using web crawler tools (import.io). The associated images, links and other graphic contents from news corpora and magazines were discarded and only the text portions were considered and saved in UTF-8 format that were carried forward for further processing. The links used for extracting Bangla text documents are provided in Dhar et al. [22], while the distribution of text documents for each of the nine domains is illustrated in Table 1.

Table 1.

Distribution of text documents among the domains.

Domain	Number of documents
Business (B)	2168
Entertainment (E)	2166
Food and Recipe (F)	1051
Literature (L)	1103
Medical (M)	1047
State affairs (P)	2122
Sports (SP)	2134
Science & Technology (ST)	1194
Travel (T)	1388
Total	14,373

In the real-world scenario, it is often observed that there are disparate documents which are partially available, that is, a portion of it is missing. It is essential for text categorization systems to be able to handle such documents. To test the performance of our system for such a scenario, three datasets D₁, D₂, D₃ and D₄ were engendered from the original data. Each of the files was divided into two halves and D₁ was composed by considering the first half of each of the files. D₂ was composed by considering the last half of each of the files. D₁ represents such a scenario where texts are missing from the beginning. D₂ represents a scenario where texts are missing from the end. D₃ was composed by considering first half of the files of D₁ and the last half from D₂. D₄ was composed by considering the first half from D₂ and the last half from D₁. This was done to test our system for a scenario where there are some files whose information is missing from the start and again there are some files whose information is missing from the end. The partition of the dataset is depicted through a block diagram (Figure 2).

Figure 2.

Flowchart of the partitioned data.

4.2. Pre-processing

Digital texts comprised a series of characters, words, and phrases. Normally, before further processing, the sentences of the raw articles need to be broken into linguistic units and the task of breaking the sentences into such units is known as ‘tokenization’ and the unit as ‘token’. Tokenization can be represented as the sense of identifying the tokens that are required for further processing. Without the proper segregation of tokens, carrying out the analysis is impossible. The sentences were segregated using ‘space’ as the delimiter to obtain the results of the individual token to be 57,22,569 tokens in total. Since the token set consists of the indispensable and dispensable tokens, thus the removal of stopwords is needed for cleaning and filtering the data. The tokens that do not directly participate in classifying the text document to its category are termed as ‘stopwords’ and they are problem-specific. In this case, the list of stopwords used is provided in Stopwords [23]. After performing the pre-processing task, the number of tokens results in 44,47,689 items. However, the selection of stopwords requires special care as in some cases the stopwords may act as a prime source of information which can be a drawback for any experiment. So, this is a challenging task while carrying out any experiment with text data.

4.3. Hybrid Feature

The generation of an appropriate feature sets can be represented as wrapper and filter approach as stated in Karegowda et al. [24]. In the first approach, the feature selection depends on the classifier being considered in the experiment because to measure the importance of features set the method of classification is itself used. However, in Filter Approach, there is no dependency on training a model rather the feature selection depends on fast computation. Using the filter method, after getting the feature selection done, the feature set can be provided to any classification algorithm according to the need of the experiments for classification purposes. The different approaches of feature selection have a great influence on the performance of text classification tasks. Here, the text-based feature, graph-based feature and the combination of the two features have been used as a filter approaches where the feature set generated by these methods used by the classifier as input. The three feature selection methods used in the experiment are briefly described in the following.

4.3.1. Text-based feature

In the present work, a new text-based feature (token probability estimation) was proposed and used to assign weights to each of the tokens. This token probability estimation (TPE) estimates the term distribution in a document. In our pilot study, it has been seen that the standard weighting schemes $t_{f}$ and $t_{f} . i d_{f}$ does not perform well as $i d_{f}$ generally evaluates the number of differences a term can measure among relevant and non-relevant articles [25]. It has been noted from the researches [25] that the chance of occurrence of a token in the articles of class c than in those articles does not belong to c if the token is the target token of c. Based on this observation, the text-based feature has been proposed for evaluating the percipient potential of a token to a respective text category. The TPE is estimated based on the following formulas for each token being extracted after the pre-processing task for a particular news article (a)

TPE (a) = \frac{\sum_{t = 1}^{N} P (t | a) * P (t | c)}{N}

(1)

where $P (t | a)$ and $P (t | c)$ is the probability estimation of a token appearing in a number of articles (A) as well as number of text-categories (C) in the dataset being considered in this experiment and N is the total set of tokens. The $P (t | a)$ and $P (t | c)$ were calculated using the following equations

P (t | A) = \sum_{t = 1}^{A} \sum_{i = 1}^{n} f_{t} (a)

(2)

P (t | C) = \sum_{t = 1}^{C} f_{t} (a) * P (t | A)

(3)

where $f_{t}$ represents the frequency of a token t in an article a and n is the set of tokens for a particular article

f_{t} (a) = \log (f_{t_{a}})

(4)

4.3.2. Graph-based feature

The standard text representation–based features are unable to deal with the inversion of tokens and their equivalent subsets like ‘article about news’ versus ‘news article’. Thus, it is believed by several researchers that graph-based techniques can overcome this problem and hence gaining a lot of attention nowadays. Various techniques have been introduced in other fields [26] but require attention in text categorization tasks for different languages. Thus, this article aims to explore the usefulness of the graph-based feature for the Bangla text categorization task.

In this experiment, a graph-based feature was proposed that uses a weighted graph algorithm described as follows:

A graph Gp was can be defined by 3 tuples: Gp = (Vt, Eg, Wg), where Vt defines the set of vertices, Eg defines the edges connecting Vt and Wg defines the set of weights assigned to the Ed.

Node: Unique words being extracted from the articles in the dataset.

Edge: Developed using the frequency of tokens in the articles. If two words arise in a certain article then an edge was assigned between those words.

Weight Vector: It can be defined by the following equation (5), where Wg(s, t) denotes the weight provided to token s and token t, f(s, t) defines the presence of tokens s and t within an article, f(s) defines the presence of token s and f(t) denotes the presence of token t individually in that respective article. The higher the weight Wg(s, t) represents the stronger edge between two vertices

Wg (s, t) = \frac{f (s, t)}{{f (s) + f (t) - f (s, t)}}

(5)

4.4. Bayesian Classifier

In the present experiment, the Naïve Bayes Multinomial (NBM) classification algorithm is initiated to train the model based on the feature values being selected for classifying the text documents. The motivation behind choosing this classifier is its excellence and the promising performance in earlier work of text classification by Kim et al. [27], Kibriya et al. [28], Rehman et al. [29] and Wilbur and Kim [30]. Mainly NBM has been widely used for the English language but never been explored for Bangla text documents. It is novel to study how it works for classifying Bangla text documents into their respective text categories. Thus, in this study, NBM was used to explore its efficiency in developing an automatic Bangla text categorization system. The Naïve Bayes Multinomial is a specialised version of Naïve Bayes mostly designed by keeping the nature of text documents in mind. It generally computes the likelihood of an article to a respective domain using equation (6) where $P (i | x)$ is the conditional likelihood of word i and $i_{y}$ is the number of words in article y

P (x | y) = P (x) \underset{1 \leq k \leq i_{y}}{Π} P (i | x)

(6)

NBM counts the token and regulates the underlying assumptions into it. The extended form is suitable in a scenario where a token may occur for multiple numbers of times. This variation estimates the conditional likelihood $P (i | x)$ as the relative frequency of a particular occurrence of a token or word i for a given category. It works based on equation (7) where i is a word of domain x, N represents the database, B denotes the bag of tokens and $D_{c_{i}}$ is the presence of i in articles from domain x, which consists of the multiple presences in an article

P (i | x) = \frac{N_{x_{i}}}{{\sum_{i' \in B} N}_{x_{i'}}}

(7)

5. Result and analysis

For this experiment, WEKA [31] which is considered to be an extremely popular open-source classification tool, was used in applying the mentioned classifier. K folds cross-validation was applied to the articles where K was chosen to be 5. Cross-validation evaluates the predictive models where the original sample was distributed among the train and test set used for evaluation. Normally, in K folds cross-validation, the original dataset is segregated into K equal size groups out of which 1 group is provided for validation and the remaining groups are provided as a train set. This process is iterated over K times with each of the K groups provided for validation exactly once. The benefit of using this scheme is that every combination is provided as train and validation data with each group being provided for validation exactly once.

Using a text-based feature and NBM algorithm, we have obtained an accuracy of 96.62% for five-fold cross-validation. Different cross-validation folds (5, 10, 15 and 20) were analysed for further experiments and the result is provided in Table 2 where it can be seen that the maximum accuracy was obtained for five-fold cross-validation.

Table 2.

Accuracy for different cross-validation folds using text-based feature.

Folds	5	10	15	20
Accuracy (%)	96.62	96.24	95.88	95.88

Bold: Highest value.

The domain-wise accuracy obtained by text-based feature for five-fold cross-validation is depicted through Figure 3.

Figure 3.

Domain-wise accuracy obtained by text-based feature using NBM.

Next, we have experimented using a graph-based feature along with the NBM algorithm for five-fold cross-validation scheme and achieved an accuracy of 97.86%. Again different cross-validation folds (5, 10, 15 and 20) were analysed for further experiments for a graph-based feature and the result is provided in Table 3 from which it can be observed that the maximum accuracy was obtained for 10-fold cross-validation.

Table 3.

Accuracy for different cross-validation folds using graph-based feature and NBM.

Folds	5	10	15	20
Accuracy (%)	97.43	97.86	96.89	96.83

NBM: Naïve Bayes Multinomial.

Bold: Highest value.

The domain-wise accuracy obtained by the graph-based feature using 10-fold cross-validation is depicted through Figure 4.

Figure 4.

Domain-wise accuracy obtained by graph-based feature using NBM.

Finally, two features (text-based and graph-based features) were combined and tested with the NBM algorithm using five-fold cross-validation and obtained an accuracy of 98.73%. Different cross-validation folds (5, 10, 15 and 20) were analysed for further experiments for a combined feature and the result is provided in Table 4 from which it can be observed that the maximum accuracy was obtained for five-fold cross-validation. Also, the experiments were analysed for different Bayesian classifiers: Naïve Bayes Multinomial (NBM), Naïve Bayes Multivariate (NBMv) and Naïve Bayes (NB). The accuracy obtained for the same is provided in Table 4 which shows that the extended form is much more effective for the categorization of Bangla news articles compared with the other two Bayesian classification algorithm. The experiments were also conducted on standard English datasets: Reuters-21578 and 20 Newsgroups, using the proposed method based on the default parameter (five-fold cross-validation) and obtained an average accuracy of 90.27% and 88.52%.

Table 4.

Accuracy for different Bayesian classifiers.

Folds	5	10	15	20	25
NBM	98.73	98.16	98.16	98.09	97.89
NBMv	97.19	96.46	96.73	96.52	97.08
NB	78.94	78.45	77.89	78.16	77.85

NBM: Naïve Bayes Multinomial; NB: Naïve Bayes.

Bold: Highest value.

The experiments were also performed for n cross-validation folds with n less than 5 (n being 2, 3 and 4) to see how the system performs in these scenarios and also making the system more robust. The obtained result is provided in Table 5.

Accuracy = \frac{Correctly_classified_text_documents}{Total_number_of_text_documents} * 100 %

(8)

Table 5.

Accuracy for n cross-validation folds with n being 2, 3 and 4 using hybrid feature and NBM.

Folds	2	3	4
Accuracy (%)	97.71	97.93	97.75

NBM: Naïve Bayes Multinomial.

The recognition accuracy is obtained based on the equation provided in the following

The accuracy achieved for text-based feature, graph-based feature and the combination of these two features using NBM is provided in Figure 5 which shows that the combined features performed better than their individual application for classifying Bangla text documents.

Figure 5.

Obtained accuracies for three approaches.

The confusion matrix obtained for text domains or categories after using NBM with five-fold cross-validation based on the hybrid feature extraction approach on 14,373 Bangla text documents is illustrated in Table 6. From the confusion matrix presented in Table 6, it can be seen that few text documents have misclassified to other text categories other than the category they belong to. One of the reasons behind this is the nature of the contents of the text documents. In the case of a few domains, the contents of the text documents have much more similarity to the contents of the text documents of other domains which leads to misclassification. The error was analysed and a part of a text document is provided as an example in Figure 6. Bangla is one of the most morphologically rich languages. An instance has been considered for ‘Science & Technology (ST)’ domain and from the confusion matrix, it can be seen that out of 1194 text documents in ‘Science & Technology’ category, 1170 documents were classified correctly while 9 documents were classified to ‘Business (B)’ domain, 12 documents to ‘Medical (M)’ domain and 3 documents to ‘State affairs (P)’ domain. The text document (Figure 6) considered as a sample to represent misclassification originally belongs to the ‘Science & Technology’ domain but it can be observed that the number of tokens related to the ‘Medical’ domain is quite high and hence the document was misclassified to ‘Medical’ category. In Figure 6, it can be seen that after pre-processing, out of 79 terms 39 terms (in red) are medical related words (that is around 50% of total tokens after pre-processing) which cannot be neglected in the experiment. The rate and percentage of misclassification obtained using hybrid feature selection approach on the dataset for all the text categories is provided in Table 7 where it can be observed that Business domain and State affairs domain have high misclassification rate due to the fact that the contents of Business and State affairs domains are quite relatable to each other.

Table 6.

Confusion matrix obtained using NBM based on hybrid feature.

	B	E	F	L	M	P	SP	ST	T
B	2139	13	0	0	2	14	0	0	0
E	12	2144	0	0	2	8	0	0	0
F	0	0	1048	0	2	0	0	0	1
L	0	0	3	1090	0	0	0	3	7
M	7	0	4	0	1032	4	0	0	0
P	39	5	0	0	5	2068	2	3	0
SP	4	3	0	0	4	1	2122	0	0
ST	9	0	0	0	12	3	0	1170	0
T	2	3	3	0	2	0	1	0	1377

NBM: Naïve Bayes Multinomial; SP: sports; ST: science and technology.

Bold: Highest value.

Figure 6.

Example of misclassified text document.

Table 7.

Rate of misclassification of text documents obtained using hybrid feature and NBM.

Domain	Misclassification (in%) out of total samples of each domain
Domain	Text-based feature	Graph-based feature	Hybrid feature
Business	146(6.73%)	54(2.49%)	29(1.34%)
Entertainment	43(1.99%)	34(1.57%)	22(1.02%)
Food and Recipe	4(0.38%)	8(0.76%)	3(0.29%)
Literature	14(1.27%)	19(1.72%)	13(1.18%)
Medical	15(1.43%)	20(1.91%)	15(1.43%)
State affairs	175(8.25%)	59(2.78%)	54(2.54%)
Sports	37(1.73%)	45(2.11%)	12(0.56%)
Science & Technology	34(2.85%)	38(3.18%)	24(2.01%)
Travel	18(1.30%)	31(2.23%)	11(0.79%)
Average	54.00(2.88%)	34.22(2.08%)	20.34(1.24%)

NBM: Naïve Bayes Multinomial.

Table 8.

Result obtained for various metrics using hybrid feature and NBM.

Domain	Sensitivity	Specificity	Precision	NegativePredictive Value	FalsePositiveRate	FalseDiscoveryRate	FalseNegativeRate	Accuracy	F1 Score
B	0.966	0.997	0.986	0.993	0.002	0.013	0.033	0.992	0.976
E	0.988	0.998	0.989	0.998	0.001	0.010	0.011	0.996	0.989
F	0.990	0.999	0.997	0.999	0.000	0.002	0.009	0.999	0.993
L	1.000	0.999	0.988	1.000	0.000	0.011	0.000	0.999	0.994
M	0.972	0.998	0.985	0.997	0.001	0.014	0.027	0.996	0.979
P	0.985	0.995	0.974	0.997	0.004	0.025	0.014	0.994	0.980
SP	0.998	0.999	0.994	0.999	0.000	0.005	0.001	0.998	0.996
ST	0.994	0.998	0.979	0.999	0.001	0.020	0.005	0.997	0.987
T	0.994	0.999	0.992	0.999	0.000	0.007	0.005	0.998	0.993

NBM: Naïve Bayes Multinomial; SP: sports; ST: science and technology.

The experiment was also performed with a deep learning-based approach using LSTM-RNN which is suitable for sequential data. We obtained an accuracy of 95.43% using the hybrid feature. We also plan to use a deep learning-based approach in the feature extraction phase as well.

To test the robustness of our system, we experimented with different scenarios involving incomplete data. The four different scenarios are detailed in section whose result is presented in Table 9 below using NBM based on hybrid feature for five-fold cross-validation.

Table 9.

Results obtained for four different scenarios involving incomplete data.

Dataset	Accuracy (%)
$D_{1}$	95.28
$D_{2}$	93.98
$D_{3}$	95.22
$D_{4}$	92.10

We had further tested our system for a scenario where the system is trained by the original dataset consists of full texts but tested with incomplete texts and observed the performance for different incomplete test sets ( $D_{1}$ , $D_{2}$ , $D_{3}$ and $D_{4}$ ). The result obtained by these four datasets for the train-test split is also provided in Table 10 below using NBM based on the hybrid feature.

Table 10.

Result obtained by the four datasets for the train-test split.

Dataset	Accuracy (%)
$D_{1}$	94.94
$D_{2}$	94.12
$D_{3}$	94.62
$D_{4}$	90.98

5.1. Performances of popular classifiers

Experimentation was also tested with classifiers other than NBM, namely, Stochastic Gradient Descent (SGD), Support Vector Machine (SVM), Decision Tree (J48) and K Nearest Neighbour (KNN). The performance of classifiers for different evaluation metrics is shown in Table 11.

Table 11.

Performances of classifiers for different evaluation metrics.

Approach	Classifiers	Precision	Recall	F-measure
Text-based feature	NBM	0.966	0.966	0.966
	SGD	0.945	0.942	0.942
	SVM	0.921	0.920	0.920
	J48	0.893	0.893	0.893
	KNN	0.734	0.733	0.733
Graph-based feature	NBM	0.979	0.978	0.978
	SGD	0.951	0.950	0.950
	SVM	0.934	0.933	0.933
	J48	0.872	0.865	0.861
	KNN	0.743	0.735	0.727
Hybrid feature	NBM	0.987	0.986	0.986
	SGD	0.969	0.968	0.968
	SVM	0.942	0.940	0.940
	J48	0.888	0.888	0.888
	KNN	0.837	0.837	0.837

NBM: Naïve Bayes Multinomial; SGD: Stochastic Gradient Descent; SVM: Support Vector Machine; J48: Decision Tree; KNN: K Nearest Neighbour.

Bold: Highest value.

5.2. Statistical significance test

The comparisons of the classification algorithms were performed using the Friedman Rank Sum test [32]. The reason behind its selection depends on its character of not relying on the proposed methodology for the specific task. Here, the dataset was divided into n number of sets and for each set, the accuracy was extracted using all the commonly used classifiers (k) being chosen for this experiment. Then, based on the accuracy being achieved for all the subset for a particular classification algorithm, the mean ranking was calculated to get the classifier with top rank. Here, in the experiment, k and n were 6 and 5 which means the database was segregated into five sets. The accuracy obtained by the classification algorithms on each of the sets was measured and rank $R_{j}^{i}$ was assigned to them in descending order and then the mean rank was estimated by equation (9) provided in Table 12

R_{j} = \frac{1}{N} \sum_{i} R_{j}^{i}

(9)

Table 12.

Accuracies and ranks for the classifiers.

Classifiers		Parts of the dataset					Mean rank
		#1	#2	#3	#4	#5
NBM	A	98.71	98.88	98.89	98.44	98.77	1.0
	R	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
SGD	A	96.48	96.33	96.79	96.79	96.88	2.0
	R	(2.0)	(2.0)	(2.0)	(2.0)	(2.0)
NB	A	77.14	78.73	78.98	76.44	77.11	6.0
	R	(6.0)	(6.0)	(6.0)	(6.0)	(6.0)
SVM	A	94.82	94.88	94.11	93.99	95.52	3.0
	R	(3.0)	(3.0)	(3.0)	(3.0)	(3.0)
J48	A	90.18	90.22	90.89	91.10	91.23	4.0
	R	(4.0)	(4.0)	(4.0)	(4.0)	(4.0)
KNN	A	85.22	85.89	84.99	84.78	83.99	5.0
	R	(5.0)	(5.0)	(5.0)	(5.0)	(5.0)

NBM: Naïve Bayes Multinomial; SGD: Stochastic Gradient Descent; NB: Naïve Bayes; SVM: Support Vector Machine; J48: Decision Tree; KNN: K Nearest Neighbour.

The Friedman Statistic $χ_{F}^{2}$ [32] was computed with the help of equation (10). The list of critical values for $χ_{F}^{2}$ (distributed following k-1 degrees of freedom (df)) shows that the value of $χ_{F}^{2}$ for 5 df along with significance ( $α$ ) of 0.05 is 25.00. The value of $χ_{F}^{2}$ for the database is given in Table 13 where it can be seen that the value of $χ_{F}^{2}$ remarkably differs thus rejecting the null hypothesis. The statistics of this test are given in Table 13 where Asymp. Sig is the significance level

χ_{F}^{2} = \frac{12 N}{k (k + 1)} [\sum_{j} R_{j}^{2} - \frac{k {(k + 1)}^{2}}{4}]

(10)

Table 13.

Friedman test statistics.

N	5
Chi-Square test	25.00
Degree of freedom (df)	5
Asymptotic significance level	0.05

This test confirms that compared with all the classification algorithms, the performance of NBM is better based on its mean rank 1.00 (Table 12) which is the highest rank followed by the lowest rank (6.00) being achieved by NB.

After rejecting the null hypothesis, the Nemenyi test [32] was used for comparing best and worst-performing classifiers depending on their average ranks diverging by at least the critical difference (CD)

CD = q α \sqrt{\frac{k (k + 1)}{6 N}}

(11)

For Nemenyi’s test, the value of q0.05 for 6 classification algorithms is 2.850 (see Table 5(a) of [32]). So, CD is 3.372 calculated using Equation (11). The calculated CD values for all the classifiers are given in Table 14. The significantly different values of mean ranks between the best (NBM) and worst-performing classifier (KNN) and CD show the difference between the performing abilities of NBM with all the remaining five classifiers. Hence, the statistical significance tests conclude that the Naïve Bayes Multinomial classification algorithm outperforms all other classifiers.

Table 14.

CD for Nemenyi’s test.

NBM	0.0
SGD	1.0	0.0
NB	5.0	4.0	0.0
SVM	2.0	1.0	3.0	0.0
J48	3.0	2.0	2.0	1.0	0.0
KNN	4.0	3.0	1.0	2.0	1.0	0.0
	NBM	SGD	NB	SVM	J48	KNN

NBM: Naïve Bayes Multinomial; SGD: Stochastic Gradient Descent; NB: Naïve Bayes; SVM: Support Vector Machine; J48: Decision Tree; KNN: K Nearest Neighbour.

5.3. Performance of system on established datasets

5.3.1. Non-Indic language

To test the performance of the system and to prove the language independency of the proposed text categorization system, the experiments were also performed on two most popular and widely used datasets: Reuters-21578 R8 (7,674 documents) and 20 Newsgroups (18,828 documents). The performance of our system was also compared with the works proposed by Mahabal et al. [33], Tellez et al. [34], Jiang et al. [35] and Ko [36]. The accuracies obtained on Reuters-21578 (R8) and 20 Newsgroups (20NG) datasets are provided in Table 15. We observe that our system outperforms the existing systems on these two mentioned datasets.

Table 15.

Comparison of the proposed system with the existing systems on Reuters-21578 R8 and 20 Newsgroups dataset.

Reference	Accuracy (%)
Reference	R8	20NG
Mahabal et al. [33]	94.40	92.00
Tellez et al. [34]	92.14	83.48
Jiang et al. [35]	86.88	82.63
Ko [36]	95.30	88.43
Our method	96.82	94.19

NG: newsgroups.

5.3.2. Indic language

Since there was no availability of Bangla text categorization dataset, we had to develop our own dataset. However, in 2018, Alam and Islam [20] made their dataset publicly available which consists of 3,76,226 documents with 8,81,34,695 tokens from the state (St), sports (Sp), economy (Ec), entertainment (En) and international (In) domains. Thus, the performance of the proposed system was tested on this dataset as well and the performance is enhanced from precision value of 0.960 to 0.977. They worked with t_f-id_f and neural network approach for categorization of documents. The confusion matrix is provided in Table 16.

Table 16.

Confusion matrix obtained on the existing Bangla dataset.

	St	Ec	In	En	Sp
St	238,445	1887	1003	923	602
Ec	326	17,912	393	190	161
In	287	365	31,281	142	128
En	264	234	116	30,358	321
Sp	380	428	144	359	49,577
Precision ((20)) = 0.960,
Precision (our method) = 0.977.

Bold: Highest value.

5.4. Performance of reported systems on our dataset

Comparisons with other researches being performed for text classification task by considering the feature selection approach as well as the classification algorithm adopted by the researchers in their work and an attempt was made to classify the text documents holistically like the approach proposed here. We have replicated the frameworks proposed by other researchers on our obtained dataset and evaluated the results. In each case, for comparison, the classifier for which the author gained maximum accuracy was selected. t_f-id_f weighting scheme was used by Islam et al. [19] to categorise the documents using SVM and obtained accuracies of 92.57%. Alam and Islam [20] utilised t_f-id_f in their work for extracting features and obtained 96% accuracy based on neural network as classifier. Hassan et al. [21] proposed a text classification system based on term-graph model for categorising texts from 12 domains. They obtained an average accuracy of 90.99% using KNN algorithm. The obtained accuracy on our dataset for all the methods is recorded in Table 17 given in the following. From the table, it can be clearly stated that the proposed method outperforms all other previously implemented techniques in terms of recognition accuracy. In every scenario, it can be seen that standard t_f-id_f has been used whose disadvantage is already been discussed in Section. Also, in work of Hassan et al. [21], position of a token was captured while calculating the similarity but they have not considered the relationship between two tokens based on the weighted edges.

Table 17.

Performance of reported systems on our dataset.

Approach	Accuracy (%)
t_f-id_f + SVM [19]	95.24
t_f-id_f + Neural network [20]	95.76
Term-graph + KNN [21]	96.22
Proposed method	98.73

SVM: Support Vector Machine; KNN: K Nearest Neighbour.

6. Conclusion and future work

The experiments reported here show that among the three supervised feature selection methods ((a) text-based feature, (b) graph-based feature and (c) hybrid method involving two features); the hybrid feature performed better on 14,373 Bangla news articles from nine domains collected from various online news corpora. Also, from this comparative study, it is observed that among all the classification algorithms used in this experiment, the Naïve Bayes Multinomial outperforms the other commonly used classifiers. In the future, we also plan to investigate some standard dimensionality reduction schemes such as PCA and others on features extracted from texts. Also, we plan to make our dataset available publicly on request for research purposes only. There is a plan to test the system with a large number of text documents with an increasing number of text domains. It is possible to use more classifiers to test the system’s level of performance so that it can be applied for classifying text documents of other languages. We also plan to implement shallow and deep knowledge information and observe the performance of the system. We also plan to explore deep learning methods for developing an automatic Bangla text classification system.

Footnotes

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iDs

Niladri Sekhar Dash

Kaushik Roy

References

Mustafa

Rashid

. Kurdish stemmer pre-processing steps for improving information retrieval. J Inform Sci 2018; 44: 15–27.

Emadi

Rahgozar

. Twitter sentiment analysis using fuzzy integral classifier fusion. J Inform Sci 2020; 46: 226–242.

Huang

Zheng

Kang

, et al. Word sense disambiguation based on positional weighted context. J Inform Sci 2013; 39: 225–237.

. Enriching a thesaurus as a better question-answering tool and information retrieval aid. J Inform Sci 2018; 44: 512–525.

Ethnologue, https://www.ethnologue.com/language/ben

Parlak

Uysal

. On classification of abstracts obtained from medical journals. J Inform Sci 2019: 16.

Lee

Han

Myaeng

. A discourse-aware neural network-based text model for document-level text classification. J Inform Sci 2018; 44: 715–735.

Malliaros

Skianis

. Graph-based term weighting for text categorization. In: Proceedings of the IEEE/ACM international conference on advances in social networks analysis and mining, Paris, 25–28 August 2015, pp. 1473–1479. New York: IEEE.

Rousseau

Kiagias

Vazirgiannis

. Text categorization as a graph classification problem. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing, Beijing, China, 26–31 July 2015, pp. 1702–1712. Reston, VA: ACL.

10.

Zou

Han

. Chinese text classification based on neural network. In: Proceedings of the international symposium on neural networks, Dalian, China, 4–6 July 2013, pp. 514–522. New York: Springer.

11.

Luo

Ohyama

Wakabayashi

, et al. A study on automatic Chinese text classification. In: Proceedings of the international conference on document analysis and recognition, Beijing, China, 18–21 September 2011, pp. 920–924. New York: IEEE.

12.

Cheng

. Chinese text classification based on character-level CNN and SVM. Int J Intel Inform Database Syst 2019; 12: 212–228.

13.

Al-Tahrawi

. Arabic text categorization using logistic regression. Int J Intel Syst Appl 2015; 06: 71–78.

14.

Ali

Ijaz

. Urdu text classification. In: Proceedings of the 7th international conference on frontiers of information technology, Abbottabad, Pakistan, 16–18 December 2009. pp. 1–7. New York: ACM.

15.

Gupta

. Punjabi text classification using naive Bayes, centroid and hybrid approach. In: Proceedings of the 3rd workshop on South and South East Asian natural language processing, Mumbai, India, 8–15 December 2012, pp. 109–122. Reston, VA: ACL.

16.

ArunaDevi

Saveeth

. A novel approach on Tamil text classification using c-feature. Int J Sci Res Develop 2014; 02: 343–345.

17.

Patil

Bogiri

. Automatic text categorization: Marathi documents. In: International conference on energy systems and applications, Pune, India, 30 October–1 November, pp. 689–694. New York: IEEE.

18.

Patil

Game

. Comparison of Marathi text classifiers. ACEEE Int J Inform Tech 2014; 4(1): 11–22.

19.

Islam

Jubayer

FEM

Ahmed

. A support vector machine mixed with tf-idf algorithm to categorize Bengali document. In: 2017 international conference on electrical, computer and communication engineering (ECCE), Cox’s Bazar, Bangladesh, 16–18 February 2017, pp. 191–196. New York: IEEE.

20.

Alam

Islam

. Bard: Bangla article classification using a new comprehensive dataset. In: Proceedings of the international conference on Bangla speech and language processing, Sylhet, Bangladesh, 21–22 September 2018, pp. 1–5. New York: IEEE.

21.

Hassan

Uddin

Khan

. Bangla document categorization using term graph. Int J Comput Appl 2019; 178: 24–32.

22.

Dhar

Dash

Roy

. Classification of Bangla text documents based on inverse class frequency. In: Proceedings of the international conference on internet of things: smart innovation and usages, Bhimtal, India, 23–24 February 2018, pp. 1–6. New York: IEEE.

23.

Stopwords, https://www.tdil-dc.in/index.php?option=com_downloadtask=showresourceDetailstoolid=1635lang=en

24.

Karegowda

Manjunath

Jayaram

. Comparative study of attribute selection using gain ratio and correlation based feature selection. Int J Inform Tech Knowl Manag 2010; 2: 271–277.

25.

Abuaiadah

. Using bisect k-means clustering technique in the analysis of arabic documents. ACM Trans Asian Low-Resour Lang Inf Process 2016; 15(3): 1–13.

26.

Blanco

Lioma

. Graph-based term weighting for information retrieval. Inform Retrieval 2012; 15(1): 54–92.

27.

Kim

Han

Rim

, et al. Some effective techniques for naive Bayes text classification. IEEE Trans Knowl Data Eng 2006; 18(11): 1457–1466.

28.

Kibriya

Frank

Pfahringer

, et al. Multinomial Naive Bayes for text categorization revisited. In: Proceedings of Australasian joint conference on artificial intelligence, Cairns, QLD, Australia, 4–6 December 2004, pp. 488–499. New York: Springer.

29.

Rehman

Javed

Babri

, et al. Selection of the most relevant terms based on a max-min ratio metric for text classification. Expert Syst Appl 2018; 114: 78–96.

30.

Wilbur

Kim

. The ineffectiveness of within-document term frequency in text classification. Inform Retrieval 2009; 12: 509–525.

31.

Hall

Frank

Holmes

, et al. The Weka data mining software: an update. SIGKDD Explor 2009; 11: 10–18.

32.

Demšar

. Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 2006; 7: 1–30.

33.

Mahabal

Baldridge

Ayan

, et al. Text classification with few examples using controlled generalization. In: Proceedings of the Conference of the North American chapter of the association for computational linguistics: human language technologies, Minneapolis, MN, 2–7 June 2019, pp. 3158–3167. New York: ACL.

34.

Tellez

Moctezuma

Miranda-Jiménez

, et al. An automated text categorization framework based on hyperparameter optimization. Knowl Based Syst 2018; 149: 110–123.

35.

Jiang

Liang

Feng

, et al. Text classification based on deep belief network and softmax regression. Neural Comput Appl 2018; 29: 61–70.

36.

. A study of term weighting schemes using class information for text classification. In: Proceedings of the international ACM SIGIR conference on research and development in information retrieval, Portland, OR, 12–16 August 2012, pp. 1029–1030. New York: ACM.

NBM	0.0
SGD	1.0	0.0
NB	5.0	4.0	0.0
SVM	2.0	1.0	3.0	0.0
J48	3.0	2.0	2.0	1.0	0.0
KNN	4.0	3.0	1.0	2.0	1.0	0.0
	NBM	SGD	NB	SVM	J48	KNN

NBM	0.0
SGD	1.0	0.0
NB	5.0	4.0	0.0
SVM	2.0	1.0	3.0	0.0
J48	3.0	2.0	2.0	1.0	0.0
KNN	4.0	3.0	1.0	2.0	1.0	0.0
	NBM	SGD	NB	SVM	J48	KNN

NBM	0.0
SGD	1.0	0.0
NB	5.0	4.0	0.0
SVM	2.0	1.0	3.0	0.0
J48	3.0	2.0	2.0	1.0	0.0
KNN	4.0	3.0	1.0	2.0	1.0	0.0
	NBM	SGD	NB	SVM	J48	KNN