An ensemble scheme based on language function analysis and feature engineering for text genre classification

Abstract

Text genre classification is the process of identifying functional characteristics of text documents. The immense quantity of text documents available on the web can be properly filtered, organised and retrieved with the use of text genre classification, which may have potential use on several other tasks of natural language processing and information retrieval. Genre may refer to several aspects of text documents, such as function and purpose. The language function analysis (LFA) concentrates on single aspect of genres and it aims to classify text documents into three abstract classes, such as expressive, appellative and informative. Text genre classification is typically performed by supervised machine learning algorithms. The extraction of an efficient feature set to represent text documents is an essential task for building a robust classification scheme with high predictive performance. In addition, ensemble learning, which combines the outputs of individual classifiers to obtain a robust classification scheme, is a promising research field in machine learning research. In this regard, this article presents an extensive comparative analysis of different feature engineering schemes (such as features used in authorship attribution, linguistic features, character n-grams, part of speech n-grams and the frequency of the most discriminative words) and five different base learners (Naïve Bayes, support vector machines, logistic regression, k-nearest neighbour and Random Forest) in conjunction with ensemble learning methods (such as Boosting, Bagging and Random Subspace). Based on the empirical analysis, an ensemble classification scheme is presented, which integrates Random Subspace ensemble of Random Forest with four types of features (features used in authorship attribution, character n-grams, part of speech n-grams and the frequency of the most discriminative words). For LFA corpus, the highest average predictive performance obtained by the proposed scheme is 94.43%.

Keywords

Text genre language function analysis ensemble learning

1. Introduction

Text classification (also known as text categorisation) is the process of assigning a text document into one or more predefined classes or categories. With the advances in information and communication technologies, the immense quantity of text documents is electronically available on the web. Several forms of text collections, such as news articles, technical papers, books, digital libraries, email messages, blogs and web pages are important sources of information [1]. Typical application fields of text mining, such as news filtering and organisation, document organisation and retrieval, opinion mining and spam filtering, can be modelled as a text classification problem [2,3]. Based on the type of category, text classification can be divided into several problems, such as topic classification, sentiment classification and genre classification [4].

Text genre classification is a classification task in computational stylistics, which concerns with the identification of functional characteristics or kinds of text documents [5]. The genre is a broad term with several aspects and definitions. The genre can be defined as a collection of several aspects, focusing on different textual characteristics, such as brow, narrative and genre [6]. The genre may also be viewed as a bundle of documents that share the same stylistic properties [7]. With the identification of the genre of a text document, the functions of the document and its communicative context become explicit. In addition, the predictions about the form, function and context of communication can be greatly enhanced by the text genre classification [8]. Language function analysis (LFA) concerns with the single aspect of genres and it aims to classify text documents into three abstract and general classes, such as expressive, appellative and informative, as intended by Wachsmuth and Bujna [9]. Due to the abundant amount of information available on the web, text genre classification and LFA are essential tasks in information retrieval, text mining and computational linguistics. For instance, information retrieval results can be presented in a more effective way depending on the preferences of users with the help of text genre classification [10]. Text genre classification would be beneficial in filtering unintended documents (spam detection) and finding relevant documents faster. In addition, language usages embedded in grammatical structures and word senses can be related to text genres. In this regard, the identification of text genres could be beneficial for several tasks in computational linguistics, such as parsing, part-of-speech tagging and word sense disambiguation [6,10]. In text mining, one useful application of text genre classification is the automated annotation of text documents, where text documents are annotated by assigning genres [11]. Text genre identification can also be useful for retrieval tasks on the web, such as improving information seeking and reducing the information overhead [12].

Typically performed by supervised machine learning algorithms, text genre classification involves the construction of a classification model based on a training set of documents with appropriate tags and a supervised learning algorithm, such as support vector machines (SVMs), Naïve Bayes (NB) and a decision tree algorithm. In the machine learning–based text genre classification, each document is generally represented by the vector space model. Based on the feature sets, a supervised learning algorithm builds a classification model. Many different feature sets are utilised to extract feature sets from text documents. The feature representation schemes utilised in text genre classification include function words, character n-grams (CNG), part of speech (POS), POS trigrams, bag of words and syntactic chunks [8]. The high-dimensional feature space and irrelevancy of the features are two typical problems encountered in text mining. In addition, a robust classification scheme with high predictive performance cannot be achieved without an appropriate set of features [13,14].

Text genre classification is an application field where a high predictive performance is highly desired. Ensemble learning aims to obtain a classification model with a better predictive performance by combining the outputs of multiple learning algorithms. The recent literature on text and web mining indicate that the predictive performance of supervised machine learning algorithms can be enhanced with the utilisation of the paradigm of ensemble learning [15 –17].

Taking these issues into account, this article examines the predictive performance of different feature sets, supervised machine learning algorithms and ensemble methods in text genre classification. Based on the empirical analysis, an ensemble classification scheme based on LFA and feature engineering is presented. To the best of our knowledge, this is the first attempt which empirically examines the effectiveness of LFA-based text genre classification in conjunction with ensemble learning algorithms. For the empirical analysis, we built a manually annotated corpus with 3759 product reviews from two domains (books and cameras). The main idea beyond the representation of text documents based on LFA is adopted from the approach in Wachsmuth and Bujna [9], where text documents are tagged as personal, commercial or informational. In the experimental analysis, five popular supervised machine learning algorithms (NB algorithm, SVMs, logistic regression (LR), k-nearest neighbour (KNN) algorithm and Random Forest (RF) algorithm) are employed as the base learners. These base learning algorithms are combined by three ensemble learning methods, namely, Boosting, Bagging and Random Subspace. To obtain an appropriate feature set from text documents, text documents are represented by five different types of features, namely, features used in authorship attribution (AA), linguistic features (LF), CNG, part of speech n-grams (PNG) and the frequency of the most discriminative words (MDW). In the experimental analysis, each subset of features (LF and CNG, CNG and POS n-grams, the frequency of the MDW and CNG etc.) that can be obtained from the five different types of features are considered. In this way, 30 different configurations are obtained for each annotated data sets. In summary, the experimental analysis seeks to identify whether the ensemble learning can be used to improve the predictive performance of classification schemes in LFA-based text genre classification. In addition, it aims to identify an optimal feature set for text genre classification. Based on the empirical analysis, an ensemble classification scheme, which utilises four types of features (features used in AA, CNG, POS n-grams and the frequency of the MDW) to represent text documents, RF algorithm as the base learning algorithm and Random Subspace algorithm as the ensemble learning method, is presented.

The rest of this article is organised as follows. Section 2 briefly reviews the literature on text genre classification. Section 3 presents the LFA. Section 4 describes the classification algorithms and ensemble learning methods utilised in the empirical analysis. Section 5 describes the proposed classification scheme. In section 6, the experimental results and discussion is presented. Finally, section 7 presents the concluding remarks.

2. Literature review

This section briefly reviews the literature on text genre classification. The identification of an appropriate feature set is an essential task in machine learning–based text genre classification. In this regard, many different categories of genres have been proposed for genre classification. For instance, Karlgren and Cutting [12] presented a genre classification scheme based on simple set of features, such as word counts, POS frequencies and word and character-level statistics, where discriminant analysis is employed to identify genres. Similarly, Kessler et al. [6] identified four different types of features, namely, structural, lexical, character-level and derivative cues, to classify genres. In another study, Wolters and Kirsten [18] examined the performance of function word frequencies and POS tags for genre identification in conjunction with three supervised learning algorithms on a newspaper corpus. In text genre classification, Stamatatos et al. [19] utilised the frequencies of occurrence of the most frequent words in the English language as the feature set. In another study, Freund et al. [20] examined the performance of genre classification on software engineering domain. In this scheme, the bag of words was employed to represent text documents, and 16 genres were identified. In the identification of genres, SVMs were utilised as the learning algorithms. In another study, Roussinov et al. [21] examined the usefulness of genres on web page genre classification and presented five major genre groups. Dewdney et al. [22] examined the performance of bag of words representation, frequencies of verb tenses, frequencies of content words, punctuation frequencies on text genre classification in web search and spam filtering. In this scheme, seven genres were identified. The genres were classified using NB, SVMs and C4.5 decision tree algorithm. Similarly, Stamatatos et al. [5] presented a stylistic approach based on the analysis-level measures for text genre classification. The experimental analysis indicated that the presented scheme outperforms representation based on the distributional lexical measures, such as functions of vocabulary richness and frequencies of the occurrence of the most frequent words. HaCohen-Kerner et al. [23] also utilised stylistic feature sets and name-based feature sets for text classification. Lee and Myaeng [10] presented a genre classification methodology based on document frequency ratios across genres and across subject classes. The empirical analysis indicated that the utilisation of subject classes with a similarity-based approach enhances the predictive performance.

To obtain a robust representation scheme based on linguistic analysis and term frequency methods, Ferizis and Bailey [24] presented a method based on the approximation of crucial POS features. The experimental analysis indicated that the approximation of crucial features can lead to the reduction in computational time while retaining the predictive performance. In another study, Xu et al. [25] examined the performance of the likelihood ratio test-based feature selection in conjunction with SVMs to classify genres of Chinese text documents.

The LFA-based text genre classification has been examined by Wachsmuth and Bujna [9]. In the empirical analysis, six different feature sets, namely, simple genre features, text type, writing style, sentiment, core trigrams and core vocabularies, were taken into consideration. In this scheme, SVMs were utilised as the supervised learning algorithms. In a similar manner, Kröll and Strohmaier [26] employed intent analysis to classify text documents. Petrenz and Webber [27] examined the performance of five existing schemes on text genre classification. In another study, Vilares et al. [28] employed LF, such as POS, syntactic and semantic information, for topic identification of Spanish twitter messages.

More recently, Hajibayova and Jacob [29] explored the influence of genres on the level of abstraction of user-generated tags. In another study, Rosso [30] examined the use of genre as a document identifier to enhance the effectiveness of web searches. Similarly, Na and Thet [31] examined the effectiveness of web search results for text genre and sentiment classification.

Tseng et al. [32] presented an automatic method for patent documents based on text summarisation. Similarly, Codina-Filba et al. [33] employed genre-specific feature sets for patent document summarisation.

Ensemble learning is a promising research direction of machine learning research. The classification algorithms and ensemble learning methods have been frequently applied for many applications, ranging from medical data analytics to predicting energy demand [34,35]. For instance, Rodger [34] examined the predictive performance of several classification schemes, such as artificial neural networks and KNN algorithms and an ensemble classification scheme based on a fuzzy-rough nearest neighbour algorithm is presented. Similarly, Rodger [35] presented an ensemble scheme for medical data analytics based on KNN algorithm.

In the classification phase of text genre identification, ensemble learning methods can be employed to improve the predictive performance. For instance, Chen et al. [36] examined the performance of several feature sets, such as features related to genre-related information, text-based features and image-based features for genre identification on document pages and they classify the genres of documents by an ensemble learner. Similarly, Finn and Kushmerick [37] examined the performance of different feature sets and machine learning methods for automatic genre classification. In this scheme, bag of words, POS statistics and text statistics were considered as the feature sets. In the classification phase, an ensemble learner was presented, which combines the predictions of classifiers constructed by different feature sets. In another study, Gencosman et al. [38] presented an ensemble classification scheme which integrates character n-gram-based representation with neural network classifiers for topic identification.

As mentioned in advance, the identification of an appropriate feature set is a critical issue in automated text genre classification. In addition, the combination of supervised learning methods by ensemble learning can yield encouraging results. However, the predictive performance of ensemble learning has been underexplored in the literature. In this regard, we make the following contributions in this article:

To examine the predictive performance of machine learning methods on text genre classification;

To explore the performance of different feature sets for constructing text genre classifiers with high predictive accuracy;

To examine whether the ensemble learning can enhance the predictive performance on text genre classification;

To investigate the performance of combined feature (CF) sets on text genre classification and to present an efficient feature set for the problem;

To present an efficient ensemble classification scheme based on an effective feature subset and classifier ensemble.

3. LFA

Text types can be assigned into three distinct categories as expressive, appellative and informative texts, as intended by Reiss [39]. According to Reiss [39], informative texts aim to transfer news, knowledge, opinions and so on in order to inform. The expressive text documents aim ‘mainly to transmit contents organised in an artistic fashion, consciously organising the content according to aesthetic criteria’ [40,41]. In contrast, appellative texts aim to transmit contents of persuasive character to induce the text’s receiver [40]. Based on the language function model of Reiss [39], LFA aims to assign text documents into one of the abstract language functions based on the functions and intentions of the text documents. In LFA, personal and expressive text documents that personal attitudes towards an entity are assigned to the class of expressive. In contrast, text documents, which aim to induce behavioural responses, are classified as appellative and text documents, which aim to transmit information, are classified as informational. For text documents containing reviews, expressive texts are personal, appellative texts are commercial and informative texts are informational [9].

4. Classification algorithms and ensemble learning methods

This section briefly reviews the classification algorithms and ensemble learning methods utilised in the empirical analysis.

4.1. Classification algorithms

Machine learning algorithms have been widely employed in text and web mining. Machine learning classifiers may be broadly classified as rule-based classifiers (such as Ripper, Part and genetic algorithms), decision tree algorithms (such as C4.5 algorithm, ID3 algorithm and RF algorithm), perceptron-based methods (such as artificial neural networks and radial basis functions), statistical learning–based methods (such as NB classifier), instance-based methods (such as KNN algorithm) and SVMs [42]. NB, SVMs and decision trees are widely employed for text classification problems with encouraging results [43].

4.1.1. NB algorithm

NB algorithm is a statistical classification algorithm which is based on Bayes’ theorem [44]. NB algorithm is based on the class conditional independence assumption, which simplifies the required computational cost. The algorithm can yield comparable results with decision trees and neural network algorithms [1]. In addition, it can scale well and be easily utilised in a number of domains.

4.1.2. SVMs

SVMs are classification algorithms for classification of linear and non-linear data, which employ a non-linear mapping for transforming the original data into a higher dimension [45]. SVMs aim to identify an optimal decision boundary for partitioning the data into classes. SVMs have relatively high predictive performance and they are less vulnerable to the overfitting problem. Due to the characteristics of text mining, such as high-dimensional feature space, sparsity and few irrelevant features, SVMs are suitable classifiers for text mining domain [1].

4.1.3. LR

LR is a statistical classification algorithm, which predicts the probability of some event’s occurring as a linear function of a set of predictor variables. In linear regression, the membership values are not proper probability values, and the least-squares regression takes errors as both statistically independent and normally distributed with the same standard deviation. To overcome the aforementioned problems of the linear regression, LR constructs a linear model based on a transformed target variable [46].

4.1.4. KNN algorithm

KNN algorithm is an instance-based learning algorithm, which classifies instances based on their similarity to the k-closest instances of the training set [47]. In KNN, the output is a class membership, and an unseen instance is assigned to a particular class based on the majority vote of its neighbours. KNN algorithm has a relatively simple structure with a limited number of parameters. The algorithm can yield encouraging results on large training sets, but the determination of the KNNs may be costly.

4.1.5. RF algorithm

RF algorithm is a decision tree algorithm, which is an ensemble of classification and regression trees, induced from bootstrap samples of the training data [48]. In this model, the generalisation error of the classifier depends on the power of the individual trees and the association between trees. The random feature selection method is employed in the tree induction phase, which enables to obtain predictive performance comparable to AdaBoost algorithm.

4.2. Ensemble learning methods

Ensemble learning methods aim to combine the predictions of multiple learning algorithms, so that a classification model with a higher predictive performance can be obtained. Compared with the base learning algorithms, the generalisation ability of classifier ensembles is greatly enhanced. In addition, ensemble learning methods can boost the predictive performance of weak learning algorithms, which are slightly better than random guessing to strong learning algorithms, which make very promising predictions [49]. The main reasons for utilising classifier ensembles can be explained by statistical, representational and computational factors. Statistically, ensemble learning methods can yield promising results in the absence of sufficient data or in the existence of many different hypotheses with limited data. In terms of representation, there may be unknown functions in the hypotheses space. In terms of computation, the use of a classifier ensemble instead of a single learning algorithm can reduce the chance of getting stuck in local optima [50]. In order to design efficient classification schemes, the two critical issues are to keep classification algorithms as accurate as possible and providing a high diversity among the classifiers. Ensemble learning methods can be broadly divided into two categories, as dependent method and independent method, based on the approach followed in ensemble construction [51]. In dependent method, the outputs of subsequent classifiers are determined based on the outputs of former classifiers. In contrast, the outputs of classifiers are obtained separately, and their outputs are combined to obtain the final prediction of the ensemble in independent method. Boosting algorithms are typical examples of dependent methods, and Bagging, Dagging and Random Subspace algorithms are typical examples of independent methods. Boosting, Bagging and Random Subspace are well-known ensemble learning methods. The primary characteristics of the ensemble learning methods can be summarised as follows:

Boosting algorithms are able to construct strong learners from weak learning algorithms by adjusting iteratively the weight of instances in the training set and the weights of classification algorithms based on the distribution. AdaBoost algorithm is one of the most frequently employed Boosting algorithm, which enhances the predictive performance of the learning algorithms by weighting instances at each iteration according to the sample distribution [46].

Bagging (Bootstrap aggregating) algorithm achieves diversity among the base learners by bootstrap sampling from the original data set to obtain training set. In the Bagging algorithm, the learning algorithms trained on newly sampled data sets are combined by a majority voting scheme. Bagging algorithm can yield promising results for data sets with limited size. It can also be used to enhance the predictive performance of unstable classification algorithms.

Random Subspace algorithm employs feature set manipulation-based diversity. Instead of modifying the training sets, as in the case of Bagging, Random Subspace algorithm trains individual classification algorithms using different feature subsets. Since different feature subsets are used to train the classifiers, a high diversity can be expected [49]. Text classification is a domain with high-dimensional feature space. By randomly partitioning the feature space, classifiers with high diversity can be achieved, which is a desired goal in ensemble design. In this way, Random Subspace algorithm generally yield promising results [3,52]. The rest of this section briefly explains these methods.

4.2.1. AdaBoost algorithm

Boosting algorithm is an ensemble learning method to enhance the predictive performance of weak learning algorithms. In Boosting algorithm, weak learning algorithms are recursively trained on the different sampling distributions to obtain strong learning schemes. AdaBoost algorithms are a widely utilised Boosting algorithm. It aims to obtain a strong learning scheme by focusing on data points that are difficult to classify [53]. In each iteration of the algorithm, the weight values for misclassified instances are increased and the weight values for correctly classified instances are decreased. In this way, the weak learning algorithm involves more iterations and classifiers to the instances that are difficult to classify [54].

4.2.2. Bagging algorithm

Bagging algorithm (also known as Bootstrap aggregating) is an ensemble learning method which combines weak learning algorithms trained on different training sets obtained from the original training set by sampling with replacement [55]. One of the critical issues in classifier combination is to provide diversity in classifier ensembles. Ensemble classifiers tend to provide better results when the weak learners have a significant diversity [56]. In Bagging, the diversity is achieved by the sampling scheme. As a sampling scheme, simple random sampling with replacement is generally utilised. To combine the outputs of weak learners, majority voting or weighted voting schemes are used.

4.2.3. Random Subspace algorithm

Random Subspace algorithm is an ensemble learning method which combines the learning algorithms trained on randomly selected modified feature spaces [57]. Random Subspace algorithm achieves diversity by training classification algorithms on the different samples of the training set, like bagging algorithm. However, Random Subspace method obtains different samples by modifying the feature space. The method aims to avoid overfitting problem and it tends to obtain encouraging results for the data sets with redundant/irrelevant features.

5. The ensemble text genre classification scheme

The ensemble text genre classification scheme relies on feature engineering (features used in AA, LF, CNG, PNG and the frequency of the MDW) to extract features for a supervised machine learning algorithm. We carry out an extensive empirical analysis on different feature sets to determine which kinds of feature sets have the greatest impact on the LFA-based text genre classification. In addition, the combination of different feature sets is taken into consideration to obtain the best feature set with a higher predictive performance. For this purpose, we first construct a LFA corpus with manually annotated English texts from two different domains: Book and Camera reviews. Then, text documents are annotated by language functions, as personal, commercial and informational. In the feature engineering stage, five different feature sets and their possible combinations are evaluated. In the classification phase, five classification algorithms (NB algorithm, SVMs, LR, KNN algorithm and RF algorithm) are employed as the base learning algorithms. To improve the predictive performance, ensemble learning methods (such as AdaBoost, Bagging and Random Subspace) and their combination with the base learning algorithms are also considered. Based on the empirical analysis, the presented ensemble text genre classification scheme utilises the combination of four types of features (features used in AA, CNG, PNG and the frequency of the MDW) to represent text documents. In the classification phase, Random Subspace ensemble of RF is utilised as the ensemble classifier. Figure 1 illustrates the general structure of our text genre classification system.

Figure 1.

The general structure of the text genre classification system.

The rest of this section briefly presents the details for LFA corpus and feature engineering.

5.1. LFA corpus

LFA corpus consists of English review text documents with 2121 commercial, informational and personal texts from book reviews and 1638 camera reviews. All of the text documents were manually annotated with respect to their LFA into three distinct classes. The genre corresponds to the language function of the text, which can take the value of commercial, informational and personal. The distribution of the data sets (Book and Camera) based on the language function annotations is presented in Table 1. The data sets utilised in the empirical analysis can be downloaded from.¹ The raw documents of Book and Camera reviews are annotated by two experts. For Book and Camera reviews, approximately 20% of the documents are annotated twice. To examine the inter-annotator agreement, Cohen’s Kappa ( κ ) metric is computed for each data sets. Cohen’s kappa (κ) metric is computed based on the documents that are annotated twice to make an estimate about the inter-annotator agreement. For Book data set, this has resulted κ = 0.84. Similarly, for Camera data set, this has resulted κ = 0.79. Since κ = 1 indicates a perfect agreement between different annotators, the agreement levels of the different experts on annotations are good. In addition, 0.61–0.80 range for Cohen’s kappa ( κ ) corresponds to substantial agreement among the annotators and 0.81–1.00 range for Cohen’s kappa ( κ ) corresponds to perfect agreement [58]. Hence, the annotators provide suitable labels for text genre annotation.

Table 1.

The distribution of data sets based on the language function annotations

Language function	Book data set	Camera data set
Commercial	638 (30.08%)	768 (46.89%)
Informational	821 (38.71%)	543 (33.15%)
Personal	662 (31.21%)	327 (19.96%)

5.2. Feature engineering

As mentioned in advance, the identification of an appropriate feature set is an essential task in text genre classification. In this regard, five different feature sets are considered for LFA corpus. These are features used in AA, LF, CNG, PNG and the frequency of the MDW. In Table 2, the number of features for each feature set group is presented.

Table 2.

The number of features for LFA corpus

Feature set	Book data set	Camera data set
Features used in authorship attribution (AA)	22	22
Linguistic features (LF)	45	45
Character n-grams (CNG)	56	97
Part of speech n-grams (PNG)	97	120
The frequency of the most discriminative words (MDW)	78	89

LFA: language function analysis.

5.2.1. Features used in AA

Features used in AA set consist of features about the writing style. This set contains typical features used in AA, such as the frequency of the most common words, the frequency of POS trigrams, the frequency of character trigrams, the frequency of capitalised uppercase and lowercase words and the frequency of parentheses, punctuation and quotation marks [9,59].

5.2.2. LF

The LF set consist of features about the linguistic expert knowledge. This set contains LF, such as the frequency of time and money entities, the frequency of POS tags, personal pronouns, possessive pronouns, adjectives and nouns [9,60].

5.2.3. CNG

The CNG set consists of the frequency of the most discriminative character trigrams of classes of each language function. In our work, character trigrams that occur over six times in one language function class are utilised.

5.2.4. PNG

The PNG set consists of the frequency of the most discriminative POS trigrams of classes of each language function. In our work, POS trigrams that occur over six times in one language function class are utilised.

5.2.5. The frequency of the MDW

This feature set consists of the frequency of the MDW. In our work, a word that occurs at least 15 times in the training texts of a particular language function class is regarded as discriminative.

6. Experimental results and discussion

6.1. Experimental procedure

In the experimental analysis, 10-fold cross validation method is employed. In this scheme, the original data set is divided into 10 sets of mutually exclusive folds. Then, nine of the sets are utilised for training, and one of the sets is utilised for testing. The process is repeated 10 times, and the average results for 10-fold are reported. The experiments are performed with the machine learning toolkit WEKA (Waikato Environment for Knowledge Analysis) version 3.7.11, which is an open-source platform with many machine learning algorithms implemented in Java [61]. The default parameters for classification algorithms and the ensemble methods are utilised. In the empirical analysis, supervised learning algorithms from different classification approaches are utilised. In the empirical analysis, NB algorithm, SVMs, LR, KNN algorithm and RF algorithm are employed as the baseline classifiers. In ensemble classification, three well-known ensemble methods (AdaBoost, Bagging and Random Subspace) are utilised. The experimental evaluations are carried on a PC with an Intel Core i7 CPU 3.40 GHz with 8.00 GB RAM.

As mentioned in advance, the empirical analysis aims to identify the performance of different feature sets for text genre classification. It aims to examine the performance of CF sets and to present an efficient feature set for the domain. In this regard, five different types of features (features used in AA, LF, CNG, PNG and the frequency of the MDW) are identified. To examine the performance of individual and CF sets, 30 different feature set configurations are obtained and evaluated.

6.2. Evaluation measures

In order to evaluate the predictive performance of different feature sets, classification algorithms and ensemble methods, classification accuracy and F-measure are utilised as the evaluation metrics. Classification accuracy (ACC) is one of the most widely employed metrics in examining the classifiers. It is the proportion of true positives and true negatives over the total number of instances as given by equation (1)

A C C = \frac{T N + T P}{T P + F P + F N + T N}

(1)

where TN, TP, FP and FN represent number of true negatives, number of true positives, number of false positives and number of false negatives, respectively. Precision (PRE) is the proportion of the true positives against the true positives and false positives as given by equation (2)

P R E = \frac{T P}{T P + F P}

(2)

Recall (REC) is the proportion of the true positives against the true positives and false negatives as given by equation (3)

R E C = \frac{T P}{T P + F N}

(3)

F-measure takes values between 0 and 1. It is the harmonic mean of precision and recall as determined by equation (4)

F - measure = \frac{2 * P R E * R E C ‘}{P R E + R E C}

(4)

6.3. Results and discussion

This section presents the classification accuracy and F-measure values of the classification algorithms and the ensemble methods by different feature sets on 30 different configurations of LFA corpus. In the tables, the best (the highest) results obtained by a particular feature set are indicated using only italics, and the highest predictive performance among all the compared results is indicated using both italics and underline.

The first concern of the study is to examine the predictive performance of different feature sets for constructing text genre classifiers. In Table 3, classification accuracies obtained by different feature sets on the supervised learning algorithms are presented. As it can be observed from Table 3, the highest (the best) classification accuracies are obtained by a feature set, which combines features used in AA, CNG, PNG and the frequency of the MDW. The second highest predictive performance is generally achieved by the representation of text documents with full feature set. For Book data set, the second highest predictive performance on KNN algorithm is achieved by a feature set, which combines features used in AA with CNG. For Book data set, the second highest predictive performance on RF algorithm is achieved by a feature set, which combines features used in AA, LF and the frequency of the MDW. For Camera data set, the second highest predictive performance on KNN algorithm is obtained by a feature set, which combines features used in AA with the frequency of the MDW. The same feature set also achieves the second highest predictive performance on RF algorithm. As it can be observed from the classification accuracies presented in Table 3, the representation of text documents with individual feature sets (such as features used in AA, LF and CNG) generally yields lower predictive performance compared with the representations obtained by CF sets.

Table 3.

Classification accuracies obtained by different feature sets on the supervised learning methods

Feature set	Book data set					Camera data set
	NB	SVM	LR	KNN	RF	NB	SVM	LR	KNN	RF
AA	58.54	66.79	66.44	77.47	81.35	56.66	64.95	62.78	80.00	84.70
LF	68.48	73.88	73.67	72.57	84.84	56.07	61.47	61.60	78.03	83.34
CNG	70.24	76.98	76.14	76.29	82.40	65.27	73.13	71.62	77.70	83.73
PNG	65.35	74.26	73.05	74.36	80.62	60.11	62.74	62.73	75.43	79.48
MDW	72.85	75.07	74.24	69.93	87.24	69.34	74.44	73.95	78.17	84.99
AA+LF	72.16	78.12	77.77	76.04	85.52	62.98	69.44	68.13	80.11	84.66
AA+CNG	73.89	80.96	79.98	78.19	84.05	71.01	78.33	76.32	79.12	85.27
AA+PNG	70.83	80.42	78.22	75.97	82.20	69.04	73.04	71.57	75.61	84.56
AA+MDW	77.90	81.91	81.93	71.51	87.81	73.09	77.99	76.90	81.79	86.80
LF+CNG	80.50	83.64	82.64	72.37	85.03	69.70	79.41	78.33	77.09	84.99
LF+PNG	72.60	78.97	76.94	70.53	83.31	62.83	71.53	69.69	75.66	82.85
LF+MDW	79.44	84.29	83.71	68.41	87.10	69.47	78.22	77.76	78.31	85.69
CNG+PNG	78.24	83.50	82.46	73.36	82.51	69.69	78.15	76.56	73.33	82.82
CNG+MDW	78.13	81.84	81.63	69.13	87.74	73.52	82.35	81.11	79.34	85.02
PNG+MDW	80.49	85.00	83.60	67.85	85.86	72.11	78.85	78.15	78.01	83.70
AA+LF+CNG	80.99	84.69	84.58	74.93	85.26	73.56	81.40	80.11	78.91	86.39
AA+LF+PNG	75.04	82.33	80.69	73.25	84.37	69.40	77.00	74.56	78.10	85.44
AA+LF+MDW	81.11	86.01	85.39	70.09	88.42	71.21	80.34	80.13	80.50	86.42
AA+CNG+PNG	81.57	85.48	84.55	74.28	83.50	74.31	81.42	80.47	76.00	85.36
AA+CNG+MDW	80.60	84.82	84.53	71.41	87.71	76.23	83.25	81.99	80.95	86.13
AA+PNG+MDW	82.71	87.42	85.95	69.70	87.04	74.98	80.79	80.14	79.61	86.21
LF+CNG+PNG	79.44	84.75	83.26	70.41	84.79	72.31	81.61	80.48	73.79	84.36
LF+CNG+MDW	83.18	86.09	85.68	68.00	86.85	75.09	83.44	83.35	77.58	85.54
LF+PNG+MDW	81.14	86.54	85.26	67.51	86.70	72.22	80.50	80.54	78.39	83.93
CNG+PNG+MDW	83.01	86.61	85.61	67.46	85.40	75.05	83.62	82.15	77.58	84.47
AA+LF+CNG+PNG	81.63	85.43	85.34	72.32	84.79	75.82	82.57	81.90	76.61	85.45
AA+LF+CNG+MDW	82.86	86.95	86.77	69.79	87.54	76.45	83.61	83.73	80.13	85.67
AA+LF+PNG+MDW	82.56	87.56	86.61	69.33	86.94	74.44	81.81	81.06	79.59	85.96
LF+CNG+PNG+MDW	83.43	87.04	86.56	66.78	86.83	76.29	83.87	84.13	76.93	85.27
AA+CNG+PNG+MDW	85.56	90.72	89.46	81.25	$\underline{91.33}$	80.18	89.09	87.28	86.87	90.24
Full set	84.07	87.84	87.63	68.73	86.80	77.97	83.96	84.18	79.37	85.76

AA: features used in authorship attribution; LF: linguistic features; CNG: character n-grams; PNG: part of speech n-grams; MDW: the frequency of the most discriminative words; full set: AA+LF+CNG+PNG+MDW; NB: Naïve Bayes; SVM: support vector machines; LR: logistic regression; KNN: k-nearest neighbour algorithm; RF: Random Forest algorithm.

Regarding the predictive performance of supervised learning algorithms on text genre classification, RF algorithm generally yields the highest classification accuracies and SVMs generally yield the second highest classification accuracies. Among all configurations compared in Table 3, the highest (best) predictive performance (91.33%) is obtained by a feature set, which combines features used in AA, CNG, PNG and the frequency of the MDW with RF algorithm.

In Table 4, F-measure values obtained by different feature sets on the supervised learning methods are presented. Similar to the classification accuracies presented in Table 3, the highest (the best) classification accuracies are obtained by a feature set, which combines features used in AA, CNG, PNG and the frequency of the MDW. The second highest predictive performance is generally achieved by the full set–based representation, which combines features used in AA, LF, CNG, PNG and the frequency of the MDW. Among all configurations compared in Table 4, the highest (best) F-measure value (0.98) is obtained by a feature set, which combines features used in AA, CNG, PNG and the frequency of the MDW with RF algorithm.

Table 4.

F-measure values obtained by different feature sets on the supervised learning methods

Feature set	Book data set					Camera data set
	NB	SVM	LR	KNN	RF	NB	SVM	LR	KNN	RF
AA	0.63	0.72	0.71	0.83	0.94	0.54	0.61	0.58	0.92	0.94
LF	0.74	0.82	0.81	0.86	0.95	0.42	0.57	0.57	0.90	0.94
CNG	0.75	0.83	0.82	0.87	0.95	0.65	0.76	0.73	0.86	0.94
PNG	0.70	0.82	0.80	0.87	0.95	0.61	0.68	0.66	0.89	0.94
MDW	0.76	0.77	0.76	0.80	0.95	0.68	0.77	0.76	0.92	0.95
AA+LF	0.78	0.86	0.86	0.88	0.96	0.57	0.70	0.68	0.91	0.94
AA+CNG	0.79	0.88	0.87	0.91	0.96	0.72	0.83	0.79	0.87	0.93
AA+PNG	0.76	0.90	0.87	0.88	0.96	0.72	0.79	0.74	0.89	0.95
AA+MDW	0.82	0.86	0.86	0.81	0.97	0.71	0.82	0.80	0.93	0.95
LF+CNG	0.88	0.92	0.91	0.85	0.96	0.68	0.84	0.81	0.86	0.95
LF+PNG	0.79	0.88	0.85	0.81	0.95	0.57	0.78	0.72	0.88	0.94
LF+MDW	0.85	0.92	0.90	0.77	0.96	0.66	0.83	0.81	0.92	0.95
CNG+PNG	0.84	0.92	0.90	0.83	0.96	0.70	0.84	0.79	0.82	0.94
CNG+MDW	0.83	0.88	0.87	0.79	0.96	0.74	0.88	0.85	0.90	0.95
PNG+MDW	0.86	0.91	0.89	0.77	0.95	0.72	0.85	0.81	0.92	0.94
AA+LF+CNG	0.89	0.95	0.94	0.86	0.97	0.73	0.87	0.83	0.87	0.95
AA+LF+PNG	0.81	0.92	0.89	0.83	0.96	0.68	0.85	0.78	0.88	0.95
AA+LF+MDW	0.87	0.94	0.92	0.78	0.97	0.67	0.85	0.84	0.91	0.95
AA+CNG+PNG	0.88	0.95	0.93	0.86	0.97	0.75	0.88	0.84	0.83	0.95
AA+CNG+MDW	0.86	0.93	0.91	0.81	0.97	0.76	0.90	0.87	0.91	0.94
AA+PNG+MDW	0.90	0.95	0.93	0.78	0.97	0.75	0.88	0.85	0.93	0.95
LF+CNG+PNG	0.86	0.94	0.92	0.80	0.96	0.72	0.88	0.85	0.83	0.94
LF+CNG+MDW	0.90	0.95	0.93	0.77	0.97	0.75	0.90	0.88	0.89	0.95
LF+PNG+MDW	0.88	0.94	0.92	0.76	0.96	0.70	0.87	0.85	0.93	0.94
CNG+PNG+MDW	0.90	0.94	0.92	0.77	0.96	0.75	0.91	0.87	0.89	0.95
AA+LF+CNG+PNG	0.89	0.96	0.95	0.82	0.97	0.77	0.90	0.87	0.85	0.95
AA+LF+CNG+MDW	0.90	0.96	0.95	0.78	0.97	0.76	0.91	0.88	0.91	0.95
AA+LF+PNG+MDW	0.90	0.96	0.94	0.77	0.97	0.73	0.89	0.86	0.92	0.95
LF+CNG+PNG+MDW	0.91	0.96	0.94	0.76	0.97	0.77	0.91	0.89	0.89	0.95
AA+CNG+PNG+MDW	0.92	0.97	0.96	0.93	$\underline{0.98}$	0.82	0.94	0.92	0.94	0.96
Full set	0.92	0.97	0.96	0.76	0.97	0.79	0.92	0.89	0.90	0.95

In Table 5, classification accuracies obtained by different feature sets on the ensemble learning methods are presented. As it can be observed from the classification accuracies presented in Table 5, the proposed feature list (CF), which combines features used in AA, CNG, PNG and the frequency of the MDW, yields the highest predictive performance in conjunction with ensemble learning methods on Book and Camera data sets. The second highest predictive performance is generally obtained by the full set–based representation of text documents. For Book and Camera data sets, the second predictive performances on AdaBoost ensemble of KNN algorithm, Bagging ensemble of KNN algorithm and Random Subspace ensemble of KNN algorithm are achieved by the representation of text documents based on the features used in AA. Regarding the classification accuracies presented in Tables 3 and 5, the predictive performance of text genre classification schemes is generally enhanced with the use of ensemble learning methods. Among the all configurations compared in Tables 3 and 5, the highest predictive performances on Book and Camera data sets are obtained by Random Subspace ensemble of RF algorithm. The predictive performances obtained by Bagging ensemble of RF are also relatively high.

Table 5.

Classification accuracies obtained by the ensemble learning methods with different feature sets

Book data set
Algorithm	AA	LF	CNG	PNG	MDW	CF	Full set
AdaBoost (NB)	58.54	68.48	71.08	71.69	72.85	88.14	86.40
AdaBoost (SVM)	65.36	73.03	75.26	73.36	73.83	91.62	87.35
AdaBoost (LR)	65.34	72.91	75.43	72.42	73.14	92.04	87.85
AdaBoost (KNN)	77.47	72.57	76.29	74.36	69.93	81.25	68.73
AdaBoost (RF)	82.96	86.12	85.57	83.53	89.17	93.01	89.45
Bagging (NB)	58.89	68.12	69.95	65.32	72.81	83.50	83.89
Bagging (SVM)	66.51	74.04	77.04	74.45	75.36	90.72	87.52
Bagging (LR)	66.42	73.81	76.23	72.73	73.85	88.95	87.35
Bagging (KNN)	75.86	70.81	74.18	72.20	67.21	77.14	65.11
Bagging (RF)	83.03	86.61	86.62	84.14	90.57	94.04	90.70
RS (NB)	57.74	67.18	70.65	64.57	71.71	84.97	83.72
RS (SVM)	64.92	71.02	72.49	72.30	71.26	89.34	87.96
RS (LR)	64.43	70.40	71.92	70.42	70.74	86.88	86.51
RS (KNN)	82.18	79.46	81.75	79.59	78.67	83.06	74.96
RS (RF)	84.30	87.94	87.91	86.26	91.43	$\underline{94.43}$	92.60
Camera data set
Algorithm	AA	LF	CNG	PNG	MDW	CF	Full set
AdaBoost (NB)	56.66	56.07	65.27	60.11	69.34	86.39	82.57
AdaBoost (SVM)	64.27	58.78	71.77	63.34	72.86	89.85	84.68
AdaBoost (LR)	62.73	59.57	69.79	62.15	73.15	90.12	84.88
AdaBoost (KNN)	80.02	78.06	77.67	75.43	78.17	86.83	79.43
AdaBoost (RF)	85.38	85.33	85.74	81.40	85.67	91.25	87.71
Bagging (NB)	56.54	55.52	65.26	60.14	69.06	80.02	77.76
Bagging (SVM)	64.65	61.06	74.06	63.87	75.80	89.17	84.63
Bagging (LR)	62.85	61.42	71.82	62.58	74.56	86.91	84.50
Bagging (KNN)	78.64	77.02	75.41	73.69	77.36	84.65	77.81
Bagging (RF)	86.78	86.74	87.75	83.93	87.28	92.36	89.17
RS (NB)	57.13	54.17	64.43	59.17	68.45	79.48	77.88
RS (SVM)	62.49	59.71	68.45	61.35	72.22	86.78	83.89
RS (LR)	62.01	59.01	67.59	59.61	71.95	83.95	82.65
RS (KNN)	84.97	83.98	82.12	79.82	83.39	89.38	82.51
RS (RF)	87.34	87.23	88.20	84.45	87.82	$\underline{93.81}$	89.26

AA: features used in authorship attribution; LF: linguistic features; CNG: character n-grams; PNG: part of speech n-grams; MDW: the frequency of the most discriminative words; full set: AA+LF+CNG+PNG+MDW; CF: AA+CNG+PNG+MDW; NB: Naïve Bayes; SVM: support vector machines; LR: logistic regression; KNN: k-nearest neighbour algorithm; RF: Random Forest; RS: Random Subspace.

In Table 6, F-measure values obtained by different feature sets on the ensemble learning methods are presented. As it can be observed from F-measure values presented in Table 6, the proposed feature list (CF), which combines features used in AA, CNG, PNG and the frequency of the MDW, yields the highest F-measure values in conjunction with ensemble learning methods on Book and Camera data sets, as in the case of classification accuracies presented in Table 5. The second highest predictive performance is generally obtained by the full set–based representation of text documents. Among the all configurations compared in Tables 4 and 6, the highest F-measure values on Book and Camera data sets are obtained by Random Subspace ensemble of RF algorithm. The predictive performances obtained by Bagging ensemble of RF are also relatively high. Regarding results presented in the experimental analysis, the representation of text documents by a CF list, which contains features used in AA, CNG, PNG and the frequency of the MDW yields the optimal representation. Regarding the predictive performance of classification algorithms and ensemble learning methods in terms of classification accuracy and F-measure values, Random Subspace ensemble of RF algorithm yields promising results.

Table 6.

F-measure values obtained by the ensemble learning methods with different feature sets

Book data set
Algorithm	AA	LF	CNG	PNG	MDW	CF	Full set
AdaBoost (NB)	0.63	0.74	0.75	0.77	0.76	0.97	0.96
AdaBoost (SVM)	0.70	0.81	0.82	0.81	0.76	0.98	0.98
AdaBoost (LR)	0.70	0.80	0.81	0.79	0.75	0.98	0.98
AdaBoost (KNN)	0.88	0.86	0.87	0.87	0.80	0.93	0.76
AdaBoost (RF)	0.96	0.96	0.97	0.97	0.97	0.98	0.98
Bagging (NB)	0.64	0.74	0.75	0.70	0.76	0.92	0.92
Bagging (SVM)	0.71	0.82	0.83	0.82	0.78	0.97	0.96
Bagging (LR)	0.71	0.82	0.81	0.79	0.76	0.96	0.95
Bagging (KNN)	0.86	0.84	0.84	0.84	0.77	0.91	0.72
Bagging (RF)	0.94	0.95	0.96	0.96	0.96	0.98	0.97
RS (NB)	0.63	0.73	0.75	0.70	0.75	0.91	0.90
RS (SVM)	0.70	0.78	0.78	0.79	0.74	0.96	0.94
RS (LR)	0.69	0.77	0.78	0.77	0.73	0.94	0.92
RS (KNN)	0.89	0.85	0.84	0.82	0.88	0.94	0.87
RS (RF)	0.96	0.97	0.97	0.97	0.97	$\underline{0.99}$	0.98
Camera data set
Algorithm	AA	LF	CNG	PNG	MDW	CF	Full set
AdaBoost (NB)	0.54	0.42	0.65	0.61	0.68	0.91	0.89
AdaBoost (SVM)	0.61	0.54	0.75	0.69	0.76	0.95	0.93
AdaBoost (LR)	0.59	0.55	0.71	0.66	0.76	0.95	0.93
AdaBoost (KNN)	0.90	0.90	0.86	0.89	0.92	0.91	0.90
AdaBoost (RF)	0.94	0.96	0.96	0.96	0.96	0.97	0.97
Bagging (NB)	0.53	0.41	0.65	0.62	0.67	0.82	0.78
Bagging (SVM)	0.62	0.58	0.77	0.70	0.79	0.94	0.91
Bagging (LR)	0.58	0.58	0.73	0.67	0.77	0.91	0.89
Bagging (KNN)	0.91	0.89	0.83	0.86	0.90	0.93	0.87
Bagging (RF)	0.95	0.96	0.96	0.96	0.96	$\underline{0.98}$	0.97
RS (NB)	0.53	0.38	0.65	0.60	0.66	0.81	0.79
RS (SVM)	0.58	0.53	0.69	0.64	0.73	0.91	0.89
RS (LR)	0.57	0.52	0.68	0.61	0.73	0.87	0.86
RS (KNN)	0.93	0.92	0.91	0.91	0.91	0.96	0.93
RS (RF)	0.95	0.94	0.94	0.94	0.96	$\underline{0.98}$	0.97

AA: features used in authorship attribution; LF: linguistic features; CNG: character n-grams; PNG: part of speech n-grams; MDW: the frequency of the most discriminative words; full set: AA+LF+CNG+PNG+MDW; CF: AA+CNG+PNG+MDW; NB: Naïve Bayes; SVM: support vector machines; LR: logistic regression; KNN: k-nearest neighbour algorithm; RF: Random Forest; RS: Random Subspace.

To further evaluate the results obtained in the empirical analysis, we performed general linear model analysis to perform analysis of variance (ANOVA) test in Minitab statistical programme. The results for ANOVA test of overall results obtained by classification algorithms and different feature sets are summarised in Table 7.

Table 7.

ANOVA test results

Source (accuracy values)	DF	SS	MS	F	p
Feature sets	6	10470.9	1745.14	429.44	0.000
Classifiers	19	10201.2	536.91	132.12	0.000
Data sets	1	359.2	359.18	88.39	0.000
Feature sets*classifiers	114	4144.9	36.36	8.95	0.000
Feature sets*data sets	6	463.9	77.32	19.03	0.000
Classifiers*data sets	19	1144.9	60.26	14.83	0.000
Error	114	463.3	4.06
Total	279	27248.3
Source (F-measure values)	DF	SS	MS	F	p
Feature sets	6	1.3056	0.217600	179.70	0.000
Classifiers	19	2.1531	0.113323	93.58	0.000
Data sets	1	0.2178	0.217843	179.90	0.000
Feature sets*classifiers	114	0.7900	0.006930	5.72	0.000
Feature sets*data sets	6	0.1550	0.025826	21.33	0.000
Classifiers*data sets	19	0.3287	0.017300	14.29	0.000
Error	114	0.1380	0.001211
Total	279	5.0883

DF: degrees of freedom; SS: adjusted sum of squares; MS: adjusted mean square; F: F-statistics; p: probability value.

As it can be observed from the results presented in Table 7, there are statistically meaningful differences between the results of compared feature sets, the results of compared classification algorithms and the results of data sets at 99% confidence level. The p-values (p < 0.001) indicate that the three main factors (feature sets, classifiers and data sets) have statistically significant effect on the evaluation metric values. It can also be seen from Table 7 that the interactions between feature sets and classifiers, feature sets and data sets and classifiers and data sets are also important at the level (p < 0.001).

To summarise the main results from the empirical analysis, we have presented the main effects plots for accuracy and F-measure values in Figures 2 and 3, respectively. The main effect plots briefly describe the change of particular metric’s value based on a factor. In our empirical analysis, feature sets, classifiers and data sets are the three main effects. As it can be observed from Figures 2 and 3, the proposed feature list (CF), which combines features used in AA, CNG, PNG and the frequency of the MDW, obtains the highest classification accuracies and F-measure values. The second representation scheme regarding the performance of feature sets on text genre classification is the utilisation of full set. The worst results for each metric are obtained by the features used in AA. Besides, the figures indicate the performance of classification algorithms and ensemble learning methods on text genre classification. Figure 2 clearly depicts that the predictive performance of Random Subspace ensemble of RF is higher compared with the supervised learning algorithms and the other ensemble learning schemes. Regarding the F-measure values of classification algorithms, RF algorithm, AdaBoost ensemble of RF, Bagging ensemble of RF and Random Subspace ensemble of RF algorithm yield higher values. As it can be observed from the figures, metric values obtained on Book data set are higher compared with the metric values of Camera data set.

Figure 2.

Main effects plot for accuracy results.

Figure 3.

Main effects plot for F-measure results.

Several managerial insights from the results of experimental analysis presented in Tables 3 –7 can be summarised as follows:

Regarding the performance of individual feature sets, the frequency of the MDW and CNG is the most significant feature sets related to the identification of text genre; although features used in AA and PNG are not significant as the frequency of the MDW and CNG, they still act as important feature sets.

Regarding the performance of classification algorithms on different feature sets, the highest predictive performance is obtained by RF algorithm. The predictive performance of SVMs is also relatively high. The higher results on classifiers, such as RF and SVMs, are also valid for other recent empirical analysis [3,52]. The use of RF algorithm in conjunction with different ensemble learning algorithms yields promising results compared with the other ensemble schemes.

Ensemble learning can be used to improve the predictive performance of classification algorithms on text genre classification. Regarding the performance of ensemble learning methods in conjunction with different classification algorithms, the highest classification accuracies are obtained by Random Subspace ensemble of RF algorithm. Random Subspace ensemble aims to achieve diversity among the base learners by partitioning the feature space. A potential explanation to the higher predictive performance obtained by Random Subspace ensembles is that the feature space–based partitioning can enhance the diversity and predictive performance on text genre classification.

Compared with the individual feature sets, CF sets generally yield better predictive performance. Although the highest predictive performance is obtained by a feature set, which combines features used in AA, CNG, PNG and the frequency of the MDW, the integration of LF to this set may degrade the predictive performance. Hence, the identification of an optimal subset of features (instead of using all available features) not only enhances the scalability but also improves the predictive performance.

The feature sets and classification schemes (classification algorithms and/or ensemble learning methods) have statistically significant impact on the predictive performance. In addition, the performance of different feature sets, classification schemes and data sets have similar performance patterns in terms of both classification accuracy and F-measure values.

6.4. Case study on topical text classification

In order to examine the predictive performance of the ensemble learning methods and the CF lists, we also performed experimental analysis on topical text classification. For topical text classification, two benchmark text categorisation data sets, namely, Reuters-21578 and 20-NG, are utilised.

6.4.1. Topical text classification benchmarks

To examine the predictive performance of ensemble learning methods and the different feature sets on text classification, Reuters-21578 and 20-NG document collections are used. The Reuters-21578 data set utilised in the empirical analysis contains 10 classes of Reuters-21578 ModApte Split [62,63]. The basic descriptive information regarding the number of training and testing instances on Reuters-21578 data set is presented in Table 8. The second document collection utilised in the empirical analysis is 20-NG (20 Newsgroups) data set. It contains 20,000 messages taken from 20 Usenet groups. The number of terms and the number of classes for the data set are 45,434 and 20, respectively. In order to examine the performance of different feature sets on topical text categorisation, the same individual feature sets outlined in section 5.2 are utilised. The details regarding the experimental procedures and evaluation measures presented in sections 6.1 and 6.2 are also valid for the empirical analysis on topical text classification.

Table 8.

Basic descriptive information about Reuters-21578 data set [63]

Class label	Training samples	Testing samples
Earn	2877	1087
Acq	1650	719
Money-fx	538	179
Grain	433	149
Crude	389	189
Trade	369	117
Interest	347	131
Ship	197	89
Wheat	212	71
Corn	181	56

6.4.2. Results on topical text classification

This section presents the predictive performance of feature sets, classification algorithms and ensemble learning methods on topical text classification. In Table 9, the classification accuracies obtained by the individual feature sets and the CF set on the classification algorithms and ensemble learning methods are presented.

Table 9.

Classification accuracies obtained by the ensemble learning methods with different feature sets

Reuters-21578 data set
Algorithm	AA	LF	CNG	PNG	MDW	CF	Full set
NB	71.54	79.44	80.83	76.09	81.08	85.96	85.04
SVM	63.65	71.27	73.25	68.38	72.51	79.72	79.49
LR	67.13	73.54	75.96	71.09	75.61	80.99	78.59
KNN	69.03	76.38	78.22	72.98	77.91	84.23	82.19
RF	76.02	81.71	84.51	78.73	83.39	90.54	88.73
AdaBoost (NB)	74.09	80.95	83.35	77.94	83.14	87.82	87.38
AdaBoost (SVM)	64.77	71.73	73.91	68.81	73.11	81.07	79.18
AdaBoost (LR)	67.70	74.98	76.61	72.52	77.65	82.45	79.75
AdaBoost (KNN)	72.22	78.64	81.23	74.75	80.03	85.52	84.06
AdaBoost (RF)	78.10	84.22	86.25	82.05	86.55	92.19	90.27
Bagging (NB)	74.57	82.09	83.83	78.61	84.32	89.34	88.33
Bagging (SVM)	65.35	71.47	75.47	71.15	73.96	81.77	81.06
Bagging (LR)	68.77	75.49	77.81	73.28	77.72	82.45	82.44
Bagging (KNN)	73.62	80.72	83.53	77.33	81.50	88.12	86.96
Bagging (RF)	79.66	85.94	88.02	83.94	87.98	91.69	91.59
RS (NB)	76.09	83.14	84.73	80.30	85.17	89.69	88.53
RS (SVM)	67.95	73.28	75.93	72.18	75.45	83.30	82.05
RS (LR)	70.87	77.82	79.81	75.08	79.77	85.11	84.49
RS (KNN)	76.79	84.47	84.76	79.27	83.65	90.20	89.16
RS (RF)	81.73	87.80	88.86	85.61	89.93	$\underline{93.02}$	92.62
20-NG data set
Algorithm	AA	LF	CNG	PNG	MDW	CF	Full set
NB	75.14	82.28	83.55	78.62	83.40	88.52	87.94
SVM	67.47	73.42	75.14	70.71	75.54	82.65	81.16
LR	70.09	76.38	78.45	73.48	79.12	83.38	81.20
KNN	71.61	78.78	80.73	75.68	80.53	86.81	84.66
RF	78.40	84.88	86.30	81.99	86.50	93.82	90.69
AdaBoost (NB)	76.25	84.22	85.24	80.72	85.49	90.52	90.10
AdaBoost (SVM)	67.33	73.75	75.87	71.61	76.17	83.42	82.13
AdaBoost (LR)	71.09	77.14	79.63	74.58	80.38	84.68	82.56
AdaBoost (KNN)	74.50	82.02	83.32	77.76	82.24	88.96	86.23
AdaBoost (RF)	80.45	86.94	89.59	84.98	88.65	94.46	92.39
Bagging (NB)	77.96	84.47	86.80	82.06	86.34	91.46	90.41
Bagging (SVM)	68.73	74.64	77.91	73.67	77.04	84.89	83.72
Bagging (LR)	71.62	78.92	80.82	75.75	81.09	85.69	84.78
Bagging (KNN)	76.75	83.64	85.31	79.91	84.44	90.52	89.44
Bagging (RF)	82.64	88.81	90.67	86.81	90.45	94.97	94.02
RS (NB)	78.41	85.85	88.15	82.22	87.94	93.35	92.20
RS (SVM)	71.16	75.78	79.27	74.60	77.45	85.97	84.54
RS (LR)	73.91	81.24	82.50	77.50	82.89	87.98	86.85
RS (KNN)	79.49	86.44	87.83	82.79	86.55	92.35	91.38
RS (RF)	84.91	90.67	91.81	88.47	92.88	$\underline{95.73}$	94.23

AA: features used in authorship attribution; LF: linguistic features; CNG: character n-grams; PNG: part of speech n-grams; MDW: the frequency of the most discriminative words; full set: AA+LF+CNG+PNG+MDW; CF: AA+CNG+PNG+MDW; NB: Naïve Bayes; SVM: support vector machines; LR: logistic regression; KNN: k-nearest neighbour algorithm; RF: Random Forest; RS: Random Subspace.

In Table 10, F-measure values obtained by the individual feature sets and the CF set on the classification algorithms and ensemble learning methods are presented. In terms of classification accuracy, the highest predictive performance is obtained by Random Subspace ensemble of RF algorithm when the CF set is utilised. The empirical analysis on topical text categorisation collections indicate that the ensemble learning methods generally yield better predictive performance in terms of accuracy and F-measure values on topical text classification, as well. As summarised in the managerial insights of the empirical analysis presented in section 6.3, CF list can enhance the predictive performance compared with the individual feature sets.

Table 10.

F-measure values obtained by the ensemble learning methods with different feature sets

Reuters-21578 data set
Algorithm	AA	LF	CNG	PNG	MDW	CF	Full set
NB	0.71	0.77	0.78	0.74	0.79	0.83	0.83
SVM	0.63	0.69	0.71	0.66	0.71	0.77	0.77
LR	0.67	0.71	0.74	0.69	0.74	0.79	0.76
KNN	0.68	0.74	0.76	0.71	0.76	0.82	0.80
RF	0.75	0.79	0.82	0.76	0.82	0.88	0.86
AdaBoost (NB)	0.73	0.79	0.81	0.76	0.82	0.85	0.85
AdaBoost (SVM)	0.64	0.70	0.72	0.67	0.72	0.79	0.77
AdaBoost (LR)	0.67	0.73	0.74	0.70	0.76	0.80	0.77
AdaBoost (KNN)	0.72	0.76	0.79	0.73	0.78	0.83	0.82
AdaBoost (RF)	0.77	0.82	0.84	0.80	0.85	0.89	0.88
Bagging (NB)	0.74	0.80	0.81	0.76	0.83	0.87	0.86
Bagging (SVM)	0.65	0.69	0.73	0.69	0.73	0.79	0.79
Bagging (LR)	0.68	0.73	0.76	0.71	0.76	0.80	0.80
Bagging (KNN)	0.73	0.78	0.81	0.75	0.80	0.86	0.84
Bagging (RF)	0.79	0.83	0.85	0.81	0.86	0.89	0.89
RS (NB)	0.75	0.81	0.82	0.78	0.84	0.87	0.86
RS (SVM)	0.67	0.71	0.74	0.70	0.74	0.81	0.80
RS (LR)	0.70	0.76	0.77	0.73	0.78	0.83	0.82
RS (KNN)	0.76	0.82	0.82	0.77	0.82	0.88	0.87
RS (RF)	0.81	0.85	0.86	0.83	0.88	$\underline{0.90}$	0.90
20-NG data set
Algorithm	AA	LF	CNG	PNG	MDW	CF	Full set
NB	0.73	0.80	0.81	0.76	0.82	0.86	0.85
SVM	0.66	0.71	0.73	0.69	0.74	0.80	0.79
LR	0.68	0.74	0.76	0.71	0.78	0.81	0.79
KNN	0.70	0.76	0.78	0.73	0.79	0.84	0.82
RF	0.76	0.82	0.84	0.80	0.85	0.91	0.88
AdaBoost (NB)	0.74	0.82	0.83	0.78	0.84	0.88	0.87
AdaBoost (SVM)	0.65	0.72	0.74	0.70	0.75	0.81	0.80
AdaBoost (LR)	0.69	0.75	0.77	0.72	0.79	0.82	0.80
AdaBoost (KNN)	0.72	0.80	0.81	0.75	0.81	0.86	0.84
AdaBoost (RF)	0.78	0.84	0.87	0.83	0.87	0.92	0.90
Bagging (NB)	0.76	0.82	0.84	0.80	0.85	0.89	0.88
Bagging (SVM)	0.67	0.72	0.76	0.72	0.76	0.82	0.81
Bagging (LR)	0.70	0.77	0.78	0.74	0.80	0.83	0.82
Bagging (KNN)	0.75	0.81	0.83	0.78	0.83	0.88	0.87
Bagging (RF)	0.80	0.86	0.88	0.84	0.89	0.92	0.90
RS (NB)	0.76	0.83	0.86	0.80	0.86	0.91	0.90
RS (SVM)	0.69	0.74	0.77	0.72	0.76	0.83	0.82
RS (LR)	0.72	0.79	0.80	0.75	0.81	0.85	0.84
RS (KNN)	0.77	0.84	0.85	0.80	0.85	0.90	0.89
RS (RF)	0.82	0.88	0.89	0.86	0.91	$\underline{0.93}$	0.91

AA: features used in authorship attribution; LF: linguistic features; CNG: character n-grams; PNG: part of speech n-grams; MDW: the frequency of the most discriminative words; full set: AA+LF+CNG+PNG+MDW; CF: AA+CNG+PNG+MDW; NB: Naïve Bayes; SVM: support vector machines; LR: logistic regression; KNN: k-nearest neighbour algorithm; RF: Random Forest; RS: Random Subspace.

To summarise, the experimental analysis on different feature sets, classification algorithms and ensemble learning methods indicate that, the use of CF sets and ensemble learning methods can enhance the predictive performance on text genre classification. Regarding the performance of topical text classification, the identification of an appropriate set of features is a key issue in predictive performance. In this regard, a recent empirical analysis on ACM and Reuters-21578 document collections for topical text classification indicates that ensemble learning methods can yield promising results in keyword-based representation of text documents [3]. Similarly, an extensive empirical analysis is conducted on sentiment analysis with different feature sets and classification algorithms [61]. In this empirical analysis, different individual feature sets (such as unigrams, part-of-speech-based features and bigrams) and the CF sets obtained from individual feature sets are taken into consideration. The experimental analysis indicated that the CF sets can yield promising results on sentiment classification. Compared with the topical text classification, text genre classification is a more challenging problem, and the idea of using a CF set from promising feature subsets and the utilisation of the paradigm of ensemble learning can be viable tools to build classification schemes on topical text classification, as well.

7. Conclusion

Text genre classification concerns with identifying functional characteristics or kinds of text documents. The genre identification can reveal information about the functions and communicative contexts of text documents. Due to the immense quantity of information available on the web, text genre classification can be extremely useful in several fields, such as document filtering and review spam detection. Genre is a broad term with several aspects, such as function and purpose of text documents. The LFA aims to focus on a single aspect of genres and it aims to classify text documents into three abstract and general classes, as expressive, appellative and informative. To build a robust classification scheme with high predictive performance, the identification of an appropriate feature set is extremely important on text genre classification. We considered five feature sets (features used in AA, LF, CNG, PNG and the frequency of the MDW) and the CF sets obtained from these feature sets. In the classification phase, five different base learners (NB, SVMs, LR, KNN and RF) in conjunction with ensemble learning methods (such as Boosting, Bagging and Random Subspace) are examined. Based on the empirical analysis, the proposed feature list (CF), which combines features used in AA, CNG, PNG and the frequency of the MDW, yields the highest classification accuracies and F-measure values in conjunction with Random Subspace ensemble of RF. For LFA corpus, the highest classification accuracy obtained on Book data set is 94.43%, and the highest classification accuracy obtained on Camera data set is 93.81%.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

Notes

References

Han

Kamber

. Data mining: concepts and techniques. 2nd ed.San Francisco, CA: Morgan Kaufmann Publishers, 2006, p. 800.

Aggarwal

Zhai

. A survey of text classification algorithms. In: Aggarwal

Zhai

(eds) Mining text data. 1st ed.Berlin: Springer-Verlag, 2012, pp. 77–128.

Onan

Korukoglu

Bulut

. Ensemble of keyword extraction methods and classifiers in text classification. Expert Syst Appl 2016; 57: 232–247.

Jebari

. A new centroid-based approach for genre categorization of web pages. J Lang Technol Comput Linguist 2009; 24(1): 73–96.

Stamatatos

Kokkinakis

Fakotakis

. Automatic text categorization in terms of genre and author. Comput Linguist 2000; 26(4): 471–495.

Kessler

Nunberg

Schütze

. Automatic detection of text genre. In: Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL), Madrid, Spain, 11–12 July 1997, pp. 32–38. Stroudsburg, PA: ACL.

Rauber

Müller-Kögler

. Integrating automatic genre analysis into digital libraries. In: Proceedings of the ACM/IEEE first joint conference on digital libraries, Roanoke, VA, 24–28 June 2001, pp. 1–10. New York: ACM.

Santini

Mehler

Sharoff

. Riding the rough waves of genre on the web. In: Mehler

Sharoff

Santini

(eds) Genres on the web. 1st ed. Berlin: Springer-Verlag, 2011, pp. 3–30.

Wachsmuth

Bujna

. Back to the roots of genres: text classification by language function. In: Proceedings of the 5th international joint conference on natural language processing, Chiang Mai, Thailand, 8–13 November 2011, pp. 632–640. Stroudsburg, PA: ACL.

10.

Lee

Y-B

Myaeng

. Text genre classification with genre-revealing and subject-revealing features. In: Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval, Tampere, 11–15 August 2002, pp. 145–150. New York: ACM.

11.

Petrenz

. Assessing approaches to genre classification. MSc Thesis, University of Edinburgh, Edinburgh, 2009.

12.

Karlgren

Cutting

. Recognizing text genres with simple metrics using discriminant analysis. In: Proceedings of the 15th conference on computational linguistics, Kyoto, Japan, 5–9 August 1994, pp. 1071–1075. Stroudsburg, PA: ACL.

13.

Onan

Korukoglu

. A feature selection model based on genetic rank aggregation for text sentiment classification. J Inf Sci. Epub ahead of print 5 November 2015. DOI: 10.1177/0165551515613226.

14.

Levering

Cutler

. Cost-sensitive feature extraction and selection in genre classification. J Lang Technol Comput Linguist 2009; 24(1): 57–72.

15.

Prabowo

Thelwall

. Sentiment analysis: a combined approach. J Informetr 2009; 3: 143–157.

16.

Wang

Sun

et al . Sentiment classification: the contribution of ensemble learning. Decis Support Syst 2014; 57: 77–93.

17.

Onan

. Classifier and feature set ensembles for web page classification. J Inf Sci 2016; 42(2): 150–165.

18.

Wolters

Kirsten

. Exploring the use of linguistic features in domain and genre classification. In: Proceedings of the 9th conference on European chapter of the association for computational linguistics, Bergen, 8–12 June 1999, pp. 142–149. Stroudsburg, PA: ACL.

19.

Stamatatos

Fakotakis

Kokkinakis

. Text genre detection using common word frequencies. In: Proceedings of the 18th conference on computational linguistics, Saarbrücken, 31 July–4 August 2000, pp. 808–814. Stroudsburg, PA: ACL.

20.

Freund

Clarke

CLA

Toms

. Towards genre classification for IR in the workplace. In: Proceedings of the 1st international conference on information interaction in context, Copenhagen, 18–20 October 2006, pp. 30–36. New York: ACM.

21.

Roussinov

Crowston

Nilan

et al . Genre based navigation on the web. In: Proceedings of the 34th Hawaii international conference on system sciences, Maui, HI, 3–6 January 2001, pp. 1–10. New York: IEEE.

22.

Dewdney

Vaness-Dykema

MacMillan

. The form is the substance: classification of genres in text. In: Proceedings of the workshop on human language technology and knowledge management, Toulouse, 6–7 July 2001, pp. 1–8. Stroudsburg, PA: ACL.

23.

HaCohen-Kerner

Beck

Yehudai

et al . Cuisine: classification using stylistic feature sets and/or name-based feature sets. J Assoc Inf Sci Technol 2010; 61(8): 1644–1657.

24.

Ferizis

Bailey

. Towards practical genre classification of web documents. In: Proceedings of the 15th international conference on World Wide Web, Edinburgh, 23–26 May 2006, pp. 1013–1014. New York: ACM.

25.

Ding

Wang

et al . Genre identification of Chinese finance text using machine learning method. In: Proceedings of the IEEE international conference on systems, man and cybernetics, 2008 (SMC 2008), Singapore, 12–15 October 2008, pp. 455–459. New York: IEEE.

26.

Kröll

Strohmaier

. Analyzing human intentions in natural language text. In: Proceedings of the fifth international conference on knowledge capture, Redondo Beach, CA, 1–4 September 2009, pp. 197–198. New York: ACM.

27.

Petrenz

Webber

. Stable classification of text genres. Comput Linguist 2011; 37(2): 385–393.

28.

Vilares

Alonso

Gomez-Rodriguez

. A linguistic approach for determining the topics of Spanish twitter messages. J Inf Sci 2015; 41(2): 127–145.

29.

Hajibayova

Jacob

. An investigation of the levels of abstraction of tags across three resource genres. Inform Process Manag 2016; 52: 1178–1187.

30.

Rosso

. User-based identification of web genres. J Assoc Inf Sci Technol 2008; 59(7): 1053–1072.

31.

J-C

Thet

. Effectiveness of web search results for genre and sentiment classification. J Inf Sci 2009; 35(6): 709–726.

32.

Tseng

Y-H

Wang

Y-M

Lin

et al . Patent surrogate extraction and evaluation in the context of patent mapping. J Inf Sci 2007; 33(6): 718–736.

33.

Codina-Filba

Bouayad-Agha

Burga

et al . Using genre-specific features for patent summaries. Inform Process Manag. Epub ahead of print 29 July 2016. DOI: 10.1016/j.ipm.2016.07.002.

34.

Rodger

. A fuzzy nearest neighbor neural network statistical model for predicting demand for natural gas and energy cost savings in public buildings. Expert Syst Appl 2014; 41: 1813–1829.

35.

Rodger

. Discovery of medical Big Data analytics: improving the prediction of traumatic brain injury survival rates by data mining Patient Informatics Processing Software Hybrid Hadoop Hive. Inform Med Unlocked 2015; 1: 17–26.

36.

Chen

Girgensohn

Cooper

et al . Genre identification for office document search and browsing. Int J Doc Anal Recog 2012; 15: 167–182.

37.

Finn

Kushmerick

. Learning to classify documents according to genre. J Am Soc Inf Sci Tec 2006; 57(11): 1506–1518.

38.

Gencosman

Ozmutlu

. Character n-gram application for automatic new topic identification. Inform Process Manag 2014; 50: 821–856.

39.

Reiss

. Möglichkeiten und Grenzen der Übersetzungskritik. Munich: Max Hueber Verlag, 1971.

40.

Reiss

Vermeer

. Fundamentos para una teoria funcional de la traduccióon. Madrid: Ediciones Akal, 1984.

41.

Alvaro

LMRC

Fuhihara

. Textual genres on discourse analysis and translation functionalism. In: Proceedings of the 24th International Meeting of the Association of Young Linguists, Barcelona, Spain, 24–26 April 2009, pp. 1–10. Poznan: Adam Mickiewicz University.

42.

Kotsiantis

Zaharakis

Pintelas

. Machine learning: a review of classification and combining techniques. Artif Intell Rev 2006; 26(3): 159–190.

43.

Ikonomakis

Kotsiantis

Tampakas

. Text classification using machine learning techniques. WSEAS Trans Comput 2005; 8(4): 966–974.

44.

John

Langley

. Estimating continuous distributions in Bayesian classifiers. In: Proceedings of the 11th conference on uncertainty in artificial intelligence, Montréal, QC, Canada, 18–20 August 1995, pp. 338–345. San Francisco, CA: Morgan Kaufmann.

45.

Vapnik

. The nature of statistical learning theory. New York: Springer, 1995.

46.

Witten

Frank

Hall

. Data mining: practical machine learning tools and techniques. Burlington, MA: Morgan Kaufmann, 2011, p. 629.

47.

Aha

Kibler

Albert

. Instance-based learning algorithm. Mach Learn 1991; 6: 37–66.

48.

Breiman

. Random forests. Mach Learn 2011; 45(1): 5–32.

49.

Zhou

Z-H

. Ensemble methods: foundations and algorithms. New York: Chapman and Hall/CRC, 2012.

50.

Dietterich

. Ensemble methods in machine learning. In: Proceedings of the 1st international workshop on multiple classifier systems, Cagliari, 21–23 June 2000, pp. 1–15. Berlin: Springer.

51.

Rokach

. Ensemble-based classifiers. Artif Intell Rev 2010; 33(1): 1–39.

52.

Wang

Zhang

Sun

et al . POS-RS: a Random subspace method for sentiment classification based on part-of-speech analysis. Inform Process Manag 2015; 51(4): 458–479.

53.

Kuncheva

. Combining pattern classifiers: methods and algorithms. Hoboken, NJ: John Wiley & Sons Publishers, 2014.

54.

Breiman

. Bagging predictors. Mach Learn 1996; 24: 123–140.

55.

Kuncheva

Whitaker

. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 2003; 51: 181–207.

56.

. The random subspace method for constructing decision forests. IEEE T Pattern Anal 1998; 20(8): 832–844.

57.

Carletta

. Assessing agreement on classification tasks: the kappa statistic. Comput Linguist 1996; 22: 249–254.

58.

Stamatatos

. A survey of modern authorship attribution methods. J Am Soc Inf Sci Tec 2009; 60(3): 538–556.

59.

Santini

. Automatic text analysis: gradations of text types in web pages. In: Proceedings of the 10th ESSLLI Student Session, Edinburgh, UK, 8–19 August 2005, pp. 276–285. Edinburgh: Heriot-Watt University.

60.

Hall

Frank

Homes

et al . The Weka data mining software: an update. ACM SIGKDD Explor Newsl 2009; 11(1): 10–18.

61.

Xia

Zong

. Ensemble of feature sets and classification algorithms for sentiment classification. Inform Sciences 2011; 181(6): 1138–1152.

62.

Asuncion

Neman

. UCI machine learning repository. Irvine, CA: University of California, Irvine, 2007.

63.

Uysal

. An improved global feature selection scheme for text classification. Expert Syst Appl 2016; 43: 82–92.