A survey on text classification and its applications

Abstract

Text classification (a.k.a text categorisation) is an effective and efficient technology for information organisation and management. With the explosion of information resources on the Web and corporate intranets continues to increase, it has being become more and more important and has attracted wide attention from many different research fields. In the literature, many feature selection methods and classification algorithms have been proposed. It also has important applications in the real world. However, the dramatic increase in the availability of massive text data from various sources is creating a number of issues and challenges for text classification such as scalability issues. The purpose of this report is to give an overview of existing text classification technologies for building more reliable text classification applications, to propose a research direction for addressing the challenging problems in text mining.

Keywords

Text classification text mining feature selection machine learning

1. Introduction

With the explosion of information resources on the Web and corporate intranets continues to increase, there is an imminent need for more effective and efficient technologies to help people search and manage these resources. Machine-learning for text classification or categorisation is an important technique for information organisation and management. The capacious researches have been invested in text classification and there are different kinds of classification techniques, including neural networks, decision trees, k-nearest neighbors, support vector machines, Naïve Bayes, linear least squares, Rocchio’s method, rules-based method etc., have been developed [7,9,28,29,43,57,85]. There are also many efficient and practical text classification applications in the areas such as information retrieval, text filtering, classification of news stories, categorisation of incoming e-mail message and memos, the classification of Web pages, classification of academic papers using technical domains and sub-domains, spam and porn filtering, bioinformatics, customer service automation, topic categorize and sentiment analysis etc. [12,20,48,70,75,86,89,99,100]. A number of researches have focused on the processing of textual information available in healthcare datasets to improve care while lowering costs [6] or using text mining technology to develop a computer-based diagnostic decision-support system for helping the physicians to make better medical decision [2] or applying medical data mining technology to detect adverse drug events [27].

However, the dramatic increase in the availability of massive text data from various sources is creating a number of issues and challenges for text analysis and classification. The main purpose of this document is to provide a survey of the development on these research issues and challenges.

The paper is organized as follows. The Section 2 begins with a brief introduction to text classification to provide the basic concept and background knowledge. In Section 3, we review some important feature selection method. Section 4 discusses classification algorithms. Section 5 presents some important applications of the real world. Section 6 concludes this study.

2. Text classification

Text classification is the process of classifying an incoming stream of documents into categories by using the classifiers learned from the training samples [49]. The Machine Learning (ML) approach to text classification has gained popularity and has eventually become the dominant one [70]. Using machine learning, the objective is to learn the classifiers from the examples that automatically perform the documents category. The input to a classifier is a training set of records, each of which is tagged with a class label. A set of attribute values defines each record. The goal is to induce a model or description for each class in terms of the attributes. The model is then used to classify future records whose classes are unknown. More formally, TC assigns a Boolean value to each pair $(d_{i}, c_{i}) \in (D \times C)$ , where D is a domain of documents and C is a set of predefined categories[70]. The task is to approximate the true function $φ : D \times C \to (1, 0)$ by means of a function $\hat{φ} : D \times C \to (1, 0)$ , such that φ and $\hat{φ}$ coincide as much as possible [18]. The function $\hat{φ}$ is called a classifier. The goal of the classifier is to precisely define and estimate this coincidence.

In generally, text classification problem can be a “binary” classification problem If there are exactly two classes or a “multi-class” problem if there are more than two classes and each document falls into exactly one class or a “multi-label categorization” problem if a document may have more than one associated category in a classification scheme [86]. The basic form of text classification is binary classification in which a text document was given one of two labels, usually referred to as positive and negative. Multi-label and multi-class tasks are often handled by reducing them to k binary classification tasks, one for each category [70,86]. For example, in [50] a multi-label classification problem was converted to be a set of multiple binary classification problems and then a simple convolutional neural networks (CNN) model applied to the text classification.

Current text classifiers cannot unambiguously describe the decision boundary between positive and negative objects owing to uncertainties caused by text feature selection and the knowledge learning process. To overcome this problem, a new three-way decision model was proposed recently. The objectives of the new model is to deal with the uncertain boundary to improve the binary text classification performance based on the rough set techniques and centroid solution [47].

Several algorithms have been proposed for binary classification including neural networks, decision trees, k-Nearest Neighbor, Naive Bayes classifier, rough set based classifier, and Support Vector Machines [90,91,97]. These algorithms can be naturally extended to the multi-class classification. Another way to solve mult-iclass problem is to convert the multi-class classification problem into a set of binary classification problems [1].

For multi-label, it must be transformed to single label first before processed in binary classification. At least four transformation approaches from a multi-label dataset into a single-label dataset were presented [8]. They are All Label Assignment (ALA), No Label Assignment (NLA), Largest Label Assignment (LLA), and Smallest Label Assignment (SLA). Among these approaches, ALA is usually the best; however, it suffers from the fact that duplicate documents with different label introduce noise and decrease categorization effectiveness. ALA is Problem Transformation 5 (PT5) in [76,77]. A new transformation Entropy-based Label Assignment (ELA) that modified ALA was proposed in [8]. In [92], more details of multi-label classificatoin are provided.

Fig. 1.

Machine learning for text classification.

Machine learning for text classification tasks can be categorised into supervised, semi-supervised, and unsupervised learning task. In supervised learning, the machine is presented with training examples consisting of input/output pair patterns where it is required to predict the output values of new examples based on their input values. Supervised learning require a set of training examples. However, sometimes, samples may be inadequate or insufficient, though available. Such a problem is referred to as semi-supervised text classification. A semi-supervised approach was proposed in [58] to learn classifiers from only partial label samples (the training documents are pre-classified into a set of possible classes with only one correct class). An approach was introduced in [18] to learn classifiers from only positive and unlabelled samples, without negative ones. The approach first extracts negative samples from the unlabelled set and then builds classifiers as usual. Supervised and semi-supervised text classification techniques more or less rely on pre-classified samples to learn classifiers. Unsupervised learning refers to the problem of trying to find hidden structure in unlabeled data. The authors of [83] proposed to build the classification model for a target class without associated training samples, by analysing the correlating auxiliary classes.

3. Feature selection

As shown in Fig. 1, the process of text categorization generally consists of the feature selection, classification model development and classification evaluation tasks. In this section, we will discuss feature selection, which is a very important task for text classification.

In text classification, a text document is typically represented as a feature vector. The feature vectors are typically high-dimensional but sparse. Feature selection is a technique that select a subset of the features available for describing the data before applying a learning algorithm. Feature selection techniques intend to remove non-informative features according to corpus statistics and to reduce dimensionality of data. Feature selection can increase the classification accuracy and decrease computational complexity by eliminating noise features. Therefore, effective feature selection is essential for improving the scalability, efficiency and accuracy of a text classifier [7]. For instance, in [87], Yang and Pedersen have shown that depending on the different classifier, the effectiveness can be moderately increased about (⩽5%) by using the term space reduction technique. In [79], rough set based feature selection method was proposed.

In the feature selection phase, the text document pre-processing task is performed first. At least three tasks in the pre-processing phase, which are the selection of initial/complete feature set, identification of related information which are not directly used as features, and dimensionality reduction.

3.1. Types of feature

The features can be simple structures (words), complex linguistic structures (e.g. phrases, lexical dependencies, part of speech (POS)), statistical structures (e.g. n-gram, patterns (a.k.a. termset)), supported information (e.g. word’s compactness, word’s first position), Named Entities (e.g., people or organization names), etc. in the document. The set of features consists of one or more types. Most systems use only a kind of feature (e.g. term). However, it was found in many works that using more than one types of features can increase classification performance [55,59].

3.1.1. Term-based feature

Terms (aka normalized words) are the most common type of feature in document representation. “The bag of words” model is a typical model used for representing and computing the feature values for a document. In that model, the features in the feature vector are a given word or phrase. The advantage of term-based feature is that a complex natural language document is transformed into a set of simple independent terms and using simple term feature make the classification efficient. However, relation information among terms is lost [72]. Another problem using terms as features is the semantic ambiguity: polysemy and synonymy problems. The polysemy problem means that terms can be used to express different things in different contexts (e.g. driving a car and driving results). This will affect precision. The Synonymy problem means that terms can be used to express the same thing (e.g. espionage and spy). This will affect recall.

Word sense is used to overcome synonym, polysemy and homonym which related to word meanings (a.k.a. senses). For example, on polysemy, word “phone” can be a noun as a device and a verb as to communicate. In English, a popular lexical database which provides senses of English words is WordNet [53]. In [30], the authors used a WordNet-based annotated (by linguist) corpus to compare word-based and sense-based features for categorization. Using a small training set (182 documents), they found that sense-based feature did not improve effectiveness significantly. The works in [54] also concluded that word-sense feature was not sufficient to improve text categorization effectiveness. Another investigation, however, indicated that WordNet’s Synsets relationship hierarchy usage help categorization performance [62].

3.1.2. Phrase-based feature

Phrase is defined as “a group of words which is part rather than the whole of a sentence” [80] in a dictionary. For example, “take away” and “pull out” are phrases. Phrases has been used as features intensively in information retrieval; however, at least in its early works, it was not effective [40]. The phrase is not a good feature because it does not fit four good feature criteria; (i) small number of indexing terms, (ii) flat distribution of values for an indexing terms, (iii) lack of redundancy among terms, and (iv) low noise in indexing terms values [41]. However, there is no detail quantitative analysis why they are not successfully improve categorization effectiveness compared to single word features. Other works [55] gave explanation about the failure of phrase for document representation in text classification; however their analyses were short and descriptive. Regarding criteria (i), the automatic syntactical phrase identification method was used by author [40] and he found 32,521 phrases, while words just only 22,791. In [19], it was reported that syntactical phrase improved precision $10 %$ , recall $5 %$ , helped in generating high-precision classification in a big data collection [3].

3.1.3. Pattern-based feature

A new approach for document representation is using termset (aka pattern). Pattern Taxonomy Model (PTM) [46,81,95] used intra-document based frequent closed sequential pattern with paragraph as the transactional unit. A pattern is closed if none of its immediate supersets have exactly the same support count. PTM defines closed patterns as meaningful patterns as most of the sub-sequence patterns of closed patterns have the same frequency, which means they always occur together in a document. Smaller patterns in the taxonomy, are usually more general because they have a high occurrence frequency in both positive and negative documents; but larger patterns are usually more specific since they have a small chance of being found in both positive and negative documents [46]. PTM pruned non-closed patterns from document representation with an attempt to reduce the size of the feature set by removing noisy patterns.

PTM is applied as effective information filtering system [46,96]. Different to most other document representation, instead of as input for machine learning classifier, PTM representation is then used to produce a set of weighted term. The set of weighted term is used as class representation, it makes scoring process (for new documents) more efficient.

3.2. Feature weighting methods

A text document is represented by a feature vector which includes selected feature and it’s associated weight. A feature weight indicates the degree of information represented by the feature occurrences in a document and reflects the relative importance of the feature.

A number of term weighting methods derived from information retrieval including term frequency ( $tf$ ) and inverse document frequency ( $idf$ ) [67], and information-theoretic term selection functions including the DIA association factor [16], Chi-Square [84,87], Information Gain [84,87], Mutual Information [42], NGL coefficient [57], have been developed for feature dimensionally reduction. In [87], Information Gain and Chi-square methods are found most effective for K-NN and linear regression learning methods. Term selection based on Document Frequency in the training set as a whole is simple but has similar performance to Information Gain and Chi-square methods.

Terms that repeat multiple times in a document are considered important. Terms that appear in many documents are considered common and are not indicative of document content. Based on this idea, the number of documents in a document collection is defined as N. The frequency of a term $t_{i}$ in a document can be used for document-specific weighting and denoted as $tf (t_{i})$ . Term Frequency is only a measure of a term’s significance within a document. It assigns high weights to terms that appear more frequently within a document. Document Frequency $df (t_{i})$ is the number of documents that contain the term $t_{i}$ . Inverse Document Frequency ( $idf$ ) is used to measure the specificity of terms in a set of documents. The intuition was that a query term that occurs in many documents is not a good discriminator, and should be given less weight than one which occurs in few documents.

Information gain (IG) [70,87] measures the number of bits of information obtained for category prediction by knowing the presence or absence of a term in a document. The information gain of term t and category $c_{i}$ is defined to be: Information Gain (IG) gauges the expected reduction in entropy of category prediction. It can also be applied for measuring term correlations [56]. Chi-Square ( $X^{2}$ ) measures the lack of independence between a term t and a category $c_{i}$ and can be compared to the chi-square distribution with one degree of freedom to judge extremeness [70,87]. It can be applied for measuring the lack of independence between a term and the specific topic [56]. Mutual Information (MI) estimates the reduction in uncertainty of the two variables. It can be used to identify term correlations and the association between a term and a specific topic.

Latent Semantic Indexing (LSI) method was developed by [11] based on Latent semantic Analysis (LSA) [14]. LSI uses a linear algebra’s matrix factorization, singular value decomposition (SVD), to transform an origin high dimensional data to a new lower, orthogonal dimension approximation, by applying truncated SVD to word-document matrix. This new space is a more compact document representation. Words and documents that are closely associated will be placed near one another in new “semantic space”. LSI compresses the original document vectors to a smaller semantic space by taking advantage of the patterns of co-occurrence as an implicit higher-order structure. This “semantic structure” or “theme” (associations of terms with documents) can be viewed as conceptual user representations on document [69]. These variety representations can be higher for document collection produced by several different people. The new set of vectors can be viewed as pseudo document vectors. However the created features are not intuitively interpretable. In recent paper [22], the authors proposed a new method for sensor storage that combined semantic web concepts and data aggregation method along with aligning sensors in hierarchical form to respond to semantic web-based queries more effectively.

There are many current work on weighting methods. Two of them are relevance frequency (RF) [35] (a supervised inter-document method which exploits the distribution of relevant documents in the collection) and distributional feature [82] (an intra document method).

4. Classification algorithms

4.1. Types of classifiers

There are different types of text classifier. In [18] text classifiers are categorized into two types: kernel-based classifiers and instance-based classifiers. Typical kernel-based classifier learning approaches include the Support Vector Machines (SVMs) [28] and regression models [70]. Typical instance-based classification approaches include the K-Nearest Neighbour (K-NN) [10] and its variants, which do not rely upon the statistical distribution of training samples. Other research works, such as [86,94], have a different way of categorizing the classifier learning techniques: Linear classifiers, including linear SVM, regularized logistic regression, ridge regression (i.e., regularized least squares fit), Naïve Bayes (NB) methods and Boosted linear classifiers; and Non-linear classifiers, including KNN, SVM with non-linear kernels, Boosting, decision trees, and Neural Networks (NN) with hidden layers. Among those classifiers, the most notable ones are SVM, KNN, Logistic regression, boosting.

4.2. Conventional classifiers

Text has the properties of high dimensional (more than 1000), few irrelevant features (dense concept vector), sparse document vectors (most feature values in document vector are zero), and most text categorization problems are linearly separable [28]. SVM is an outstanding text categorization method because of its capability to overcome text properties [28]. Even if all available features were used (no dimensional reduction), SVM had good effectiveness, and difficult to be beaten [63]. The investigation in [35] showed that SVM’s effectiveness rapidly increased to level 0.8–0.9 in F measure then steady relatively. It was different with kNN which had a peak at level 0.7–0.8 in F measure and went down as the number of feature increase.

SVM is based on the structure risk minimization principle from the statistical learning theory [78]. This method has been introduced in Text Classification by Joachims [28]. It is an inductive learning technique. SVM can be used to find a hyperplane h that has the maximum margin as decision boundary in the linear space. A classification task usually involves with training and testing data that consist of some data instances. Each instance in the training set contains one class labels and several features (attributes).

For the linear classification, SVMs learn linear decision rules $h (\vec{x}) = sign \vec{w} . \vec{x} + b$ described by a weight vector $\vec{w}$ and a threshold b. Given a sample of n training examples, $T_{n} = ((\vec{x_{1}}, y_{1}), \dots (\vec{x_{n}}, y_{n}), x_{i} \in ℜ^{N}$ , and $y_{i} \in - 1, + 1$ be the classification for $T_{n}$ (+ for being a positive example and $- 1$ for being a negative example of the given class), the SVM finds the hyperplane with maximum Euclidean distance to the closest training examples to linearly separate $T_{n}$ . This distance is called the margin δ. For non-separable training sets, the amount of training error is measured using slack variable $ξ_{i}$ . The Fig. 2 describes a binary support vector classifier. In the Fig. 2, “+” and “−” represent the positive and negative training examples, respectively. The hyperplane h is represented by a thicker line and it separates positive and negative training examples with maximum margin δ. The examples closest to the hyperplane h are support vectors that are represented by small boxes.

Fig. 2.

A binary support vector classifier in two dimensions.

The SVM require the solution of the following optimization problem [78]: $\begin{array}{l} minimize: \\ (1) & V (\vec{w,} b, \vec{ξ}) = \frac{1}{2} \vec{w} . \vec{w} + C \sum_{i = 1}^{n} ξ_{i} \\ (2) & subject to: \forall_{i = 1}^{n} : y_{i} [\vec{w} . \vec{x} + b] ⩾ 1 - ξ_{i} \\ (3) & \forall_{i = 1}^{n} : ξ_{i} > 0 \end{array}$ $C > 0$ is the penalty parameter of the error term. The margin of the resulting hyperplane is $δ = 1 / ‖ \vec{w} ‖$ . The decision surface is determined only by the data points which have exactly the distance $\frac{1}{‖ \vec{w} ‖}$ from the decision plane. Those points are called the support vectors, which are the only effective elements in the training set.

However, SVM is not suitable for classification of large datasets or text corpora since the training complexity of SVMs is highly dependent on the input size. A new multikernel SVM was developed in [65] to deal with highly dimensional data. Their experimental results demonstated that the new multikernel SVM classifier achived better accuracy compared with the classic SVM, while the training is significantly faster than several other SVM classifiers.

Logistic regression is another commonly-used statistical approach for obtaining a linear classifier. One regression model is called the Linear Least Squares Fit, a mapping approach developed in [85]. A multivariate regression model is automatically learned from a training set of documents and their categories. The training data are represented in the form of input/output vector pairs where the input vector is a document in the conventional vector space model (consisting of words with weights), and the output vector consists of categories (with binary weights) of the corresponding document. By solving the linear least-squares fit on the training pairs of vectors, one can obtain a matrix of word-category regression coefficients. The matrix defines a mapping from an arbitrary document to a vector of weighted categories. By sorting these category weights, a ranked list of categories is obtained for the input document.

KNN (K-Nearest Neighbour) classification is a well-known statistical approach that has been intensively studied in pattern reconciliation for over four decades [10]. KNN has been applied to text categorization since the early stages of the research [85]. KNN approach is most common due to its simplicity and a prediction accuracy. Given an arbitrary input document, the system ranks its nearest neighbours among the training documents and uses the categories of the k top-ranking neighbours to predict the categories of the input document. The similarity score of each neighbour document to the new document being classified is used as the weight of each of its categories; the sum of the category weights over the k nearest neighbours are used for category ranking. The model complexity is controlled by the choice of k. Formally, the degree of freedom of a kNN classification model is defined as $d . f . = n / k$ where n is the number of documents in the training set. When $k = 1$ , KNN has the maximum model complexity and tends to overfit to the training set; when k increases, the model complexity reduces accordingly [86].

The Rocchio is a classic vector-space-model method for document routing or filtering in information retrieval. Applying it to text categorization, the basic idea is to construct a prototype vector for each category using a training set of documents. Given a category, the vectors of the documents belonging to this category are given a positive weight, and the vectors of the remaining documents are given a negative weight. By summing up these positively and negatively weighted vectors, the prototype vector of this category is obtained. This method is easy to implement, efficient in computation, and has been used as a baseline in several evaluations [9,44]. A potential weakness of this method is the assumption of one centroid per category, and consequently, the Rocchio does not perform well when the documents belonging to a category naturally form separate clusters [84].

Other classifiers such as Bayesian probabilistic classifiers can be found in [42], decision trees in [16,42], inductive rule learning algorithms in [9], neural networks in [57] and on-line learning approaches in [9,44].

In general, SVM has very good performance, however in many situations other methods have comparable performance or even better. SVM and KNN significantly outperform NN and NB for low frequency category (less than 10 documents), but all the methods perform comparably when the categories are sufficiently common (over 300 documents) [84]. In high skew class distribution (imbalance data), SVM performance can drop sharply [15]. For distributional features (i.e. features that represent word compactness in a document), KNN is more suitable than SVM [82]. For spam email filtering that need high efficiency, Naïve Bayes is a popular method [68].

4.3. Hierarchical text classification

Hierarchical text classification is an important text mining and natural language processing task in many real-world applications and it aims at classifying text documents into classes that are organized into a hierarchy [52]. The hierarchical information can be added into feature representation for hierarchical text classification [64,66]. The basic idea is that the notion of class attributes will allow generalization to take place across (similar) categories and not just across training examples belonging to the same category [64].

In recent years, the authors of [25,26] have developed a simpler recursive regularization of weight vectors of linear classifiers in the hierarchical model for the large-scale hierarchical text classification. A new weakly-supervised neural method for hierarchical text classification was proposed in [52]. This new method is different from a deep neural model that requires a large amount of training data. The weakly-supervised model requires only easy-to-provide weak supervision signals such as a few class-related documents or keywords.

4.4. Deep learning for classificatoin

Deep learning [37] is a machine learning technique based on representation learning where the system automatically learns and discovers the features needed for classification from the processing of multiple layers of input data. Deep learning has become a mainstream machine learning technique with capacity in various nonlinear modelling tasks such as the classification and feature extraction process from complex datasets. In recent years, many deep learning techniques including the convolutional neural networks (CNN) [38,39], and recurrent neural networks (RNN) [4], have been explored for text classification.

Deep learning has been proven to be effective to perform end-to-end learning of hierarchical feature representations and deep neural networks have demonstrated superior performance for flat text classification [52]. RNN has been proposed for document classification in [34,73] and a hierarchical attention network model that emphasizes important sentences and words was presented in [88]. Deep convolutional neural networks has been applied for short texts classificatoin and sentence level classification in [13,31]. The character-level convolutional networks (ConvNets) model was designed for text classification in [93]. A graph-CNN based deep learning model is proposed in [61] to convert text to graph-of-words, on which the graph convolution operations are applied for feature extraction.

5. Applications in the real world

The most popular text classification algorithms, such as Support Vector Machine (SVM), K Nearest Neighbours (K-NN), logistic regression, Rocchio method, have been used for information retrieval and filtering tasks [17,70,87]. The combination of the lexico-syntactic and statistical learning approaches was proposed in [36] to extract and build more reliable domain ontology for an intelligent information retrieval system. In [98], to represent user profile by relevant topic ontology, this paper presents a new method capable of measuring the user profile more objectively and hence has great potential to enhance the information retrieval and filtering processes. The authors of [74] used ontologies techniques to build user profile for knowledge-based and personalized Web information gathering systems.

Text classification techniques are widely used in concept-based Web information gathering systems. The paper [21] described how text classification techniques are used for concept-based Web information gathering. Web users submit a topic associated with some specified concepts. The gathering agents then search for the Web documents that are referred to by the concepts. A list of tasks in Web information gathering to which text classification techniques may contribute is outlined in [70], including automatic indexing for Boolean information retrieval systems, document organization (particularly in personal organization or structuring of a corporate document base), text filtering, word sense disambiguation, and hierarchical categorization of web pages. Text classification techniques have been utilized by [51] to classify Web documents into the best matching interest categories, based on their referring semantic concepts.

Most research has focused on classification into topic categories, other types of classification such as sentiment classification (positive, neutral or negative), have received a lot of attentions in recent years. Sentiment classification classifies the expressed opinion in a document, or a sentence into positive, negative or neutral [60]. Many supervised and unsupervised text classification methods have been used in sentiment classification. In [100], the authors used unsupervised method to analyse tweets data for social event prediction. The authors of [75] analyse the tweets data for depression detection.

Recurrent neural network also benefits sentiment classification because it is capable of capturing the sequential information. In [45], the authors investigated tree-structured long-short term memory (LSTM) networks on text or sentiment classification and discussed when tree structures are necessary. There are also some hierarchical models proposed to deal with document-level sentiment classification [73], which generate different levels (e.g., phrase, sentence or document) of semantic representations within a document.

There are many practical text classification applications in diverse domains. For bioinformatics [32], text mining applications include subcellular localization prediction [5,71] and protein clustering [23]. For the applications in finance field, the various applications of text mining to finance are presented in a survey paper [33]. In [33], text mining applications are categorized broadly into FOREX rate prediction, stock market prediction, customer relationship management (CRM) and cyber security in finance domain. In survey [24], text mining techniques were applied for decision support systems in dental clinics.

6. Conclusions

The purpose of this survey has been to describe and analyse the state-of-the-art of text classification, and to convey to the reader a sense of our excitement about the intellectual richness and breadth of the area. In the recent years, many research groups have invested much effort on automated text analysis and classification, and have made many great achievements. The rich literature growing around these topics. However, challenging problems still exist in these areas. In particular, the research issues on how to make breakthrough on the current text classification for very large-scale text categorization problems, how to build more effective and efficient feature selection model have attracted lots of the research attentions.

We very much hope we have provided some helpful information to the readers who are encouraged to take up the many challenges that remain in the area.

References

E.L.

Allwein,

R.E.

Schapire,

Singer and

Kaelbling, Reducing multiclass to binary: A unifying approach for margin classifiers, Journal of Machine Learning Research 1 (2000), 113–141.

G.O.

Barnett,

J.J.

Cimino,

J.A.

Hupp and

E.P.

Hoffer, DXplain: An evolving diagnostic decision-support system, Jama 258(1) (1987), 67–74. doi:10.1001/jama.1987.03400010071030.

Bekkerman and

Gavish, High-precision phrase-based document classification on a modern scale, in: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’11, ACM, New York, NY, USA, 2011, pp. 231–239.

Bengio,

Ducharme,

Vincent and

Jauvin, A neural probabilistic language model, Journal of Machine Learning Research 3 (2003), 1137–1155.

Brady and

Shatkay, EpiLoc: A (working) text-based system for predicting protein subcellular location, in: Biocomputing 2008, World Scientific, 2008, pp. 604–615.

P.B.

Cerrito and

J.C.

Cerrito, Data and Text Mining the Electronic Medical Record to Improve Care and to Lower Costs, 2006.

Chen,

Huang,

Tian and

Qu, Feature selection for text classification with Naïve Bayes, Expert Systems with Applications 36(3, Part 1) (2009), 5432–5435. doi:10.1016/j.eswa.2008.06.054.

Chen,

Yan,

Zhang,

Chen and

Yang, Document transformation for multi-label feature selection in text categorization, in: Proceedings of the 2007 Seventh IEEE International Conference on Data Mining, IEEE Computer Society, Washington, DC, USA, 2007, pp. 451–456. ISBN 0-7695-3018-4.

W.W.

Cohen and

Singer, Context-sensitive learning methods for text categorization, in: ACM Transactions on Information Systems, ACM Press, 1996, pp. 307–315.

10.

B.V.

Dasarathy, Nearest neighbor (NN) norms: NN pattern classification techniques, IEEE Computer Society Tutorial (1991).

11.

Deerwester,

S.T.

Dumais,

G.W.

Furnas,

T.K.

Landauer and

Harshman, Indexing by latent semantic analysis, Journal of the American Society for Information Science 41(6) (1990), 391–407. doi:10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9.

12.

Díaz and

Gervás, Personalisation in news delivery systems: Item summarization and multi-tier item selection using relevance feedback, Web Intelligence and Agent Systems: An International Journal 3(3) (2005), 135–154.

13.

Dos Santos and

Gatti, Deep convolutional neural networks for sentiment analysis of short texts, in: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, 2014, pp. 69–78.

14.

S.T.

Dumais,

G.W.

Furnas,

T.K.

Landauer,

Deerwester and

Harshman, Using latent semantic analysis to improve access to textual information, in: SIGCHI Conference on Human Factors in Computing Systems, ACM, 1988, pp. 281–285.

15.

Forman,

Guyon and

Elisseeff, An extensive empirical study of feature selection metrics for text classification, Journal of Machine Learning Research 3 (2003), 1289–1305.

16.

Fuhr and

Buckley, A probabilistic learning approach for document indexing, ACM Transaction on Information Systems 9(3) (1991), 223–248. doi:10.1145/125187.125189.

17.

Fuhr,

Hartmann,

Lustig,

Schwantner,

Tzeras,

T.H.

Darmstadt,

Informatik and

Knorz, AIR/X – a rule-based multistage indexing system for large subject fields, in: Proceedings of RIAO’91, 1991, pp. 606–623.

18.

G.P.C.

Fung,

J.X.

Yu,

Lu and

P.S.

Yu, Text classification without negative examples revisit, IEEE transactions on Knowledge and Data Engineering 18(1) (2006), 6–20. doi:10.1109/TKDE.2006.16.

19.

Furnkranz,

Mitchell,

Riloff et al., A case study in using linguistic phrases for text categorization on the WWW, in: Working Notes of the AAAI/ICML, Workshop on Learning for Text Categorization, 1998, pp. 5–12.

20.

Gao,

Xu and

Li, Pattern-based topics for document modelling in information filtering, IEEE Transactions on Knowledge and Data Engineering 27(6) (2014), 1629–1642. doi:10.1109/TKDE.2014.2384497.

21.

Gauch,

Chaffee and

Pretschner, Ontology-based personalized search and browsing, Web Intelligence and Agent Systems 1 (2003), 1–3.

22.

Gheisari,

A.A.

Movassagh,

Qin,

Yong,

Tao,

Zhang and

Shen, Nsssd: A new semantic hierarchical storage for sensor data, in: 2016 IEEE 20th International Conference on Computer Supported Cooperative Work in Design (CSCWD), IEEE, 2016, pp. 174–179. doi:10.1109/CSCWD.2016.7565984.

23.

Glenisson,

Coessens,

Van Vooren,

Mathys,

Moreau and

De Moor, TXTGate: Profiling gene groups with text-based information, Genome Biology 5(6) (2004), R43. doi:10.1186/gb-2004-5-6-r43.

24.

W.P.

Goh,

Tao,

Zhang and

Yong, Decision support systems for adoption in dental clinics: A survey, Knowledge-Based Systems 104 (2016), 195–206. doi:10.1016/j.knosys.2016.04.022.

25.

Gopal and

Yang, Recursive regularization for large-scale classification with hierarchical and graphical dependencies, in: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2013, pp. 257–265. doi:10.1145/2487575.2487644.

26.

Gopal and

Yang, Hierarchical Bayesian inference and recursive regularization for large-scale classification, ACM Transactions on Knowledge Discovery from Data (TKDD) 9(3) (2015), 18. doi:10.1145/2629585.

27.

Henriksson,

Kvist,

Dalianis and

Duneld, Identifying adverse drug event information in clinical notes with distributional semantic representations of context, Journal of Biomedical Informatics 57 (2015), 333–349. doi:10.1016/j.jbi.2015.08.013.

28.

Joachims, Text categorization with support vector machines: Learning with many relevant features, in: ECML, 1998, pp. 137–142.

29.

Joachims, Transductive inference for text classification using support vector machines, in: The 16th International Conference on Machine Learning, Bled Slovenia, 1999, pp. 200–209.

30.

Kehagias,

Petridis,

V.G.

Kaburlasos and

Fragkou, A comparison of word- and sense-based text categorization using several classification algorithms, Journal of Intelligent Information Systems 21 (2001), 227–247. doi:10.1023/A:1025554732352.

31.

Kim, Convolutional neural networks for sentence classification, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Doha, Qatar, 2014, pp. 1746–1751. doi:10.3115/v1/D14-1181.

32.

Koussounadis,

O.C.

Redfern and

D.T.

Jones, Improving classification in protein structure databases using text mining, BMC Bioinformatics 10(1) (2009), 129. doi:10.1186/1471-2105-10-129.

33.

B.S.

Kumar and

Ravi, A survey of the applications of text mining in financial domain, Knowledge-Based Systems 114 (2016), 128–147. doi:10.1016/j.knosys.2016.10.003.

34.

Lai,

Xu,

Liu and

Zhao, Recurrent convolutional neural networks for text classification, in: Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.

35.

Lan,

C.L.

Tan,

Su and

Lu, Supervised and traditional term weighting methods for automatic text categorization, IEEE Transations on Pattern Analysis and Machine Intelligence 31 (2009), 721–735. doi:10.1109/TPAMI.2008.110.

36.

R.Y.

Lau,

J.X.

Hao,

Tang and

Zhou, Towards context-sensitive domain ontology extraction, in: 2007 40th Annual Hawaii International Conference on System Sciences (HICSS’07), IEEE, 2007, pp. 60–60. doi:10.1109/HICSS.2007.570.

37.

LeCun,

Bengio and

Hinton, Deep learning, Nature 521(7553) (2015), 436–444. doi:10.1038/nature14539.

38.

LeCun,

Boser,

J.S.

Denker,

Henderson,

R.E.

Howard,

Hubbard and

L.D.

Jackel, Backpropagation applied to handwritten zip code recognition, Neural computation 1(4) (1989), 541–551. doi:10.1162/neco.1989.1.4.541.

39.

LeCun,

Bottou,

Bengio,

Haffner et al., Gradient-based learning applied to document recognition, Proceedings of the IEEE 86(11) (1998), 2278–2324. doi:10.1109/5.726791.

40.

D.D.

Lewis, An evaluation of phrasal and clustered representations on a text categorization task, in: SIGIR, 1992, pp. 37–50.

41.

D.D.

Lewis, Representation and learning in information retrieval, PhD thesis, University of Massachusetts, Amherst, United States, 1992.

42.

D.D.

Lewis and

Ringuette, A comparison of two learning algorithms for text categorization, in: Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, Las Vegas, US, 1994, pp. 81–93.

43.

D.D.

Lewis,

R.E.

Schapire,

J.P.

Callan and

Papka, Training algorithms for linear text classifiers, in: SIGIR’96: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, USA, 1996, pp. 298–306. ISBN 0-89791-792-8. doi:10.1145/243199.243277.

44.

D.D.

Lewis,

R.E.

Schapire,

J.P.

Callan and

45.

Li,

Luong,

Jurafsky and

Hovy, When are tree structures necessary for deep learning of representations? in: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp. 2304–2314. doi:10.18653/v1/D15-1278.

46.

Li,

Algarni and

Zhong, Mining positive and negative patterns for relevance feature discovery, in: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’10, ACM, New York, NY, USA, 2010, pp. 753–762. doi:10.1145/1835804.1835900.

47.

Li,

Zhang,

Xu,

Yao,

R.Y.K.

Lau and

Wu, Enhancing binary classification by modeling uncertain boundary in three-way decisions, IEEE Transactions on Knowledge and Data Engineering 29(7) (2017), 1438–1451. doi:10.1109/TKDE.2017.2681671.

48.

Li,

Zhou,

Bruza,

Xu and

R.Y.

Lau, A two-stage text mining model for information filtering, in: Proceedings of the 17th ACM Conference on Information and Knowledge Management, ACM, 2008, pp. 1023–1032.

49.

Liu,

Dai,

Li,

W.S.

Lee and

P.S.

Yu, Building text classifiers using positive and unlabeled examples, in: ICDM03, 2003, pp. 179–186.

50.

Liu,

W.-C.

Chang,

Wu and

Yang, Deep learning for extreme multi-label text classification, in: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, 2017, pp. 115–124.

51.

Ma,

Pant and

O.R.L.

Sheng, Interest-based personalized search, ACM Trans. Inf. Syst. 25 (2007).

52.

Meng,

Shen,

Zhang and

Han, Weakly-supervised hierarchical text classification, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp. 6826–6833.

53.

G.A.

Miller, WordNet: A lexical database for English, Communications of the ACM 38 (1995), 39–41. doi:10.1145/219717.219748.

54.

Moschitti, Syntactic and semantic kernels for short text pair categorization, in: Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, EACL’09, Association for Computational Linguistics, Stroudsburg, PA, USA, 2009, pp. 576–584.

55.

Moschitti and

Basili, Complex linguistic features for text classification: A comprehensive study, in: Proceedings of the 26th European Conference on Information Retrieval (ECIR), Springer Verlag, 2004, pp. 181–196.

56.

Nanas,

V.S.

Uren and

Roeck, A comparative evaluation of term weighting methods for information filtering, in: DEXA Workshops, 2004, pp. 13–17.

57.

H.T.

Ng,

W.B.

Goh and

K.L.

Low, Feature selection, perceptron learning, and a usability case study for text categorization, in: SIGIR, Vol. 97, 1997, pp. 67–73. doi:10.1145/258525.258537.

58.

Nguyen and

Caruana, Classification with partial labels, in: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’08, ACM, New York, NY, USA, 2008, pp. 551–559. doi:10.1145/1401890.1401958.

59.

Özgür and

Güngör, Text classification with the support of pruned dependency patterns, Pattern Recognition Letters 31(12) (2010), 1598–1607. doi:10.1016/j.patrec.2010.05.005.

60.

Pang and

Lee, Thumbs up? Sentiment classification using machine learning techniques, in: Proceedings of EMNLP, 2002, pp. 79–86. doi:10.3115/1118693.1118704.

61.

Peng,

Li,

He,

Liu,

Bao,

Wang,

Song and

Yang, Large-scale hierarchical text classification with recursively regularized deep graph-cnn, in: Proceedings of the 2018 World Wide Web Conference, International World Wide Web Conferences Steering Committee, 2018, pp. 1063–1072. doi:10.1145/3178876.3186005.

62.

Peng and

Choi, Document classifications based on word semantic hierarchies, in: Proceedings of the International Conference on Artificial Intelligence and Applications (AIA’05), 2005, pp. 362–367.

63.

L.-Q.

Qiu,

R.-Y.

Zhao,

Zhou and

S.-W.

Yi, An Extensive Empirical Study of Feature Selection for Text Categorization, 2008, pp. 312–315.

64.

Qiu,

Huang,

Liu and

Zhou, Hierarchical text classification with latent concepts, in: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers – Volume 2, Association for Computational Linguistics, 2011, pp. 598–602.

65.

Romero,

Iglesias and

Borrajo, A linear-RBF multikernel SVM to classify big text corpora, BioMed Research International 2015 (2015).

66.

Rousu,

Saunders,

Szedmak and

Shawe-Taylor, Learning hierarchical multi-category text classification models, in: Proceedings of the 22nd International Conference on Machine Learning, ACM, 2005, pp. 744–751.

67.

Salton and

Buckley, Term-weighting approaches in automatic text retrieval, Inf. Process. Manage. 24(5) (1988), 513–523. doi:10.1016/0306-4573(88)90021-0.

68.

R.E.

Schapire and

Singer, BoosTexter: A boosting-based system for text categorization, in: Machine Learning, 2000, pp. 135–168.

69.

Schütze,

D.A.

Hull and

J.O.

Pedersen, A comparison of classifiers and document representations for the routing problem, in: SIGIR, 1995, pp. 229–237.

70.

Sebastiani, Machine learning in automated text categorization, ACM Computing Surveys 34(1) (2002), 1–47. doi:10.1145/505282.505283.

71.

Shatkay,

Höglund,

Brady,

Blum,

Dönnes and

Kohlbacher, SherLoc: High-accuracy prediction of protein subcellular localization by integrating text and protein sequence data, Bioinformatics 23(11) (2007), 1410–1417. doi:10.1093/bioinformatics/btm115.

72.

Shen,

J.-T.

Sun,

Yang,

Zhao and

Chen, Text classification improved through automatically extracted sequences, in: ICDE’06, 2006, pp. 1–1.

73.

Tang,

Qin and

Liu, Document modeling with gated recurrent neural network for sentiment classification, in: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp. 1422–1432. doi:10.18653/v1/D15-1167.

74.

Tao,

Li and

Zhong, A knowledge-based model using ontologies for personalized web information gathering, Web Intelligence and Agent Systems: An International Journal 8(3) (2010), 235–254. doi:10.3233/WIA-2010-0189.

75.

Tao,

Zhou,

Zhang and

Yong, Sentiment analysis for depression detection on social networks, in: International Conference on Advanced Data Mining and Applications, Springer, 2016, pp. 807–810. doi:10.1007/978-3-319-49586-6_59.

76.

Tsoumakas and

Katakis, Multi-label classification: An overview, Int J Data Warehousing and Mining 2007 (2007), 1–13. doi:10.4018/jdwm.2007070101.

77.

Tsoumakas,

Katakis and

Vlahavas, Mining Multi-label Data, Transformation (2010), 1–19.

78.

Vapnik, The Nature of Statistical Learning Theory, Springer Science & Business Media, 2013.

79.

Wakaki,

Itakura,

Tamura,

Motoda and

Washio, A study on rough set-aided feature selection for automatic web-page classification, Web Intelligence and Agent Systems: An International Journal 4(4) (2006), 431–441.

80.

Weedford,

Walter and

Shenton, Cambridge Learner’s Dictionary, 3rd edn, Cambridge University, 2007.

81.

S.-T.

Wu,

Li and

Xu, Deploying approaches for pattern refinement in text mining, in: Proceedings of the 6th IEEE International Conference on Data Mining (ICDM 2006), 2006, pp. 1157–1161. doi:10.1109/ICDM.2006.50.

82.

X.-B.

Xue and

Z.-H.

Zhou, Distributional features for text categorization, IEEE Transactions on Knowledge and Data Engineering 21(3) (2009), 428–442.

83.

Yang,

Jin,

A.K.

Jain,

Zhou and

Tong, Unsupervised transfer classification: Application to text categorization, in: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2010, pp. 1159–1168. doi:10.1145/1835804.1835950.

84.

Yang, An evaluation of statistical approaches to text categorization, Information Retrieval 1 (1999), 69–90. doi:10.1023/A:1009982220290.

85.

Yang and

C.G.

Chute, An example-based mapping method for text categorization and retrieval, ACM TOIS 12 (1994), 281–295. doi:10.1145/183422.183424.

86.

Yang and

Joachims, Text categorization, Scholarpedia 3(5) (2008), 4242. doi:10.4249/scholarpedia.4242.

87.

Yang and

J.O.

Pedersen, A comparative study on feature selection in text categorization, in: Icml, Vol. 97, 1997, p. 35.

88.

Yang,

Dyer,

He,

Smola and

Hovy, Hierarchical attention networks for document classification, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1480–1489.

89.

Ye,

T.-S.

Chua and

J.R.

Kei, Clustering web pages about persons and organizations, Web Intelligence and Agent Systems: An International Journal 3(4) (2005), 203–216.

90.

Zhang,

Li,

Sun and

Nadee, Rough set based approach to text classification, in: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) – Volume 03, IEEE Computer Society, 2013, pp. 245–252.

91.

Zhang,

Li,

Xu,

Tjondronegoro and

Sun, Centroid training to achieve effective text classification, in: 2014 International Conference on Data Science and Advanced Analytics (DSAA), IEEE, 2014, pp. 406–412. doi:10.1109/DSAA.2014.7058104.

92.

M.-L.

Zhang and

Z.-H.

Zhou, A review on multi-label learning algorithms, IEEE Transactions on Knowledge and Data Engineering 26(8) (2013), 1819–1837. doi:10.1109/TKDE.2013.39.

93.

Zhang,

Zhao and

LeCun, Character-level convolutional networks for text classification, in: Advances in Neural Information Processing Systems, 2015, pp. 649–657.

94.

Zhang and

Callan, Maximum likelihood estimation for filtering thresholds, in: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, 2001, pp. 294–302.

95.

Zhong,

Li and

S.-T.

Wu, Effective pattern discovery for text mining, IEEE Transactions on Knowledge and Data Engineering (2010), 1–36.

96.

Zhou,

Li,

Bruza,

Xu and

R.Y.K.

Lau, Pattern mining for a two-stage information filtering system, in: Proceedings of the 15th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining – Volume Part I, PAKDD’11, Springer-Verlag, Berlin, Heidelberg, 2011, pp. 363–374. ISBN 978-3-642-20840-9.

97.

Zhou,

Li,

P.D.

Bruza,

Xu and

R.Y.

Lau, Rough sets based reasoning and pattern mining for a two-stage information filtering system, in: Proceedings of the 19th ACM International Conference on Information and Knowledge Management, ACM, 2010, pp. 1429–1432.

98.

Zhou,

Li,

Xu and

Lau, Relevence assessment of topic ontology, in: Proceedings of the 2006 Conference on Advances in Intelligent IT: Active Media Technology 2006, IOS Press, 2006, pp. 44–51.

99.

Zhou,

Tao,

M.M.

Rahman and

Zhang, Coupling topic modelling in opinion mining for social media analysis, in: Proceedings of the International Conference on Web Intelligence, ACM, 2017, pp. 533–540. doi:10.1145/3106426.3106459.

100.

Zhou,

Tao,

Yong and

Yang, Sentiment analysis on tweets for social events, in: Proceedings of the 2013 IEEE 17th International Conference on Computer Supported Cooperative Work in Design (CSCWD), IEEE, 2013, pp. 557–562. doi:10.1109/CSCWD.2013.6581022.