Using data-driven feature enrichment of text representation and ensemble technique for sentence-level polarity classification

Abstract

As an important issue in sentiment analysis, sentence-level polarity classification plays a critical role in many opinion-mining applications such as opinion question answering, opinion retrieval and opinion summarization. Employing a supervised learning paradigm to train a classifier from sentences often faces the data sparseness problem owing to the short-length limit introduced to texts. In this article, regarding this problem, we exploit two different feature sets learned from external data sets as additional features to enrich data representation: one is a latent topic feature set obtained using a topic model, and the other is a related word feature set derived using word embeddings. Furthermore, we propose an ensemble approach by using these additional features to guide the design of different members of the ensemble. Experimental results on the public movie review dataset demonstrate that the enriched representations are effective for improving the performance of polarity classification, and the proposed ensemble approach can further improve the overall performance.

Keywords

Sentiment analysis sentence polarity classification topic model word embeddings

1. Introduction

Sentiment analysis (SA) is a field of study that analyses people’s opinions, sentiments, attitudes and emotions expressed in written language [1]. With the explosive growth of user-generated opinionated contents such as blogs, reviews, forum discussions and tweets, SA is becoming increasingly important and has grown to be one of the most active research areas in natural language processing (NLP). Among various SA tasks, an important sub-task is polarity classification, which determines if an opinionated piece of text expresses positive or negative sentiment.

Generally, polarity classification has been performed at different granularity levels: word level [2], phrase level [3], sentence level [4, 5] and document level [6 –8]. In this article, we focus on polarity classification at the sentence level and treat reviews in a sentence-level dataset as short texts as they generally consist of a few phrases or sentences.

Unlike traditional normal texts, short texts are usually noisier, sparser and much shorter. Because of the short length, short texts do not provide enough word co-occurrence or shared context for a good similarity measure, which presents great challenges in clustering and classification. Owing to the data sparseness, normal machine-learning methods usually encounter significant performance degradation and fail to achieve the desired results when they are directly applied to short text tasks [9, 10].

In traditional short-text classification and clustering tasks, to deal with the data sparseness problem, most existing work try to enrich the representation of a short text by exploiting additional features derived from various sources, for example, the short-text collection itself [11, 12], extra information obtained from search engines [13], a large external collection of unlabelled documents [9, 14] or some external knowledge base such as WordNet, Open Directory Project and Wikipedia [12, 15, 16].

Inspired by the success of representation enrichment approach in short-text classification and clustering tasks, we also adopt this approach in this work. However, different from the previous works, we propose to explore both latent topics and related words as additional features to enrich the representation of reviews.

Consider the following sentences, in which sentence S1 is labelled with positive polarity, and sentence S2 is used as the test data.

S1. The iPod is great.

S2. I bought iPhone. What a product!

From S1, we can see that ‘iPod’ appears in a review text of positive polarity. Since ‘iPhone’ and ‘iPod’ are likely to be grouped under the same topic named ‘topic_Apple’, if we can identify the common topic shared by S1 and S2, then we can enrich S1 and S2 with topic name as follows:

S1′. The iPod is great. topic_Apple

S2′. I bought iPhone. What a product! topic_Apple

As a result, the shared topic makes S2′ and S1′ more related in a semantic way, and S2′ is more likely to have a positive polarity than S2.

Next, consider the following sentences, in which sentence S3 is labelled with positive polarity, and sentence S4 is used as the test data.

S3. a pleasant movie.

S4. an enjoyable film.

In the sentences S3 and S4, words ‘pleasant’ and ‘enjoyable’ would be treated with no relation in traditional bag-of-words (BOW) representations,whereas they are semantically related in the real context. The situation is similar for both ‘movie’ and ‘film’ words. Hence, from this perspective, if we can identify the semantic relatedness of words, then we can enrich S3 and S4 with related words as follows:

S3′: a pleasant movie enjoyable film.

S4′: an enjoyable film pleasant movie.

We can see that S3′ and S4′ share common words. By using BOW representation in a traditional classifier, we can get the same polarity for both sentences.

From the above examples, we can see that both latent topics and related words are possible ways to alleviate the data-sparseness problem. Since probabilistic topic models have been widely used to perform latent topic analysis, in this work, we also use the models to obtain latent topics. Inspired by the success of continuous space word representations in capturing the semantic similarities in various NLP tasks, we use word embeddings to derive related words. We then utilize latent topics and related words as additional features to enrich the representation of reviews. Furthermore, we come up with an ensemble approach for polarity classification by combining the individual classifiers.

The approach consists of four steps. First, we train a topic model and learn word embeddings on a large external dataset. Second, we utilize two types of additional features including latent topics and related words to enrich the original feature space of the test corpus, respectively. Third, we train two classifiers for classification. One is trained on the representations enriched with the latent topics, and the other is trained on the representations enriched with related words. Finally, an ensemble technique is applied to the two classifiers to obtain a combination classifier.

To the best of our knowledge, no previous work has explored both topic model and word embeddings to alleviate the data-sparseness problem in sentence-level sentiment analysis, and this work is the first attempt towards the problem. We demonstrate that our approach can effectively take advantage of external data to improve the performance. Moreover, since a large external dataset can be gathered easily because huge document collections are widely available from the Web, the approach is suitable and practical for real applications.

The remainder of this article is organized as follows. Section 2 introduces related work. The proposed approach is described in detail in Section 3. The experimental setup and results are presented in Sections 4 and 5, respectively. Finally, Section 6 concludes the paper and outlines directions for future research.

2. Related work

Since our work is closely related to short-text processing and sentiment analysis, in this section, we first present related work on short-text classification and clustering, and then related work on sentence-level polarity classification is presented.

2.1. Short-text classification and clustering

Different from traditional normal documents, short texts typically only include a few words or sentences. Therefore, the sparsity of content in short texts brings new challenges to classification and clustering. To deal with the data-sparseness problem, researchers made considerable efforts to enrich the representation of short texts in classification and clustering. There are mainly three major approaches that have been exploited. The first is acquiring additional information from the Web by using search engines to evaluate similarity measures for short texts, and then performing the short-text classification based on similarity [13, 17]. The second is using world knowledge such as Wikipedia or WordNet to expand feature terms [12, 16]. The third is mining latent topics from an existing large corpus using topic model such as latent Dirichlet allocation (LDA), and then using latent topics as features to enrich the representation of short texts [9, 14]. In Phan et al. [9] and Chen et al. [14], the classification performance is significantly improved by enriching feature vector with relevant latent topics derived from the topic model.

2.2. Sentence-level polarity classification

In the SA field, much work has been done on sentence-level sentiment analysis during recent years. Usually, two types of approaches are proposed to tackle the problem: lexicon-based approaches and machine learning-based approaches.

Lexicon-based approaches assume that the sentiment orientation of a sentence is determined by summing up the orientation scores of all sentiment words in the sentence. For example, a positive word is given the sentiment score of +1 and a negative word is given the sentiment score of −1. Early work using the lexicon-based approach includes Hu and Liu [4]. Contextual valence shifters such as negation and intensification are considered in subsequent works [18, 19]. Thelwall et al. [20] used a lexicon-based classifier called as SentiStrength to simultaneously detect positive and negative sentiment strength from short informal text. For each text, the SentiStrength assigns a score of 1–5 for positive sentiment strength and a separate score for negative sentiment strength.

Machine learning-based approaches treat the problem as a text classification task. They use machine-learning algorithms and features such as unigrams, bigrams, part-of-speech (POS) tags, etc. to train classifiers. Davidov et al. [21] proposed a supervised learning approach that utilizes 50 Twitter tags and 15 smileys as sentiment labels to avoid the need for labour-intensive manual annotation. Gamon et al. [22] used a semi-supervised learning algorithm based on Expectation Maximization to learn a classifier from a small set of labelled sentences and a large set of unlabelled sentences. By encoding lexical and discourse knowledge as expressive constraints and integrating them into the learning of conditional random field models, Yang and Cardie [5] proposed a context-aware method for analysing sentiment at the level of individual sentences.

In the domain of SA, tweets, short product reviews can be treated as short texts. To alleviate the data sparseness problem, Saif et al. [23] used semantic concept features and sentiment-topic features in Twitter classification and improved the performance. Montejo-Ráez et al. [24] proposed an unsupervised approach that is based on the expansion of the concepts expressed in the tweets through the application of PageRank over the WordNet graph, and the approach can improve overall performance.

3. The proposed approach

3.1. The general framework

In this section, we introduce the framework of our approach. The processing flow of the framework is shown in Figure 1. It is mainly composed of the following steps:

Figure 1.

Framework of the proposed approach.

carrying out topic analysis and performing learning of word embeddings on a large external dataset T;

performing topic inference on test corpus;

enriching the training set D_train and testing set D_test with latent topics, then building the classifier C_topic on enriched training set $D_{train}^{*}$ and classifying the enriched test set $D_{test}^{*}$ ;

enriching the training set D_train and testing set D_test with related words learned from word embeddings, then building the classifier C_embeddings on enriched training set $D_{train}^{#}$ and classifying the enriched test set $D_{test}^{#}$ ;

getting the final classifier via an ensemble of the classifier C_topic and C_embeddings.

In the presented framework, choosing a proper large external dataset is important. For the classification task, we need to collect a dataset that is large enough to cover a lot of words, concepts and topics that are relevant to the classification problem. Besides that, the large external dataset and test dataset should be from the same domain, so that the large dataset is consistent with the training and future unseen data that the sentiment classifier will work with. It should be noted that the large dataset can be gathered easily because huge document collections are widely available on the Web.

3.2. Building classifiers with latent topics

To automatically discover latent topics from a text collection to enrich the representation of short texts, previous works of [9, 14] used LDA as topic model to perform topic analysis and achieved significant results. However, in this work, we focus on a different task: sentiment analysis. As stated in Pang et al. [7], compared with traditional text classification, polarity classification is a more difficult task, and sentiment can often be expressed in a more subtle manner, making it more difficult to identify. Naturally, when confronted with polarity classification task, we want to know whether the topics extracted from text collections using LDA model are still effective for enriching the representation of texts. Moreover, since sentiment polarities are intuitively dependent on topics, we also try to check whether sentiment-specific topics extracted using the joint sentiment-topic (JST) model are effective for enriching the text representations. In this section, we first describe the LDA and JST model used for topic analysis respectively. Second, we describe two different integration strategies for enriching the text representations. Finally, we build classifiers on the enriched dataset.

3.2.1. LDA model

As a well-known probabilistic generative model, LDA has been widely used in various NLP tasks to extract latent topics from a given text corpus. The generative graphical model of LDA is shown in Figure 2.

Figure 2.

Graphical model of LDA.

Assume that we have a document collection D, for each document d that is a sequence of N_d words, the generative process corresponding to the graphical model shown in Figure 2 is as follows:

For each topic $k \in {1 . . K}$ , choose a distribution $φ_{k} ~ Dir (β)$ .

For each document $d \in D$ , choose a distribution $θ_{d} ~ Dir (α)$ .

For each word $w$ in document d,

choose a topic $z ~ Mult (θ_{d})$ ,

choose the word $w ~ Mult (ϕ_{z})$ .

Here, $α$ and $β$ denote Dirichlet parameters, $θ_{d}$ denotes the topic distribution for document d, $Θ = (θ_{d})_{d = 1}^{D}$ , $φ_{k}$ denotes the word distribution for topic k, $Φ = (φ_{k})_{k = 1}^{K}$ , and K denotes the total number of topics. To estimate the parameters $Θ$ and $Φ$ of LDA model, approximate estimation methods such as variational inference methods [25] and Gibbs sampling [26] are often applied. After estimating a topic model on a text corpus by using Gibbs sampling, given a new document, topic inference for the document also needs to perform Gibbs sampling with a much smaller sampling iterations than that for the parameter estimation.

3.2.2. JST model

By adding an additional sentiment layer between the document and the topic layer, JST can be used to detect sentiment and topic simultaneously from text [27]. The graphical model of JST is represented in Figure 3. The generative process in JST corresponding to the graphical model shown in Figure 3 is as follows:

Figure 3.

Graphical model of JST.

For each topic $k \in {1 . . K}$ associated with each sentiment label $l \in {1 . . S}$ , pick a distribution $φ_{k}^{l} ~ Dir (β)$ . S is the number of distinct sentiment labels, K is the total number of topics, and $φ_{k}^{l}$ is the per-corpus joint sentiment-topic word distribution.

For each sentiment label $l \in {1 . . S}$ under document d, choose a distribution $θ_{l}^{d} ~ Dir (α)$ , where $θ_{l}^{d}$ is the sentiment label l specific topic proportion for document d.

For each document d, choose a distribution $π_{d} ~ Dir (γ)$ . $π_{d}$ is the sentiment distribution of the document d.

For each word w in document d,

choose a sentiment label $l ~ Mult (π_{d})$ ,

choose a topic $z ~ Mult (θ_{l}^{d})$ ,

choose the word $w ~ Mult (φ_{z}^{l})$ .

In JST, there are three parameters that need to be estimated: the joint sentiment/topic-document distribution $Θ$ , which is a $S \times K \times | D |$ matrix; the joint sentiment/topic-word distribution $Φ$ , which is an $S \times K \times V$ matrix, where V is vocabulary size; and the sentiment-document distribution $π$ , which is an $S \times | D |$ matrix $π = (π_{d})_{d = 1}^{| D |}$ It should be noted that the topic distribution of JST is different from that of LDA. In JST, each document is associated with S topic distributions. In contrast, there is only one topic distribution for each document in LDA. By applying Gibbs sampling, these parameters can be estimated from text corpus. As stated in Lin and He [27], sentiment prior information obtained by using different ways can be incorporated into the JST model. Specifically, in this article, we use the MPQA¹ subjectivity lexicon as prior information incorporated into the model learning.

3.2.3. Strategies for integrating latent topics into data

Strategy 1: after doing topic inference, to integrate $θ_{d} = (θ_{k}^{d})_{k = 1}^{K}$ into a document d, following the technique described in Phan et al. [9], topics that have high probability will be treated as external terms and added to the corresponding review. The frequency of the name of topic k added in d is denoted as ${freq}_{k}^{d}$ ; it is determined by topic probability value using the formula (1):

{freq}_{k}^{d} = {\begin{matrix} round (scale \times θ_{k}^{d}), if θ_{k}^{d} \geq cut - off \\ 0, if θ_{k}^{d} \leq cut - off \end{matrix}

(1)

The formula (1) has two parameters: scale and cut-off, where scale is the parameter that determines topic frequency added into review d, and cut-off is the topic probability threshold. We can extend the original review d by adding latent topics with high probabilities to its content, which makes it enriched and more topic-focused. After integrating latent topics into the review, a new extended vector $d^{*}$ can be obtained.

Strategy 2: it can be observed from a trained model that words grouped under one topic are often informative and coherent, and we can augment the original text with topic assigned to each word so grouping words under the same topic could potentially reduce data sparseness in sentiment analysis. Following the technique described in Saif et al. [23], we consider another strategy to enrich the text representation using latent topics. After training a topic model on the large external dataset with class labels being discarded, we can perform topic inference on test dataset using the trained model, and the resulting model will assign each word in reviews with a topic label. Note that for the JST model, besides a topic label, each word in reviews will also be assigned with a sentiment label. To avoid introducing too much noise, rather than augmenting the topic label of each word in reviews, we just consider those words that may express sentiments. Since most sentiment words are adjectives, adverbs and verbs, we consider adjectives, adverbs and verbs in the reviews and augment their corresponding topic labels as additional features into the original feature space for classifier training.

3.2.4. Training the classifier

After integrating latent topics into data, in this study, for the sake of comprehensiveness, we choose NB and MaxEnt as classification methods to build classifiers on enriched data. They are robust and have been applied successfully to a wide range of NLP tasks. In addition, they are very fast in training speed and can be applied to almost real-time applications.

3.3. Building classifiers with related words

In this section, we will describe how to use related words obtained from word embeddings to enrich the training and testing set, and then build a classification model on the enriched dataset. In traditional BOW representation, it is common to represent words as indices in a vocabulary. Although this choice is simple and robust, it fails to sufficiently capture the complex linguistic characteristics of words. With the revival of interest in deep learning, representation of words as continuous vectors (also called ‘word embeddings’) has been effectively used in a variety of NLP tasks including parsing, language modelling and NER [28]. Word embeddings typically represent words with low-dimensional, dense and real-valued vectors. Most recently, two log-linear models for learning word embeddings were proposed by Mikolov et al. [29], namely the Continuous Bag-of-Words (CBOW) and Skip-gram model. Figure 4 shows the architectures of CBOW and Skip-gram model.

Figure 4.

CBOW and Skip-gram model.

In Figure 4, w_t denotes the tth word in a corpus. The CBOW model predicts the current word based on a set of surrounding context words in a window of size c, and the Skip-gram model predicts surrounding words given the current word w_t as input. In Mikolov et al. [29], the authors demonstrated that the learned word representations could measure syntactic and semantic word similarities. In Baroni et al. [30], the authors performed a systematic comparison of context-counting and context-predicting semantic vectors on a wide range of lexical semantics tasks. The results show that predict models trained with the word2vec toolkit² achieve an impressive overall performance. Hence, in this article, we propose to use word embeddings to capture the semantic word similarities and find related words for feature terms in the reviews.

The word2vec toolkit implements both the Skip-gram and CBOW models for learning word embeddings. In this work, we employ the toolkit to pre-train the word embeddings on the large external dataset. In Mikolov et al. [29], experiment results have shown that the Skip-gram model performs better than the CBOW model in identifying semantic relationship among words. Therefore, we employ the Skip-gram model for estimating word embeddings.

After training word embeddings on the large external dataset, our next task is to find semantically related words for the reviews by using the embeddings. Our goal is to use word w_s in the review to select word w_t in the large external dataset as the closest matching word to enrich review text.

Having obtained vector representations of words, we could determine the semantic relatedness of two words (e.g. w_s and w_t) as distance between word vectors in a high-dimensional space.

Relatedness (w_{s}, w_{t}) = ψ (w_{s}) . ψ (w_{t})

(2)

In formula (2), $ψ (w_{s})$ denotes the word vector of w_s. Given a sentence-level review dataset, we first split the reviews into individual sentences and carry out POS tagging on each sentence. Certain words such as adjectives, adverbs and verbs are filtered out since their related words are often associated with positive or negative polarity, thus, for each adjective, adverb or verb denoted as w_s in the review, we use the formula (2) to measure word relatedness between w_s and each word w_t in the large external dataset, then treat the word w_t which has the highest score as a related word of w_s and add it into the review text as a feature.

For the original review d, if we denote those filtered words as $s_{d} = {s_{1}, . ., s_{m}}$ , where m denotes the total number of filtered words, then we can denote the related words that will be added into the review as $s^{#} = {s_{1}^{#}, \dots, s_{m}^{#}}$ , where $s_{i}^{#}$ is the corresponding related word of $s_{i}$ , and a new extended vector $d^{#} = d \cup s^{#}$ can be obtained. By using the extended vectors, we then train NB and MaxEnt classifiers from the labelled reviews to predict positive and negative sentiment in unlabelled reviews.

3.4. Ensemble combination

The ensemble technique, which combines the outputs of base classification models to obtain an integrated output, has become an effective method for polarity classification [31, 32]. Inspired by these works, after obtaining classifiers by using the approaches presented in the previous sections, we exploit the ensemble technique to build an ensemble classifier by combining the class predictions of different classifiers.

By associating each individual prediction value with a weight, we can use the following ensemble method for deriving a new semantic orientation value.

p (s | d) = α * p_{topic} (s | d) + (1 - α) * p_{emb} (s | d), s \in {pos, neg}

(3)

where d denotes the review text and s denotes the class label of d, $α \in [0, 1]$ is the weight associated with each individual classifier. p_topic(s|d) is the confidence value of the C_topic classifier for d, and C_topic is trained from the training set enriched with latent topics. Similarly, $p_{emb} (s | d)$ denotes the confidence value of the C_embeddings classifier for d, and C_embeddings is trained from the training set enriched with related words. The coefficient $α$ is used to control the influence of the each individual classifier. By setting $α$ to 0, the ensemble classifier becomes the C_embeddings classification model without using any latent topics information. On the other hand, by setting $α$ to 1, the ensemble classifier becomes the C_topic classification model without using any related words information.

3.5. The algorithm of the proposed approach

On the basis of the above discussions, a main technical flow has been established for applying our approach to the sentence-level polarity classification problem. The algorithm of the approach is illustrated as follows.

Algorithm 1. ensemble-based approach.
Input: The training set D_train and testing set D_test; a large external dataset T; parameter $α$ which is used as the weight associated with classifier; parameter K which is the total number of topics
Output: The ensemble classifier;
1. Train a topic model on T, and the model has K topics.
2. Infer latent topics on D_train and D_test by using the model.
3. $D_{train}^{} = \emptyset$ , $D_{test}^{} = \emptyset$ .
4. for each review d in {D_train, D_test}
5. Augment d as $d^{*}$ with the latent topics by using one of the two strategies described in Subsection 3.2.3.
6. if (d∈D_train) then $D_{train}^{} = D_{train}^{} \cup d^{*}$
7. if (d∈D_test) then $D_{test}^{} = D_{test}^{} \cup d^{*}$
8. end for
9. Train a classifier C_topic on $D_{train}^{}$ , and then use C_topic* to classify. $D_{test}^{*}$
10. Learn word embeddings on T with the document labels discarded.
11. $D_{train}^{#} = \emptyset$ , $D_{test}^{#} = \emptyset$ .
12. for each review d in {D_train, D_test}
13: Augment d as $d^{#}$ with related word feature set obtained from word embeddings.
14. if (d∈D_train) then $D_{train}^{#} = D_{train}^{#} \cup d^{#}$
15. if (d∈D_test) then $D_{test}^{#} = D_{test}^{#} \cup d^{#}$
16. end for
17. Train a classifier C_embeddings on $D_{train}^{#}$ , and then use C_embeddings to classify. $D_{test}^{#} = {d_{m}^{#}}_{m = 1}^{N_{test}}$
18. $D_{test}^{#}$ for each review d in D_test
19. p(s\|d)=αp_topic(s\|d)+(1 −α)* p_emb* (s\|d), where s∈{pos,neg}
20. the predicted class label of d is argmax p(s\|d)
21. end for

In the framework of our approach, the topic model can be LDA or JST model. Also, when enriching text representation using latent topics, we can adopt one of the two strategies to add latent topics as additional features.

4. Experimental setup

4.1. Dataset description and evaluation metrics

We make use of the two public datasets in our experiments. The summary of the datasets is shown in Table 1. The first is the sentence polarity dataset,³ which is used as the test corpus; the sentence-level dataset was introduced in Pang and Lee [33], and it consists of 10,662 sentences selected from movie review websites. The other dataset is large movie review dataset⁴ first proposed by Maas et al. [34] as a benchmark dataset for sentiment analysis. The document-level dataset consists of 100,000 informal movie reviews from the Internet Movie Database. The dataset is divided into three parts: 25,000 labelled training instances, 25,000 labelled test instances and 50,000 unlabelled instances. In both the training and test set, there are two types of labels: positive and negative, and these labels are balanced.

Table 1.

The summary of the datasets.

Dataset	Labeled positive reviews	Labeled negative reviews	Unlabelled reviews
Sentence polarity dataset	5331	5331	0
Large movie review dataset	25,000	25,000	50,000

Intuitively, since the large movie review dataset and the sentence polarity dataset are from the movie domain, and the former dataset has a much larger size than the latter, it is reasonable to use the former as the large external dataset in our approach. Note that, when the dataset is used as the external dataset, the class labels are discarded.

In the experiments, we divide the sentence polarity dataset into two parts with balanced labels. The first part is used as development set for parameter tuning and it has 2000 reviews. The remaining 8662 reviews are used as training set and test set data. We have two settings, a development setting (DEV) and a test setting (TEST). In the development setting, we run the typical five-fold cross-validation where we train on four folds and test on the other fold, and then average the results. In the test setting, we run with the best configurations yielded from the development setting, and the results are averaged over 10 different training/test splits of the 8662 reviews. For each split, we randomly select 6000 reviews as training set and the remaining 2662 reviews are treated as test set for the final evaluation. The test set is not inspected while we develop the algorithms.

To measure overall performance, we use accuracy metric. Accuracy is a measure of what percentage of the identified reviews are correctly classified in the whole test collection.

4.2. Data pre-processing

Before the subsequent experiments, we conduct the following pre-processing steps on the sentence polarity dataset and the large movie review dataset. First, punctuations, numbers and other non-alphabetical characters are removed. Stop words are also removed based on the default stop word list provided in Mallet toolkit.⁵ Second, for the purpose of reducing the vocabulary size, stemming is performed using Porter’s stemmer algorithm. Besides, on the sentence polarity dataset, the reviews are annotated with POS tags using Stanford POS tagger.⁶

4.3. Experiment settings

To perform the review enrichment with latent topics derived from the large external dataset, we use GibbsLDA++ toolkit⁷ to estimate LDA models on the external dataset, and use JST C++ implementation⁸ to estimate JST models on the external dataset. All of the LDA and JST models are estimated with different number of topics (3, 9, 15, 21, 30, 45 and 60), and the influence of the number of topics will be discussed further in this work. When estimating the LDA and JST models, the hyper parameter α is set to 50/K, where K is the number of topics. The hyper parameter β is set to 0.01, and the number of Gibbs sampling iterations is set to 1200. Based on pilot experiments on the development data, the parameter αis set to 0.6; it is used to control the influence of each individual classifier in the ensemble based method.

In the experiments, when using strategy 1 to integrate latent topics extracted from topic model into the text, the two parameters cut-off and scale need to be set. For integrating latent topic features extracted from the LDA models, we use the following settings that are tuned on the development set: {(3, 0.4, 5), (9, 0.13, 20), (15, 0.08, 30), (21, 0.06, 40), (30, 0.04, 50), (45, 0.025, 80), (60, 0.02, 100)}, where (3, 0.4, 5) denotes when K=3, cut-off=0.4 and scale=5, and the rest of triples are similar. For integrating sentiment-topic feature sets extracted from the JST models, the following settings tuned on the development set are used: {(3, 0.6, 4), (9, 0.5, 5), (15, 0.4, 6), (21, 0.35, 7), (30, 0.3, 8),(45, 0.25, 9), (60, 0.2, 10)}. Note that, in the JST model, for example, when K is set to 21, it indicates that under each of the positive, negative and neutral sentiment labels there are seven topics. Hence the total number of sentiment-topics is 21.

Learning of word embeddings is performed by using the word2vec toolkit. We use the Skip-gram model to train word vectors with vector dimensionality value set to 50. For the context window size, as pointed out in Levy and Goldberg [35], a window size of 5 is commonly used to capture broad topical content, whereas smaller context window of size 2 may miss some important contexts. Following this point, we also use a window of size 5 around the target word w. To build classifiers, we choose Mallet toolkit implementations of NB and MaxEnt with default parameter settings.

Rather than just comparing the accuracies, we make most of the decisions based on statistical significance test. For simplicity, in the following analysis, unless otherwise specified, statistical significance is evaluated using a paired t-test at the p<0.05 level.

5. Experiment results and evaluation

In the experiments presented below, for comparing the effectiveness, the following seven different text representation methods are evaluated on the sentence polarity dataset:

BOW (baseline 1) – traditional ‘bag of words’ model with the term frequency (TF) weighting method.

BOW+LDA (baseline 2) – BOW integrated with additional latent topic features learned from the LDA model. The topic distribution is integrated into the original text by using strategy presented in Phan et al. [9], and the strategy is named as strategy 1 in this work, as described in Subsection 3.2.3.

BOW+JST (baseline 3) – BOW integrated with additional latent topic features learned from the JST model. The topic distribution is integrated into the original text using strategy 1.

BOW+LDAtsgn (baseline 4) – BOW integrated with additional latent topic features learned from the LDA model. Unlike BOW+LDA, the topic distribution is integrated into the original text by using strategy presented in Saif et al. [23], and the strategy is named strategy 2 in this work, a described in Subsection 3.2.3.

BOW+JSTtsgn (baseline 5) – BOW integrated with additional latent topic features learned from the JST model; the topic distribution is integrated into the original text using strategy 2.

BOW+RelWord (baseline 6) – BOW integrated with additional related word features learned from the word embeddings using the word2vec toolkit.

BOW+Lex (baseline 7) – BOW integrated with additional features derived from the publicly available sentiment lexicon. The lexicon features include NumPositive and NumNegative. The former denotes the number of positive expressions in the review, and the latter denotes the number of negative expressions in the review. We use the MPQA subjectivity lexicon as the sentiment lexicon. The lexicon contains 2718 positive and 4912 negative words. When recognizing the positive or negative expression, we take into account negation words, and there are 88 negation words defined in the MPQA subjectivity lexicon. Specifically, within the window of two words previous to the polarity word w, if there is a term w1 that belongs to the negation lexicon, then the polarity of the word w is reversed.

We also use the two types of features learned from the large-scale external dataset as joint features to enrich text representations, and the following four different methods are evaluated on the sentence polarity dataset:

BOW+BothLDA – BOW integrated with additional features learned from the LDA model and word embeddings as in BOW+LDA and BOW+RelWord.

BOW+BothJST – BOW integrated with additional features learned from the JST model and word embeddings as in BOW+JST and BOW+RelWord.

BOW+BothLDAtsgn – BOW integrated with additional features learned from the LDA model and word embeddings as in BOW+LDAtsgn and BOW+RelWord.

BOW+BothJSTtsgn – BOW integrated with additional features learned from the JST model and word embeddings as in BOW+JSTtsgn and BOW+RelWord.

We then investigate using the ensemble method introduced in earlier section to combine different text representation methods, and the following eight different representation combinations are also evaluated on the sentence polarity dataset:

Ensemble-LDA – this is an ensemble of individual BOW+RelWord and BOW+LDA classifiers.

Ensemble-JST – this is an ensemble of individual BOW+RelWord and BOW+JST classifiers.

Ensemble-LDAtsgn – this is an ensemble of individual BOW+RelWord and BOW+LDAtsgn classifiers.

Ensemble-JSTtsgn – this is an ensemble of individual BOW+RelWord and BOW+JSTtsgn classifiers.

Ensemble-BothLDA – this is an ensemble of individual BOW+RelWord and BOW+BothLDA classifiers.

Ensemble-BothJST – this is an ensemble of individual BOW+RelWord and BOW+BothJST classifiers.

Ensemble-BothLDAtsgn – this is an ensemble of individual BOW+RelWord and BOW+ BothLDAtsgn classifiers.

Ensemble-BothJSTtsgn – this is an ensemble of individual BOW+RelWord and BOW+ BothJSTtsgn classifiers.

5.1. Experiments on test data

In TEST mode, for each training/test split, to better understand the effect of the proposed approach on the test data, we used different sized subsets of the training set: starting from 10% of its original size, and increasing the size by 10% each time, and a test on the test data exclusively. The classification results of the above methods using NB and MaxEnt as the base classifier are reported in Tables 2 and 3, respectively. In Tables 2 and 3, under different topic numbers, the value of each method is the average value of 10 different training/test splits, and for each split, the accuracy value is averaged over the 10 classifiers that are trained with different sized training sets and tested on the test set. Ave-Acc denotes the average accuracy for each method over different topic numbers. Note that BOW, BOW+Lex and BOW+RelWord do not use latent topics information and they have nothing to do with the topic number. However, to facilitate comparison with the other methods, we also present the results of these methods in Tables 2 and 3, and the values of each method under different topic numbers are the same. For the purpose of discussion, we also give the detailed results of classifiers in Tables 4 and 5. Without loss of generality, we only show the results of classifiers with the topic number K set to 21.

Table 2.

Accuracies (%) on test data: using NB as the base classifier.

Method	Topic number							Ave-Acc
	3	9	15	21	30	45	60
BOW	71.85	71.85	71.85	71.85	71.85	71.85	71.85	71.85
BOW+Lex	73.26	73.26	73.26	73.26	73.26	73.26	73.26	73.26
BOW+LDA	71.87	72.95	72.15	72.18	73.56	72.18	72.07	72.42
BOW+JST	71.73	71.89	72.71	72.21	72.76	71.82	71.68	72.12
BOW+LDAtsgn	72.20	72.97	72.89	73.05	73.38	73.01	72.99	72.93
BOW+JSTtsgn	72.49	72.97	73.44	73.88	74.00	73.80	73.08	73.38
BOW+RelWord	73.03	73.03	73.03	73.03	73.03	73.03	73.03	73.03
BOW+BothLDA	73.01	73.75	73.16	73.43	74.11	73.12	73.24	73.40
BOW+BothJST	73.01	73.13	73.85	73.26	73.75	73.11	72.87	73.28
BOW+BothLDAtsgn	73.20	73.60	73.50	73.77	73.86	73.59	73.69	73.60
BOW+BothJSTtsgn	73.38	73.77	74.24	74.37	74.72	74.39	73.91	74.11
Ensemble-LDA	73.08	73.88	73.28	73.55	74.21	73.39	73.37	73.54
Ensemble-JST	73.10	73.26	73.94	73.56	73.93	73.19	73.11	73.44
Ensemble-LDAtsgn	73.29	73.71	73.65	73.78	73.97	73.72	73.78	73.70
Ensemble-JSTtsgn	73.37	73.77	74.25	74.52	74.75	74.42	74.08	74.16
Ensemble-BothLDA	73.05	73.87	73.33	73.78	74.00	73.64	73.80	73.64
Ensemble-BothJST	73.25	73.48	73.99	73.82	74.08	73.74	73.50	73.70
Ensemble-BothLDAtsgn	73.21	73.50	73.45	73.62	73.72	73.53	73.63	73.52
Ensemble-BothJSTtsgn	73.56	73.79	74.19	74.36	74.48	74.29	74.02	74.10

Table 3.

Accuracies (%) on test data: using MaxEnt as the base classifier.

Method	Topic number							Ave-Acc
	3	9	15	21	30	45	60
BOW	70.24	70.24	70.24	70.24	70.24	70.24	70.24	70.24
BOW+Lex	71.66	71.66	71.66	71.66	71.66	71.66	71.66	71.66
BOW+LDA	70.33	71.17	70.67	71.14	71.82	71.21	71.07	71.06
BOW+JST	70.70	70.51	71.48	71.26	71.14	70.84	70.60	70.93
BOW+LDAtsgn	70.91	70.99	70.98	71.23	71.40	71.02	70.73	71.04
BOW+JSTtsgn	71.13	71.31	71.97	72.03	71.84	71.62	71.18	71.58
BOW+RelWord	70.85	70.85	70.85	70.85	70.85	70.85	70.85	70.85
BOW+BothLDA	70.90	71.79	71.11	71.56	71.83	71.51	71.27	71.42
BOW+BothJST	71.26	71.08	71.77	71.47	71.52	71.33	71.22	71.38
BOW+BothLDAtsgn	71.33	71.20	71.27	71.45	71.39	71.10	70.90	71.23
BOW+BothJSTtsgn	71.47	71.73	72.16	72.27	71.99	71.76	71.53	71.84
Ensemble-LDA	71.46	72.40	71.72	72.19	72.76	72.33	72.27	72.16
Ensemble-JST	71.85	71.73	72.38	72.25	72.18	72.08	72.02	72.07
Ensemble-LDAtsgn	71.84	72.00	72.02	72.23	72.29	72.07	71.89	72.05
Ensemble-JSTtsgn	72.16	72.27	72.70	72.84	72.71	72.47	72.19	72.48
Ensemble-BothLDA	70.92	71.81	71.18	71.66	72.01	71.79	71.63	71.57
Ensemble-BothJST	71.27	71.31	71.78	71.63	71.65	71.60	71.55	71.54
Ensemble-BothLDAtsgn	71.28	71.33	71.35	71.51	71.54	71.29	71.22	71.36
Ensemble-BothJSTtsgn	71.62	71.50	71.99	72.21	72.09	71.83	71.74	71.85

Table 4.

Accuracies (%) on test data: using NB as the base classifier and the topic number K=21.

Method	Percentage of the training set										Ave-Acc
	10%	20%	30%	40%	50%	60%	70%	80%	90%	100%
BOW	64.42	68.28	70.24	71.21	72.74	73.33	74.04	74.27	74.68	75.26	71.85
BOW+Lex	67.19	70.21	71.59	73.31	73.81	74.27	74.89	75.23	75.82	76.22	73.26
BOW+LDA	68.06	70.00	71.22	71.81	72.58	72.97	73.29	73.72	74.01	74.14	72.18
BOW+JST	68.54	70.40	71.50	71.85	72.39	72.86	73.55	73.61	73.58	73.83	72.21
BOW+LDAtsgn	67.76	70.49	71.65	72.61	73.50	74.05	74.62	75.07	75.08	75.67	73.05
BOW+JSTtsgn	70.02	71.95	73.08	73.59	74.13	74.51	74.91	75.31	75.53	75.79	73.88
BOW+RelWord	66.42	70.29	71.78	72.72	73.66	74.14	74.76	75.20	75.50	75.87	73.03
BOW+BothLDA	69.23	71.49	72.60	73.16	73.67	74.19	74.57	74.93	75.06	75.39	73.43
BOW+BothJST	69.52	71.45	72.50	73.35	73.43	73.83	74.38	74.40	74.70	75.06	73.26
BOW+BothLDAtsgn	68.29	71.57	72.63	73.34	74.29	74.75	75.13	75.50	75.90	76.29	73.77
BOW+BothJSTtsgn	70.61	72.71	73.46	74.27	74.68	75.09	75.41	75.53	75.79	76.12	74.37
Ensemble-LDA	68.99	71.32	72.38	73.22	74.06	74.34	74.91	75.08	75.43	75.82	73.55
Ensemble-JST	69.42	71.50	72.43	73.14	73.92	74.41	74.89	75.03	75.29	75.59	73.56
Ensemble-LDAtsgn	67.92	71.03	72.58	73.35	74.42	74.78	75.46	75.74	76.06	76.47	73.78
Ensemble-JSTtsgn	70.25	72.43	73.68	74.07	74.88	75.27	75.84	75.95	76.18	76.62	74.52
Ensemble-BothLDA	68.94	71.59	72.78	73.37	74.16	74.72	75.12	75.34	75.69	76.14	73.78
Ensemble-BothJST	69.56	71.76	72.96	73.76	74.17	74.67	75.11	75.06	75.48	75.73	73.82
Ensemble-BothLDAtsgn	67.65	71.30	72.46	73.17	74.22	74.73	75.11	75.49	75.87	76.19	73.62
Ensemble-BothJSTtsgn	70.13	72.47	73.44	74.12	74.76	75.23	75.58	75.63	76.00	76.26	74.36

Table 5.

Accuracies (%) on test data: using MaxEnt as the base classifier and the topic number K=21.

Method	Percentage of the training set										Ave-Acc
	10%	20%	30%	40%	50%	60%	70%	80%	90%	100%
BOW	63.72	67.17	68.86	69.84	71.03	71.26	71.92	72.18	73.05	73.36	70.24
BOW+Lex	67.21	69.37	70.30	71.28	72.15	72.34	72.92	73.12	73.92	74.02	71.66
BOW+LDA	66.50	68.95	69.82	70.82	71.75	71.80	72.41	72.58	73.30	73.47	71.14
BOW+JST	66.97	68.92	70.14	70.88	71.70	71.64	72.58	72.77	73.28	73.69	71.26
BOW+LDAtsgn	66.96	68.79	70.07	70.86	71.81	71.88	72.43	72.62	73.36	73.47	71.23
BOW+JSTtsgn	68.90	70.32	71.19	71.79	72.42	72.31	73.04	73.10	73.64	73.61	72.03
BOW+RelWord	65.06	68.43	69.85	70.63	71.32	71.79	72.24	72.45	73.31	73.46	70.85
BOW+BothLDA	67.22	69.41	70.56	71.39	72.12	72.25	72.57	72.97	73.45	73.65	71.56
BOW+BothJST	67.32	69.36	70.71	71.29	71.68	71.99	72.67	72.80	73.37	73.55	71.47
BOW+BothLDAtsgn	66.93	69.55	70.55	71.17	71.87	72.03	72.66	72.80	73.31	73.61	71.45
BOW+BothJSTtsgn	69.07	70.52	71.53	72.16	72.47	72.73	73.37	73.08	73.83	73.99	72.27
Ensemble-LDA	67.64	70.02	71.06	71.94	72.72	73.00	73.35	73.58	74.16	74.46	72.19
Ensemble-JST	67.92	70.00	71.15	72.01	72.73	72.81	73.49	73.66	74.22	74.46	72.25
Ensemble-LDAtsgn	67.76	70.11	71.17	71.97	72.86	72.89	73.12	73.52	74.36	74.49	72.23
Ensemble-JSTtsgn	69.55	70.94	71.97	72.69	73.42	73.05	73.88	73.79	74.46	74.61	72.84
Ensemble-BothLDA	67.67	69.70	70.73	71.44	72.13	72.26	72.66	72.93	73.49	73.62	71.66
Ensemble-BothJST	67.61	69.50	70.90	71.44	71.90	72.16	72.94	72.74	73.55	73.52	71.63
Ensemble-BothLDAtsgn	67.10	69.65	70.64	71.30	71.94	71.94	72.65	72.81	73.42	73.68	71.51
Ensemble-BothJSTtsgn	68.93	70.39	71.52	72.05	72.34	72.59	73.17	73.07	73.93	74.11	72.21

Next, we will evaluate the experiment results from the following aspects:

Whether the related word feature set is effective for enriching the representation of texts.

Whether the latent topic feature set is effective for enriching the representation of texts.

Whether the combination of the two types of additional features helps to improve the performance.

Whether the performance of a polarity classification system can benefit from the ensemble technique. Among various combination methods, which one can be selected as the winner.

For simplicity, in the following analysis, unless otherwise specified, we focus on the Ave-Acc result on the test data.

First, in Table 2, we observe that BOW+ RelWord yields an improvement of 1.18% (73.03 vs 71.85%) over BOW. By inspecting detail information in Table 4, we can see that the improvements in the results are statistically significant at the p<0.01 level using a paired t-test over 10 pairs of different-sized training sets, which confirms the benefit of integrating related words into the representation of review text.

To answer the second question, we then assess the performances of the following text representations methods including BOW+LDA, BOW+JST, BOW+LDAtsgn and BOW+JSTtsgn. In Tables 2 and 3, among the four methods, in terms of Ave-Acc, BOW+JSTtsgn performs statistically significantly better than the other methods using a paired t-test over pairs of different topic numbers, so BOW+JSTtsgn is considered to be the most effective. Also, BOW+JSTtsgn gains an improvement of 1.53% (73.38 vs 71.85%) and 1.34% (71.58 vs 70.24%) over BOW in Tables 2 and 3. In Tables 4 and 5, we find that BOW+JSTtsgn consistently outperforms BOW at the p<0.01 level using a paired t-test over 10 pairs of different training/test splits, which demonstrates the benefit and potential of integrating external sentiment topics knowledge into the representation of short texts. We also note that, in Tables 2 and 3, when K is changed from 3 to 60, in some cases, BOW+LDA does not show significant difference over BOW in sentence polarity classification. While in traditional short-text classification, the integration of hidden topics into BOW can always achieve significant improvements over BOW representations with different numbers of topics [9]. Moreover, in traditional short-text classification, BOW+LDA has a much larger margin of improvement than that in sentence polarity classification. For example, in Phan et al. [9], when K=50, BOW+LDA can achieve an impressive improvement of 23.14% (80.25 vs 57.11%) over BOW when classifying test data. Generally speaking, BOW+LDA in sentiment analysis is not as effective as that in traditional short-text classification; we speculate that the reason for this is that the LDA model focuses on discovering and analysing topics of documents without any analysis of sentiment in the text, which limits the usefulness of the mining topic results in sentiment analysis. When comparing strategy 1 with strategy 2, since in Tables 2 and 3, BOW+JSTtsgn significantly outperforms BOW+JST in terms of Ave-Acc, we conclude that strategy 2 is more effective than strategy 1. Furthermore, considering the fact that strategy 2 does not need to tune parameters, we recommend using it as the strategy to integrate latent topics extracted from topic model into the text representation.

When using the combination of the latent topics and related words as additional features to enrich the representation of texts, the corresponding methods will show performance improvements over individual additional features-based enriched representations. For example, in Tables 2 and 3, in terms of Ave-Acc, BOW+BothJSTtsgn significantly outperforms BOW+JSTtsgn and BOW+RelWord by using a paired t-test across different number of topics. So, joint use of features learned from the external dataset to enrich text representations is superior to the use of individual features as additional features. We believe that the improvements stem from the ability of the combination-based method to better utilize the characteristics of latent topics and related words to generate high-quality semantics for more accurate classification.

Next, we further investigate the effectiveness of ensemble technique. We find that, when using an ensemble of BOW+RelWord and one of the methods including BOW+LDA, BOW+JST, BOW+LDAtsgn and BOW+JSTtsgn, the ensemble-based methods can consistently obtain improvements over individual classifiers. For example, in Table 3, in terms of Ave-Acc, Ensemble+JSTtsgn significantly outperforms BOW+JSTtsgn and BOW+RelWord with an improvement of 0.9% (72.48 vs 71.58%) and 1.63% (72.48 vs 70.85%), respectively. The results show that the individual methods in the ensemble can complement each other, and even the combination of two weak individual methods can improve the overall accuracy. Furthermore, to gain a more comprehensive view of the ensemble method, we also investigate combining BOW+RelWord with one of the methods including BOW+BothLDA, BOW+BothJST, BOW+BothLDAtsgn, and BOW+BothJSTtsgn. We find that, in most cases, there is no significant difference between the performances of the joint features-based method and the ensemble-based method. We guess that, since the methods, which use joint extra features, have included latent topics and related word information, BOW+RelWord cannot guarantee being complementary to the other ensemble components, and the ensemble methods do not show superiority over the ensemble components. Generally, an ensemble usually benefits more from leveraging the distinct strengths of individual classifiers that can complement each other.

Finally, we compare BOW+BothJSTtsgn with Ensemble-JSTtsgn. In Table 2, there is no significant difference between the performances of both methods in terms of Ave-Acc. However, in Table 3, we find that Ensemble-JSTtsgn significantly outperforms BOW+BothJSTtsgn in terms of Ave-Acc, which provides suggestive evidence that, to achieve the same level of performance, we cannot replace our ensemble approach with a simpler setup that relies on a single classifier using all of the features currently exploited by different members of the ensemble. Generally, among the various kinds of methods, Ensemble-JSTtsgn is recommended. In Tables 2 and 3, in terms of Ave-Acc, an improvement is achieved of 2.31% (74.16 vs 71.85%) and 2.24% (72.48 vs 70.24%) over BOW, respectively. With the specified topic number, the improvements can be increased. For example, in Tables 4 and 5, in terms of Ave-Acc, Ensemble-JSTtsgn achieves an improvement of 2.67% (74.52 vs 71.85%) and 2.6% (72.84 vs 70.24%) over BOW respectively.

An interesting phenomenon observable from Tables 4 and 5 is that the overall trend of performance improvement for Ensemble-JSTtsgn over BOW is downwards along with increasing training set size, as shown in Figure 5. A similar trend can be observed for the other methods such as Ensemble-JST, Ensemble-LDAtsgn and BOW+JSTtsgn. This is most likely due to the fact that, when the training set size is small, the data becomes sparser and the additional features can help the classifier to alleviate the data sparseness problem. On the other hand, when the training set size becomes larger, the sparseness problem becomes less severe, and the margin of performance improvement achieved using additional features is lower.

Figure 5.

Accuracy improvement of Ensemble-JSTtsgn over BOW with different sizes of training data.

5.2. Data-driven features vs knowledge-based features

From the above all discussions, we can see that the two types of features learned from the large-scale external dataset are helpful for improving classification performance. In this respect, the work reported in this paper incorporates data-driven features as additional features to enrich the representation of texts. In the SA field, previous works [3, 36] have shown that sentiment lexicons can be treated as informative features for the sentiment analysis task, and sentiment lexicon-based features can be seen as knowledge-based features. In this section, we will compare the data-driven features with the knowledge-based features. From Tables 4 and 5, we find that BOW+Lex significantly outperforms BOW by using a paired t-test over pairs of different sized training sets and achieves an improvement of 1.41% (73.26 vs 71.85%) and 1.42% (71.66 vs 70.24%), respectively, and we can see that knowledge-based features are helpful for improving the performance, which is consistent with the previous works [3, 36]. In Tables 4 and 5, we find that the performances of BOW+Lex and BOW+JSTtsgn show no significant difference. We can conclude that the sentiment topic features derived from JST model can be as effective as the knowledge-based features. This is reasonable since the JST C++ implementation used by us also incorporates prior knowledge that is obtained from the MPQA subjectivity lexicon. When comparing BOW+Lex with BOW+RelWord, we find that the former significantly outperforms the later in Table 5. However, when the two types of data-driven features are used together as a single feature set, BOW+BothJSTtsgn significantly outperforms BOW+Lex.

From the above discussions, we can conclude that, in contrast to the knowledge-based features, the data-driven features can achieve equal or better performance, which also proves the effectiveness of the data-driven features learned from large-scale external dataset.

5.3. Analysis of parameters

There are two parameters in the proposed ensemble approach: the combination weight parameter α and the number of topics K. In this section, we will look into the influence of the two parameters on the proposed approach.

Figure 6 shows the results of the representative approaches by varying K from 3 to 60. We can observe that Ensemble-JSTtsgn, which utilizes the two types of features, steadily outperforms the individual feature-based approaches, which means our recommended approach is effective. Similar results can be observed for the other methods such as Ensemble-JST, Ensemble-LDAtsgn and Ensemble-LDA.

Figure 6.

Ensemble results with different numbers of topics.

By using NB as the base classifier, the peak performance of Ensemble-JSTtsgn is obtained when the number of topics K is set to 30. By using MaxEnt as the base classifier, the peak performance of Ensemble-JSTtsgn is obtained when the number of topics K is set to 21.

Next, we study the influence of the αsettings on the performance of the proposed approach. With K fixed as 21, by varying α value from 0.1 to 0.9, we perform another set of experiments. The results are shown in Figure 7. We can see that the peak performance of Ensemble-JSTtsgn is achieved when α is set to 0.6. Note that, when α is set to another value, in most cases, Ensemble-JSTtsgn still outperforms the other ensemble technique-based approaches, which also indicates that Ensemble-JSTtsgn is most suitable for polarity classification.

Figure 7.

Ensemble results with different α values.

5.4. Effect of external dataset size

To explore the effect of external dataset size on classification performance, on the basis of the large movie review dataset, we construct the following different sized external datasets. The first is a subset of the large movie review dataset; it only consists of 40% of the data randomly selected from the large movie review dataset. For convenience, we name it E-1 and name the large movie review dataset E-2. Furthermore, to facilitate comparisons with larger external dataset, we make use of the Amazon movie review dataset⁹ used in McAuley and Leskovec [37], which spans 15 years with 7,911,684 reviews. Specifically, we randomly select 200,000, 300,000 and 400,000 reviews from the Amazon movie review dataset and add these reviews into the large movie review dataset. As a result, we have three different sized external datasets, and we name these datasets E-3, E-4 and E-5, respectively. After pre-processing, external dataset statistics are given in Table 6.

Table 6.

External dataset statistics after pre-processing.

Dataset	Total reviews	Unzipped size(Mb)	Vocabulary size	Total words
E-1	40,000	48.2	62,627	3,065,283
E-2	100,000	120	218,791	7,766,186
E-3	300,000	286	277,352	18,681,719
E-4	400,000	369	293,428	24,129,043
E-5	500,000	451	307,675	29,545,934

To assess the effect of external dataset size on the performance of our approach, we conduct a set of experiments on the test data with the external datasets presented in Table 6. Representative results are shown in Table 7. For each training/test split, we obtain the average value of 10 classifiers with different-sized training sets on the test data. Results reported in Table 7 are averaged over 10 different splits. We conduct a paired t-test between E-2 and E-1 on the accuracy values of different numbers of topics, and find that the performance of Ensemble-JSTtsgn using E-2 as the external dataset is statistically significant better than that of Ensemble-JSTtsgn using E-1 as the external dataset. For the other external datasets including E-3, E-4 and E-5, we can obtain the same conclusion. Moreover, in the set M = {E-2, E-3, E-4, E-5}, there is no significant difference between the performance of Ensemble-JSTtsgn, which uses x∈M as the external dataset, and that of Ensemble-JSTtsgn, which uses y ∈ (M − x) as the external dataset. The results for the development data are also consistent with the trends observed on the test data. Owing to space limitation, we do not present detailed experiment results on the development data here.

Table 7.

Accuracies (%) of test data: using different-sized external datasets.

Topic number	Ensemble-JSTtsgn (NB as base classifier)					Ensemble-JSTtsgn (MaxEnt as base classifier)
Topic number	E-1	E-2	E-3	E-4	E-5	E-1	E-2	E-3	E-4	E-5
3	72.95	73.37	73.68	73.48	73.79	71.55	72.16	72.36	72.18	72.15
9	73.28	73.77	73.75	73.56	73.86	71.66	72.27	72.23	72.35	72.48
15	73.61	74.25	74.65	74.64	74.54	71.86	72.70	72.54	72.57	72.69
21	73.85	74.52	74.73	74.75	75.46	71.95	72.84	72.86	72.66	72.86
30	73.86	74.75	74.69	74.86	74.83	71.67	72.71	72.35	72.88	72.76
45	73.96	74.42	74.25	75.19	74.52	71.74	72.47	72.34	72.64	72.45
60	73.68	74.08	73.94	74.24	74.02	71.63	72.19	72.28	72.16	72.25

From the above discussions, we can see that, when the external dataset size is small, adding more data (e.g. from E-1 to E-2) to the external dataset can lead to better results. However, when the external dataset size reaches a certain level, adding more data (from E-2 to E-5) does not increase the performance, and the effect of external dataset size on the performance of our approach is minor.

6. Conclusion

In this article, we exploit both latent topics and related words as additional features to enrich the representation of texts, and then obtain an ensemble classifier by using the two additional feature sets to guide the design of different members of the ensemble. The latent topics features are extracted from a large external dataset using a topic model and the related words features are learned from word embeddings trained on the large external dataset. Our approach provides a way to make sparse short reviews more related and topic-focused. We carry out extensive experiments and the results show that the proposed approach is effective for improving the overall performance for sentence-level polarity classification. To summarize, the contributions mainly include three parts:

By making an extensive study of latent topic feature sets learned from topic model in combination with different integration strategies, we find that latent feature set extracted from the JST model is most effective for enriching the representation. Furthermore, for integration strategies, Strategy 2 is superior to Strategy 1.

We propose enriching the short-text representation with related words learned from word embeddings.

We propose an effective ensemble approach using the two types of additional features to guide the design of members of the ensemble, and experimental results show that the proposed approach effectively takes advantage of external data and can further improve the performance.

In future work, there are still several other avenues that might be explored. First, although the experiments are conducted only on the English corpus in this paper, we will attempt to apply the proposed approach to corpus in other languages. Second, as pointed out in Tang et al. [28], most existing algorithms for learning continuous word representations typically only model the syntactic context of words but ignore the sentiment of text, and we will investigate sentiment-specific word embeddings in the proposed approach. Finally, we will further apply the approach to various domains (e.g. hotel, travel, camera, laptops) in future work.

Footnotes

Acknowledgements

We sincerely thank anonymous reviewers for their valuable and insightful comments.

Funding

This work was supported by the National Planning Office of Philosophy and Social Science (grant number 14CTQ026).

Notes

References

Liu

. Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 2012; 5(1): 1.

Esuli

Sebastiani

. Determining the semantic orientation of terms through gloss classification. In: Proceedings of the 14th ACM international conference on Information and knowledge management (CIKM), 2005, pp. 617–624.

Wilson

Wiebe

Hoffmann

. Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis. Computational Linguistics 2009; 35(3): 399–433.

Liu

. Mining and summarizing customer reviews. In: Proceedings of the tenth ACM SIGKDD international conference on knowledge discovery and data mining, 2004, pp. 168–177.

Yang

Cardie

. Context-aware learning for sentence-level sentiment analysis with posterior regularization. In: Proceedings of the 52nd annual meeting of the Association for Computational Linguistics (ACL), Baltimore, MD, 2014, pp. 325–335.

Devitt

Ahmad

. Sentiment polarity identification in financial news: A cohesion-based approach. In: Proceedings of the 45th annual meeting of the Association of Computational Linguistics (ACL), 2007, pp. 984–991.

Pang

Lee

Vaithyanathan

. Thumbs up? Sentiment classification using machine learning techniques. In: Conference on empirical methods in natural language processing (EMNLP), Philadelphia, PA, 2002, pp. 79–86.

Yessenalina

Yue

Cardie

. Multi-level structured models for document-level sentiment classification. In: Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2010, pp. 1046–1056.

Phan

Nguyen

. A hidden topic-based framework toward building applications with short Web documents. IEEE Transactions on Knowledge and Data Engineering 2011; 23(7): 961–976.

10.

Peng

. Extended information inference model for unsupervised categorization of web short texts. Journal of Information Science 2012; 38(6): 512–531.

11.

Dai

Sun

Liu

. Crest: Cluster-based representation enrichment for short text classification. In: Advances in knowledge discovery and data mining. Berlin: Springer, 2013, pp. 256–267.

12.

Sun

Zhang

Chua

. Exploiting internal and external semantics for the clustering of short texts using world knowledge. In: Proceedings of the 18th ACM conference on Information and knowledge management, 2009, pp. 919–928.

13.

Shen

Pan

Sun

Pan

. Query enrichment for web-query classification. ACM Transactions on Information Systems 2006; 24(3): 320–352.

14.

Chen

Jin

Shen

. Short text classification improved by learning multi-granularity topics. In: Proceedings of the twenty-second international joint conference on artificial intelligence (IJCAI), 2011, pp. 1776–1781.

15.

Gabrilovich

Markovitch

. Harnessing the expertise of 70, 000 human editors: Knowledge-based feature generation for text categorization. Journal of Machine Learning Research 2007; 8(10): 2297–2345.

16.

Cai

Zhou

Liu

Zhao

. Large-scale question classification in cQA by leveraging Wikipedia semantic knowledge. In: Proceedings of the 20th ACM international conference on information and knowledge management, 2011, pp. 1321–1330.

17.

Sahami

Heilman

. A web-based kernel function for measuring the similarity of short text snippets. In: Proceedings of the 15th international conference on World Wide Web, 2006, pp. 377–386.

18.

Kennedy

Inkpen

. Sentiment classification of movie reviews using contextual valence shifters. Computational Intelligence 2006; 22(2): 110–125.

19.

Taboada

Brooke

Tofiloski

Voll

. Lexicon-based methods for sentiment analysis. Computational Linguistics 2011; 37(2): 267–307.

20.

Thelwall

Buckley

Paltoglou

Cai

. Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology 2010; 61(12): 2544–2558.

21.

Davidov

Tsur

Rappoport

. Enhanced sentiment learning using twitter hashtags and smileys. In: Proceedings of the 23rd international conference on computational linguistics, 2010, pp. 241–249.

22.

Gamon

Aue

Corston-Oliver

Ringger

. Pulse: Mining customer opinions from free text. In: Advances in intelligent data analysis VI, 2005, pp. 121–132. Berlin: Springer.

23.

Saif

Alani

. Semantic sentiment analysis of twitter. In: 11th International Semantic Web conference, Boston, MA, 11–15 November 2012.

24.

Montejo-Ráez

Martínez-Cámara

Martín-Valdivia

Ureña-López

. A knowledge-based approach for polarity classification in twitter. Journal of the Association for Information Science and Technology 2014; 65(2): 414–425.

25.

Blei

Jordan

. Latent dirichlet allocation. Journal of Machine Learning Research, 2003; 3: 993–1022.

26.

Griffiths

Steyvers

. Finding scientific topics. The National Academy of Sciences 2004; 101: 5228–5235.

27.

Lin

. Joint sentiment/topic model for sentiment analysis. In: Proceedings of the 18th ACM conference on information and knowledge management, 2009, pp. 375–384.

28.

Tang

Wei

Yang

Zhou

. Learning sentiment-specific word embedding for twitter sentiment classification. In: Proceedings of the 52nd annual meeting of the Association for Computational Linguistics, Baltimore, MD, 2014, pp. 1555–1565.

29.

Mikolov

Chen

Corrado

Dean

. Efficient estimation of word representations in vector space. In: Proceedings of workshop at ICLR, 2013.

30.

Baroni

Dinu

Kruszewski

. Don’t count, predict! A systematic comparison of context-counting vs context-predicting semantic vectors. In: Proceedings of the 52nd annual meeting of the Association for Computational Linguistics, Baltimore, MD, 2014, pp. 238–247.

31.

Wan

. Using bilingual knowledge and ensemble techniques for unsupervised Chinese sentiment analysis. In: Proceedings of the conference on empirical methods in natural language processing, 2008, pp. 553–561.

32.

Whitehead

Yaeger

. Sentiment mining using ensemble classification models. In: Innovations and Advances in Computer Sciences and Engineering. Berlin: Springer, 2010, pp. 509–514.

33.

Pang

Lee

. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In: Proceedings of the 43th annual meeting of the Association of Computational Linguistics, 2005.

34.

Maas

Daly

Pham

Huang

. Learning word vectors for sentiment analysis. In: Proceedings of the 49th annual meeting of the Association for Computational Linguistics, 2011.

35.

Levy

Goldberg

. Dependency-based word embeddings. In: Proceedings of the 52nd annual meeting of the Association for Computational Linguistics, Baltimore, MD, 2014, pp. 302–308.

36.

Mohammad

Kiritchenko

Zhu

. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In: Proceedings of the second joint conference on lexical and computational semantics (SEMSTAR’13), 2013.

37.

McAuley

Leskovec

. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In: Proceedings of the 22nd international conference on World Wide Web, 2013, pp. 897–908.