A cross-lingual sentiment topic model evolution over time

Abstract

Sentiment analysis in various languages has been a hot research topic with several applications. Most of the existing models have been reported to work well with widely used language. Were the lass directly applying these models to poor-quality corpora often leads to low results. Thus, to deal with these shortcoming we propose a cross-lingual sentiment topic model evolution over time (CLSTOT) which jointly models time with topic and sentiment. In CLSTOT, we consider the mapping between sentiment-aware topics under different cultures and analyze their evolution over time. The topic-specific sentiment is extracted using the entire data and not for each single document. As long as providing sentiment-topic, we can predict the timestamps for each test document by finding its most likely location over the timeline. This is achieved by using inference algorithm which is based on Gibbs Sampling. The experimental results on Chinese and English newsreader dataset; Chinese from SinaNews2, and English from Yahoo1, show that CLSTOT achieves significant improvement over the state-of-the-art.

Keywords

Cross-Lingual sentiment analysis Joint sentiment topic models Cross-Lingual-Time-aware topic-sentiment models

1. Introduction

Cross-lingual sentiment analysis is a hot research topic nowadays,and plays a significant role in creating links between articles written in different languages [36]. Sentiment analysis is the approach we used for extracting information from topics [23]. The topics are mostly based on subjective positions,therefore sentiment is usually issued about a topic. The sentiment analysis in topic is carried out at different levels which are sentence, document and aspect levels [20, 24, 6, 9, 33, 37, 26, 27]. Recently, probabilistic topic models that collectively detect aspects and sentiments have gained much success in determine topic evolution over time [14, 32, 19]. The probabilistic topic models are important for two reasons. Firstly, they provide us with an unsupervised way for discovering topics from documents. Secondly, they result in language models that can show how much words are related to topics and possibly lead to a sentiment. The consumer usually talks about aspects rather than things. Owing to the difficulties and challenges in dealing with such huge tasks, scientists argue that the idea of using sentiment analysis was useful especially in such circumstances[16, 15, 13]. While other state that it is essential to consider the aspects-based sentiment analysis classification by using time [1].

In recent years, developing topic models to explore topic evolution over time has been discussed in many topic applications[30]. Some studies only examined topic and its variation over time rather than discussing the joint modeling word co-occurrence and time [31]. While Sgonet always concentrates on an alternative generative process description of TOT, which is better suited to generate an unseen document, although starting Gibbs sampling using parameters from the previous time slice[28]. Similarly, The Time Mines system[29] for TDT task (a single topic in each document) constructs overview time-lines of a set of news stories. Dynamic Topic Model (DTM) considered as another extension to LDA [9]. is used to model time series data applications. In [26, 27, 14, 19, 16], some pilot studies on cross-lingual sentiment classification were done and had significant impact with various applications. However, most of these studies focused on extracting sentiments about topics in a static way and some of them ignored to use time in the textual data. The intent of these studies was to analyze and evaluate the contents of the topics, but they neglected the area of the topic-sentiment associations between the levels without progressing in the field of the scientific research[19, 31, 11].

Owing to the previous learning observations in this field, we intend to develop a novel CLSTOT approach to extract topic-sentiment associations from the text as well as their evolutions over time. Moreover, the idea based on the principle of our proposed model result in mainly 3 levels, which comprises the topics, topic sentiments, and topic-sentiment evolution over time. This allows our models to preform various functions. For example, the model extracts the hidden topical structures from a document collection, focusing on building the associations between topics and sentiments. Consequently, we provide an efficient tool for tracking and visualizing the strength of topic-sentiment correlation over time. Such information is reported and extracted at once and jointly without any post-processing which is very useful [7]. To address the above challenges, in this paper, we discuss the issue of joint topic modelling over time which mainly focuses on three proposed novel features that are not addressed by researchers in the field of CLSTOT. We first consider time, that is jointly modeled with topics and sentiments which allows capturing the evolution of the topics sentiment over time. Secondly, topic-specific sentiments are extracted from the whole data at once and not for every single document, this provides an overall sight of topic-sentiment correlations. Finally, no post-processing is needed to match similar topics under different sentiment polarities.

In general, the contributions of this study can be summarized as follows: firstly, time is jointly modeled with topics and sentiments providing a quantitative analysis of topic-sentiment evolution over time. Secondly, we specifically present an unsupervised cross-lingual topic sentiment evolution over time model, by integrating the state-of-the art model STOT into the above cross-lingual topic model framework, which can improve sentiment classification for target language by exploiting correspondences with the source language. Finally, through extensive experiments on real world datasets belonging to different domains and languages, and via comparison with existing state-of-art models, we demonstrate the effectiveness and validity of the proposed cross-lingual topic model framework and the CLSTOT model. Experimental results show that the proposed model can be successfully applied to practical sentiment classification applications in order to improve the accuracy of sentiment classification in target languages.

The rest of this paper is organized as follows: in Section 2 we introduce the related work which includes joint topic-sentiment modeling, topic evolution over time, and cross-lingual topic models. In Section 3, we present our proposed model Cross-lingual Sentiment Topic Model Evolution Over Time. Generative process of our model and parameter inference are shown in Section 4. Experimental settings and results are shown in Section 5. Finally, in Section 6, we conclude this paper and future work.

2. Related work

2.1 Joint topic-sentiment modeling

Topic and sentiment modeling tasks are entirely related to the extent that sentiments are usually issued about a topic at hand[7]. To examine the model topic-sentiment conjunction, much work has been carried out using the topic model-sentiment approaches: statistical models for discovering low-dimensional structures (topics) from text based on word co-occurrence patterns. These patterns are captured using latent or hidden variables. Earlier topic models, like LDA and PLSA [2, 12] have mainly focused on extracting similar topics, but more recently progress has been made to these models to capture other aspects of the text, such as sentiment. As an example, Joint Sentiment-Topic model (JST) [18, 17]. Was developed for dealing with topic extraction under different sentiment labels. This is performed by extending LDA with a new sentiment layer inserted before the topic layer. Thus, to generate a word for a document, a sentiment labels $s$ are drawn first, then a topic is drawn conditioned on $s$ . Reverse-JST[18] is a variant of JST where the order of sentiment and topic layers is inverted. Slightly different models also exist like Topic Sentiment Mixture (TSM) [21], sentiment-LDA[16], Aspect-and-Sentiment Unifica-tion Model (ASUM)[13], and Sentiment-Topic model with Decomposed Prior (STDP)[15]. All of the models were built based on LDA except TSM which based on PLSA.

2.2 Topic evolution over time

Here, the documents were collected over time such as online discussion forums, emails, news, etc. and consequently their content may evolve and change over time. Here, we focus on the quantitative evolution, i.e., the “amount” of data to discussing some topic at some time-stamp (T). Topic over time (TOT)[37] is an LDA-based model for quantitative topic evolution. For this, the meaning of the topic is assumed to be constant, its size is supposed to evolves and it is captured using a Beta distribution as presented by[33]. LDA model has been used to capture quantitative topic evolution over time by counting the number of documents associated with each topic at each timestamp. On the other hand, qualitative evolution focuses on all other aspects of a topic (word distribution, inter-topic correlation, vocabulary, etc.). In [13]. Alice proposed the Sentence-LDA (SLDA), a probabilistic generative model that assumes all words in a single sentence are generated from one aspect. Furthermore, the authors extended SLDA to aspect and sentiment Unification Model (ASUM), which incorporates aspect and sentiment together to model sentiments toward different feature and applications [9].

2.3 Cross-lingual topic models

A cross-lingual topic model (CL-LDA) opens up new opportunities for knowledge extraction research [35, 5]. Our aim is to find a bridge between languages. by using the existing models to combine different languages via parallel or comparable corpora, for example, CL-LDA[8]. Furthermore, we preserve connections between languages by sharing distribution over the latent topic of the bilingual document. Switch LDA[22] extends CL-LDA such as that it can control the proportions of languages in each multilingual topic by introducing a per-word switch variable. Symmetric Correspondence LDA proposed in[10] is a topic model that incorporates a hidden variable to control a pivot language. Since parallel or comparable corpora are relatively limited resources, this kind of models are confined in their applications.

CLLDA on unaligned texts may be applied to a broader class of corpora. For instance, in [34], they used to incorporate soft bilingual dictionary-based constraints into Probabilistic Latent Semantic Analysis (PLSA) so that it can extract shared Latent topics in text data of different languages. Boyd-Graber and Blei in 2009[3] developed the Multilingual Topic (MUTO) model to exploit matching across languages on term level to detect multilingual latent topics from unaligned texts. However, these models do not put into consideration the sentiment analysis[3].

Moreover, in early 2010 Boyd-Graber and Resnik proposed an LDA-based holistic model for multilingual sentiment analysis on the word level. Nevertheless, this model is supervised, while the model proposed in this paper is unsupervised. Besides still our model is also LDA-based on the sentence level which makes it more appropriate for aspect and sentiment detection from reviews[4].

3. Proposed method

3.1 Cross-lingual model-sentiment evolution over time

In this section, we present the CLSTOT model, our approach to modeling cross-lingual topic-sentiment associations as well as their quantitative evolution over time. Our scheme is motivated by the following remarks made on the earlier proposed models:

1.
Time is not jointly modeled with topics and sentiments.
2.
Topic-specific sentiments are estimated for each document separately.
3.
Similar topics from different sentiment polarities are not automatically matched.

To deal with these shortcoming, we suggest a novel topic model based on three main features: Firstly, time is jointly modeled with topics and sentiments providing a quantitative analysis of topic-sentiment evolution over time. Secondly, topic-specific sentiments are extracted from the whole data and not for every single document, which gives an overall view of topic-sentiment correlations. Lastly, no post-processing is needed to match topics under different sentiment polarities, because the same topic has multiple distributions over words; one for each sentiment polarity. We extend the CLLDA model by adding two new layers, S and T, to capture sentiment and time respectively as shown in Fig. 1.

Figure 1.
The graphical representation of CLTOT.

The proposed method is developed on the traditional topic modeling assumptions: each document in the learning collection is a mixture of topics (multinomial distribution over topics). Also, we assume that each topic has multiple facets; one for each sentiment polarity, and consequently multiple multinomial distributions over words. Moreover, we assume that the “force” of topic-sentiment association would evolve over time. Documents from the learning data must be annotated with time (e.g., creation date). Time is first discretized, and each document receives a discrete time-stamp label (e.g., day, month, and year). In the learning step, time modality is captured using the variable $t$ , and consequently, topic-sentiment evolution is captured using a multinomial distribution over timestamps $\psi$ . The notations used in the rest of this paper are presented in Tables 1 and 2.

Table 1
The notation for all models in the paper

Symbol Description

$D_{sc}/D_{tg}$ Number of the document in a corpus

$N_{d}$ Number of words in the document

$\alpha$ 10.8

$\beta^{sc}/\beta^{tg}$ The Dirichlet prior vector for $\phi^{sc}/\phi^{tg}$

$\mu$ The Beta prior vector for $\psi_{ts}$

$\gamma$ The Beta prior vector for $\pi_{t}$

$\gamma^{sc}/\gamma^{tg}$ The Beta prior vector for $\eta^{d}$

$W_{d,n}$ Words of the entire corpus including the source and target language

$S_{d,n}$ Sentiment

$T_{d,n}$ Time

$Z_{d,n}$ Topics

$\eta^{d}$ Multinomial distribution over $x_{d,n}$

$\theta_{d}$ Multinomial distribution over Topics $z_{d,n}$

$\pi_{d}$ Multinomial distribution over sentiment $s_{d,n}$

$\psi_{d}$ Multinomial distribution over time $T_{d,n}$

$\varphi^{sc}_{T,S}/\varphi^{tg}_{T,S}$ Multinomial distribution over words in the source and target languages

$y_{d,n}$ Intermediate variable assignment to all words (excluding the nth word in review d)

$x_{d,n}$ Language label assignment to all words (excluding the nth words in review d).

Table 2
The notation specified for the CLSTOT models

Symbol Description

$z/z\neg(d,n)$ The topic assignments to all words (excluding the nth word in review d)

$x/x\neg(d,n)$ The language label assignments to all word (excluding the nth word in review d)

$y/y\neg{(d,n)}$ The intermediate variable assignments to all words (excluding thenth word in review d)

$s/s\neg{(d,n)}$ The sentiment label assignments to all word (excluding the nth word in review d)

$t/t\neg{(d,n)}$ The time label assignments to all words (excluding the nth word in review d)

$C^{d}(C^{d}_{k})$ The number of the word (assigned topic k) in review d

$c^{d}_{sc/tg}$ The time that any word is assigned language label sc/tg and review $d$

$C^{sc/tg,k}$ The time that any word assigned language label $sc/tg$ and topic $k$

$C^{d}(C^{d}_{s})$ The number of words (assigned sentiment label $S$ ) in review $d$

3.2 Generative process of cross-lingual sentiment topic model evolution over time

Symbol	Description
$D_{sc}/D_{tg}$	Number of the document in a corpus
$N_{d}$	Number of words in the document
$\alpha$	10.8
$\beta^{sc}/\beta^{tg}$	The Dirichlet prior vector for $\phi^{sc}/\phi^{tg}$
$\mu$	The Beta prior vector for $\psi_{ts}$
$\gamma$	The Beta prior vector for $\pi_{t}$
$\gamma^{sc}/\gamma^{tg}$	The Beta prior vector for $\eta^{d}$
$W_{d,n}$	Words of the entire corpus including the source and target language
$S_{d,n}$	Sentiment
$T_{d,n}$	Time
$Z_{d,n}$	Topics
$\eta^{d}$	Multinomial distribution over $x_{d,n}$
$\theta_{d}$	Multinomial distribution over Topics $z_{d,n}$
$\pi_{d}$	Multinomial distribution over sentiment $s_{d,n}$
$\psi_{d}$	Multinomial distribution over time $T_{d,n}$
$\varphi^{sc}_{T,S}/\varphi^{tg}_{T,S}$	Multinomial distribution over words in the source and target languages
$y_{d,n}$	Intermediate variable assignment to all words (excluding the nth word in review d)
$x_{d,n}$	Language label assignment to all words (excluding the nth words in review d).

Symbol	Description
$z/z\neg(d,n)$	The topic assignments to all words (excluding the nth word in review d)
$x/x\neg(d,n)$	The language label assignments to all word (excluding the nth word in review d)
$y/y\neg{(d,n)}$	The intermediate variable assignments to all words (excluding thenth word in review d)
$s/s\neg{(d,n)}$	The sentiment label assignments to all word (excluding the nth word in review d)
$t/t\neg{(d,n)}$	The time label assignments to all words (excluding the nth word in review d)
$C^{d}(C^{d}_{k})$	The number of the word (assigned topic k) in review d
$c^{d}_{sc/tg}$	The time that any word is assigned language label sc/tg and review $d$
$C^{sc/tg,k}$	The time that any word assigned language label $sc/tg$ and topic $k$
$C^{d}(C^{d}_{s})$	The number of words (assigned sentiment label $S$ ) in review $d$

For each Topic $z$ :

1.
Draw a multinomial distribution of the source language:

–
$\phi^{sc,z}\sim\textit{dir}(\beta^{sc})$
–
$\psi_{T,S}\sim\textit{dir}(\mu)$
–
$\pi_{T}\sim\textit{dir}(\gamma)$

2.
Draw a multinomial distribution of the target language:

–
$\phi^{tg,z}\sim\textit{dir}(\beta^{tg})$
–
$\psi_{T,S}\sim\textit{dir}(\mu)$
–
$\pi_{T}\sim\textit{Dir}(\gamma)$

3.
For each document $d$ in the corpus of the Source language:

–
Draw a distribution $\theta^{d}\sim dir(\alpha)$
–
Draw a binomial distribution, $\eta^{d}\sim\textit{Beta}(\gamma^{sc})$
–
Draw a sentiment label $\S_{d,n}\sim\pi_{T},z_{d,n}$
–
Draw a time label $T_{d,n}\sim Z_{d,n},S_{d,n},\psi_{T,S}$
–
Draw a topic $Z_{d,n}\sim\textit{dir}(\textit{theta})$
–
Draw the language label $x_{d,n}\sim\eta^{d}$

*
If $x_{d,n}=sc$ , drawn intermediate word in the Source language $y_{d,n}\sim\phi^{sc,s_{d,n},z_{d,n}}$ ; generate a word directly: $w_{d,n}=y_{d,n}$
*
If $x_{d,n}=tg$ , draw an intermediate word in the target language $y_{d,n}\sim\phi^{tg,s_{d,n},z_{d,n}}$ , generate a word in translate form $y_{d,n}$ according to the translation probability $T^{sc,w_{d,n}}_{tg,y_{d,n}}$ ;

4.
For each document $d$ in the corpus of the Source language:

–
Draw a distribution $\theta^{d}\sim dir(\alpha)$
–
Draw a binomial distribution, $\eta^{d}\sim Beta(\gamma)$
–
Draw a sentiment label $\S_{d,n}\sim\pi_{T},z_{d,n}$
–
Draw a time label $T_{d,n}\sim Z_{d,n},S_{d,n},\psi_{T,S}$
–
Draw a topic $Z_{d,n}\sim dir(theta)$
–
Draw the language label $x_{d,n}\sim\eta^{d}$

*
If $x_{d,n}=sc$ , drawn intermediate word in the Source $y_{d,n}\sim\phi^{sc,s_{d,n},z_{d,n}}$ ; generate a word by translating from $y_{d,n}$ according to the translation probability $T^{tg,w_{d,n}}_{T_{sc},y_{d,n}}$
*
If $x_{d,n}=tg$ , draw an intermediate word in the target language $y_{d,n}\sim\phi^{tg,s_{d,n},z_{d,n}}$ ; generate a word directly: $w_{d,n}=y_{d,n}$

3.3 Parameters inference

Like LDA, exact inference is intractable in our model. Hence approximate estimation approaches, such as Gibbs Sampling [26], are utilized to solve the problem. Gibbs Sampling is a particular case of Markov Chain Monte Carlo (MCMC)[9], which is a relatively simple algorithm of approximating inference for our models. Due to space limitations, only the final formulas are described here.

3.3.1 Joint distribution

The joint probability of sentiment topic evolution over time is calculated using Bayes conditional independence rule, the joint probability of words, topics, sentiments, and timestamps can be calculated as follows:

The first term is obtained by integrating over $\varphi$ .

$\displaystyle p(w|s,z,\beta)=\left(\frac{\Gamma(V\beta)}{\Gamma(\beta)^{v}}% \right)^{t.s}\prod_{j}\prod_{k}\frac{\prod_{i}\Gamma(n_{i,j,k}+\beta)}{\Gamma(% n_{j,k}+V\beta)}$ (2)

Where $\Gamma$ denotes Gamma function. Subscript $i$ , $j$ , $k$ , $h$ are used to loop over topics, sentiments, and time-stamps respectively. The second term of the Eq. (1) is obtained by integrating over $\Psi$ .

$\displaystyle p(t|s,z,\mu)=\left(\frac{\gamma(H\mu)}{\gamma(\mu)^{H}}\right)^{% t.s}\prod_{j}\prod_{k}\frac{\prod_{h}\Gamma(n_{i,k,h}+\mu)}{\Gamma(n_{j,k}+H_{% \mu})}$ (3)

The remaining terms of Eq. (1) are obtained in the same way by integrating over $\pi$ , $\Theta$ respectively

$\displaystyle p(s|z,\gamma)=\left(\frac{\Gamma(\sum_{k}{\gamma}{k})}{\prod_{k}% \Gamma({\gamma}{k})}\right)^{t}\prod_{j}\frac{\prod_{k}\Gamma(n_{j,k}+{\gamma}% {k})}{\Gamma(n_{j}+\sum_{k}{\gamma}{k})}$ (4) $\displaystyle p(z|\alpha)=\left(\frac{\Gamma(\sum_{j}{\alpha}{j})}{\prod_{j}% \Gamma({\alpha}{j})}\right)^{D}\prod_{d}\frac{\prod_{j}\Gamma(n_{d,k}+{\alpha}% {j})}{\Gamma(n_{d}+\sum_{j}{\alpha}{j})}$ (5)

3.3.2 Posterior distribution inference

For the source language: If $X_{d,n}=sc$

$\displaystyle p(Z_{d,n}=k,S_{d,n}=s,Y_{d,n}=W_{d,n},X_{d,n}=sc|w,t,Z_{(d,n)},S% _{(d,n)},X_{(d,n)},Y_{(d,n)})\propto\frac{C_{sc}^{d}+\gamma_{sc}^{sc}}{C^{d}+% \gamma_{tg}^{sc}+\gamma_{sc}^{sc}}\cdot\frac{C_{k}^{d}+{\alpha k}}{C^{d}+\sum_% {k^{\prime}}{\alpha k^{\prime}}}\cdot\frac{C_{s}^{t}+{\gamma}}{C^{t}+\sum_{s^{% \prime}\in\{ps,ng\}}{\gamma s^{\prime}}}\cdot\frac{C_{t_{d,n}}^{sc,s,k}+\mu_{t% _{d,n}}^{sc}}{C^{sc,s,k}+\sum_{t^{\prime}\in V_{sc}}{\mu_{t^{\prime}}^{sc}}}% \cdot\frac{C_{W_{d,n}}^{sc,s,k}+\beta_{W_{d,n}}^{sc}}{C^{sc,s,k}+\sum_{w^{% \prime}\in V_{sc}}{\beta_{w^{\prime}}^{sc}}}$ (6)

Moreover, if $x_{d,n}=tg$ and the translation word of $y_{d,n}$ is $t$ ,

$\displaystyle p(Z_{d,n}=k,S_{d,n}=s,Y_{d,n}=t,X_{d,n}=tg|w,t,Z_{\neg(d,n)},S_{% \neg(d,n)},X_{\neg(d,n)},Y_{\neg(d,n)})\propto\frac{C_{tg}^{d}+\gamma_{tg}^{sc% }}{C^{d}+\gamma_{sc}^{sc}+\gamma_{tg}^{sc}}\cdot\frac{C_{k}^{d}+{\alpha}{k}}{C% ^{d}+\sum_{k^{\prime}}{\alpha}{k^{\prime}}}\cdot\frac{C_{s}^{t}+{\gamma}{s}}{C% ^{t}+\sum_{s^{\prime}\in\{ps,ng\}}{\gamma}{s^{\prime}}}\cdot\frac{C_{t_{d,n}}^% {sc,s,k}+\mu_{t_{d,n}}^{sc}}{C^{sc,s,k}+\sum_{t^{\prime}\in V_{sc}}{\mu_{t^{% \prime}}^{sc}}}\cdot{T_{tg,t}^{sc,w_{d,n}}}\frac{C_{W_{d,n}}^{tg,s,k}+\beta_{W% _{d,n}}^{tg}}{C^{tg,s,k}+\sum_{w^{\prime}\in V_{tg}}{\beta_{w^{\prime}}^{tg}}}$ (7)

For the target language: $X_{d,n}=tg$

$\displaystyle p(Z_{d,n}=k,S_{d,n}=s,Y_{d,n}=W_{d,n},X_{d,n}=tg|w,t,Z_{(d,n)},S% _{(d,n)},X_{(d,n)},Y_{(d,n)})\propto\frac{C_{tg}^{d}+\gamma_{tg}^{tg}}{C^{d}+% \gamma_{sc}^{tg}+\gamma_{sc}^{tg}}\cdot\frac{C_{k}^{d}+{\alpha}{k}}{C^{d}+\sum% _{k^{\prime}}{\alpha}{k^{\prime}}}\cdot\frac{C_{s}^{t}+{\gamma}{s}}{C^{t}+\sum% _{s^{\prime}\in\{ps,ng\}}{\gamma}{s^{\prime}}}\cdot\frac{C_{t_{d,n}}^{tg,s,k}+% \mu_{t_{d,n}}^{tg}}{C^{tg,s,k}+\sum_{t^{\prime}\in V_{sc}}{\mu_{t^{\prime}}^{% tg}}}\cdot\frac{C_{W_{d,n}}^{tg,s,k}+\beta_{W_{d,n}}^{tg}}{C^{tg,s,k}+\sum_{w^% {\prime}\in V_{sc}}{\beta_{w^{\prime}}^{tg}}}$ (8)

Moreover, if $X_{d,n}=sc$ and the translation word of $Y_{d,n}$ is $t$

$\displaystyle p(Z_{d,n}=k,S_{d,n}=s,Y_{d,n}=t,X_{d,n}=sc|w,t,Z_{(d,n)},S_{(d,n% )},X_{(d,n)},Y_{(d,n)})\propto\frac{C_{sc}^{d}+\gamma_{sc}^{tg}}{C^{d}+\gamma_% {sc}^{tg}+\gamma_{tg}^{tg}}\cdot\frac{C_{k}^{d}+{\alpha}{k}}{C^{d}+\sum_{k^{% \prime}}{\alpha}{k^{\prime}}}\cdot\frac{C_{s}^{t}+{\gamma}{s}}{C^{t}+\sum_{s^{% \prime}\in\{ps,ng\}}{\gamma}{s^{\prime}}}\cdot\frac{C_{t_{d,n}}^{sc,s,k}+\mu_{% t_{d,n}}^{sc}}{C^{sc,s,k}+\sum_{t^{\prime}\in V_{sc}}{\mu_{t^{\prime}}^{sc}}}% \cdot{T_{sc,t}^{tg,w_{d,n}}}\cdot\frac{C_{W_{d,n}}^{sc,s,k}+\beta_{W_{d,n}}^{% sc}}{C^{sc,s,k}+\sum_{w^{\prime}\in V_{tg}}{\beta_{w^{\prime}}^{sc}}}$ (9)

Therefore, the approximate probability of sentiment $S$ in the topic $t$ is:

$\displaystyle\pi_{s}^{T}=\frac{C_{s}^{t}+{\gamma}{s}}{C^{t}+\sum_{s^{\prime}% \in\{ps,ng\}}{\gamma}{s^{\prime}}}\cdot$ (10)

Moreover, the approximate probability of topic $k$ in review $d$ is:

$\displaystyle\theta_{k}^{d}=\frac{C_{k}^{d}+{\alpha}{k}}{C^{d}+\sum_{k^{\prime% }}{\alpha}{k^{\prime}}}.$ (11)

The approximate probability of time $t$ in sentiment $S$ coupled with the topic $K$ in the source language can be obtained as follows:

$\displaystyle\psi_{t_{d,n}}^{sc,s,k}=\frac{C_{t_{d,n}}^{sc,s,k}+\mu_{t_{d,n}}^% {sc}}{C^{sc,s,k}+\sum_{t^{\prime}\in V_{x}}{\mu_{t^{\prime}}^{sc}}}.$ (12)

Where $S\in\{ps,ng\}$ .

Lastly, the approximate probability of the word $W$ in sentiment $S$ with a topic $K$ for language $X$ can be obtained as follows:

$\displaystyle\Phi_{w_{d,n}}^{x,s,z}=\frac{C_{w_{d,n}}^{x,s,k}+\beta_{w_{d,n}}^% {x}}{C^{x,s,k}+\sum_{w^{\prime}\in v_{x}}{\beta_{w^{\prime}}^{x}}}$ (13)

Where $S\in\{ps,ng\}$ and $X\in\{sc,tg\}$ . where $S\in\{ps,ng\}$ sentiment positive and negative, $X\in\{sc,tg\}$ language source and target language.

4. Experiment

In this section, we validate the proposed CLSTOT model via extensive experiments. Specifically, we first examine its validity on discovering CLSTOT, then we observe the effectiveness of its cross-lingual mechanism to determined the effect of the bilingual dictionary on its behavior. Next, we investigate its performance by applying it to a practical sentiment classification task and quantitatively comparing it with four state-of-the-art baselines.

4.1 Experimental setting

4.1.1 Data set and preprocessing

We managed to compile our data set via collecting English and Chinese observations from different regions reflecting different cultures for some significant events as depicted in Table 3. The English reader comments are collected from Yahoo1,and the Chinese reader comments are collected from sina News2. We first remove news reader comments whose length is less than five words. We then remove the punctuation and the stop words. For English comments, we also stem each word to its root from using Porter stemmer (Porter, 1980). For the Chinese reader comments, we use the Jieba package3 to segment and remove Chinese stop words. We utilize an English-Chinese dictionary from MDBG4.

Table 3
The statistics for the data set

Time	Title	English comments	Chinese comments
1	MH370flight accident	8608	5223
2	ISIS in Iraq	6341	3263
3	Ebola occurs	2974	1622
4	Taiwan Crashed Plane	6780	2648
5	iphone6 publish	5837	4352
6	Shooting of Michael Brown	17547	3693
7	Charlie Hebdo shooting	1845	551

4.1.2 Parameters settings

To optimize the number of topics, we empirically ran the models with four values of $k=$ 10, 15, 20 and 100 in sentiment 140 and ran the models with three values of $k=$ 10, 15, 20 in News reader comments.(In news reader comments, these News only contain a small number of electronic product related topics). In our model, we only selected symmetric Dirichlet prior vectors as it is empirically done in JST and ASUM. For JST, ASUM, the values were $\alpha=50/(k)$ , $\beta=$ 0.01 and. $\gamma=$ 0.01. For CLSTOT we set $\alpha=$ 0.5, $\gamma=$ 0.001, $\mu=$ 0.001 and $\beta=$ 0.01. These LDA-based models are not sensitive to the hyper parameters[25]. In all the methods, Gibbs sampling was run for 1.000 iterations with 200 burn-in periods.

4.1.3 Baselines

For performance comparison, we adopted four representative baselines which are CLJAS (Cross-lingual joint Aspect Sentiment), JSTM (Joint Sentiment-Topic Model), ASUM (Aspect Sentiment Unification Model), and TS-LDA (Topic Sentiment Latent Dirichlet Allocation). We chose these models because they are considered as baselines, and they also state-of-the-art. However, they all are monolingual models. CLJST is an extended model for JST, while CLSTOT is an extended model for STOT. Thus, we compare CLJST with JST and CLASUM with ASUM respectively. Through comparing the cross-lingual model with the corresponding monolingual model, we can validate the effectiveness of the proposed cross-lingual mechanism of our model.

4.2 Topic coherence

CLSTOT can also extract coherent sentiment-aware topics from news reader generated post collection documents, and evaluate the efficiency of topic and sentiment captured by our models. In order to carry out a quantitative evaluation of topic coherence, we used an automated metric proposed in[25], which is shown in Eq. (14), where topic coherence; denoted as $D(\nu)$ , is the document frequency of the word $\nu$ , $D(\nu,\nu^{\prime})$ is the co-document frequency of the word and $(\nu^{k}_{1},\cdots,\nu^{k}_{T})$ is a list of the $T$ most probable words in the topic. The fundamental idea of the coherence score is that if a word pair is related to the same topic, they will frequently co-occur in the corpus. To quantify the overall coherence of the discovered topic, the average coherence score $\frac{1}{k}\sum^{K}c(z_{k};v^{(z_{k})})$ was utilized. We conducted and evaluated the topic extraction experiments on the news of reader comments. Here we also compared CLSTOT. In this collection, we set the number of topics $k=$ 10, 20 for all the methods both in the source language and target language, and the results are listed in Tables 4 and 5. From the coherent topic result, it is clear that aggregating news in time slices directly leads to significant improvement of topic coherence. Note that for topic coherence, CLSTOT outperforms CLJST.

$\displaystyle c\left(t;\mathop{V}^{\left(t\right)}\right)=\sum\nolimits_{m=2}^% {M}{\sum\nolimits_{l=1}^{m-1}{\log\frac{\left(\mathop{v}_{m}^{\left(t\right)},% \mathop{v}_{l}^{\left(t\right)}\right)+1}{D\left(\mathop{v}_{l}^{\left(t\right% )}\right)}}}$ (14)

Where $V^{(t)}=(\nu^{t}_{1},\cdots,\nu^{t}_{M})$ is a list of the M most probable words in topic t. A smoothing count of 1 is included to avoid taking the logarithm of zero.

Table 4

Average coherence score on the top M words in the K topic discovered on news reader of News reader comments in the source language

Topic coherence	Source-K $=$ 10-M $=$ 10	Source-K $=$ 10-M $=$ 20	Source-K $=$ 15-M A $=$ 10	Source-K $=$ 15-M $=$ 20
CLJAS	$-$ 244.68	$-$ 1128.76	$-$ 245.68	$-$ 1130.76
JSTM	$-$ 245.79	$-$ 1138.43	$-$ 246.79	$-$ 1139.43
ASUM	$-$ 242.97	$-$ 1133.44	$-$ 243.97	$-$ 1135.44
TS	$-$ 243.23	$-$ 1138.56	$-$ 245.23	$-$ 1139.56
CLSTOT	$-$ 242.04	$-$ 1047.57	$-$ 243.04	$-$ 1048.57

Table 5

Average coherence score on the top M words in the K topic discovered on news reader of News reader comments in the target language

Topic coherence	target-K $=$ 10-M $=$ 10	target-K $=$ 10-M $=$ 20	target-K $=$ 15-M $=$ 10	target-K $=$ 15-M $=$ 20
CLJAS	$-$ 246.68	$-$ 1138.23	$-$ 249.28	$-$ 1140.27
JSTM	$-$ 246.79	$-$ 1142.01	$-$ 243.23	$-$ 1144.21
ASUM	$-$ 244.97	$-$ 1131.96	$-$ 244.17	$-$ 1130.2
TS	$-$ 244.23	$-$ 1141.69	$-$ 246.51	$-$ 1145.45
CLSTOT	$-$ 244.04	$-$ 1069.44	$-$ 242.08	$-$ 1071.36

Figure 2.

Proportion of Coherent topics generated by each model in $K=$ 10, 20, for Source positive and negative.

4.3 Human evaluation

As our objective is to discover more coherent sentiment topics over time, we chose to evaluate the topics manually, which is based on human judgments. Due to the fact that without enough knowledge, the an notation will not be credible and following [25], we asked two human judges who are familiar with shared knowledge and skilled in looking up the test news dataset, to annotate the discovered sentiment-aware topics evolution over time manually. For this annotation to be reliable, we labeled the generated topics by all the baseline models and our proposed model at the tenth (10) learning iteration.

4.3.1 Topic labeling

Furthermore, we asked the judges to label each sentiment-aware topic evolution over time as coherent or incoherent. Each sentiment-aware topic is represented as a list of the twenty (20) most likely words in the word distribution $\varphi$ of the topic. Here, they annotated a sentiment-aware topic as coherent when at least half of the top twenty (20) words were related to the same semantic-coherent concept (e.g., an event of News Reader Comment), and the sentiment polarities of the words are accurate, others were calculated as incoherent. The results are shown in Figs 2 and 3.

Figure 3.

Proportion of Coherent topics generated by each model in $K=$ 10, 20, for Target Positive and negative.

4.3.2 Word labeling

We chose coherent sentiment evolution topic over time which was judged before and asked judges to label each word of the top twenty (20) words among these coherent topics.

When a word was related to the central semantic-coherent concept which represents the topic labeling, the judges had known the concept for each topic and the overall sentiment of the topic. The process of labeling words of each sentiment evolution over time is shown in Figs 4 and 5. The previous figures shows the topic precision generated by each model. When $K=$ 10, 20 our model performs better for labeling words of each sentiment topic evolution blei2003latentn over time. Thus, we can conclude that the more time we give to coherence topic the better our CLSTOT model achieves in discovering word labeling in comparison to other models.

Figure 4.

Average Precision @10, 20 for sources positive and negative.

4.4 Qualitative analysis

To explore the quality of topics discovered by CLSTOT, we randomly chose some topics for visualization. We selected six topics, i.e., three real topics and three negative topics. For each topic, we chose the top 10 words, which mostly represent the topic. In Table 6, we present the top words of the selected topics. The three topics with a positive sentiment label, and the three with negative sentiment label topics are respectively shown in Table 6. The six topics are quite explicit and coherent, where each of them tried to capture the topic of a news reader comment. In terms of topic, our model CLSTOT achieves reliable results. Moreover, the CLSTOT model can obtain bilingually aligned aspects due to the feature of topic distribution sharing. This mechanism enables us to develop the CLSTOT on the data in the target language by leveraging the precious resources in the source language. Analysis of these topics shows that CLSTOT can indeed discover coherent sentiment evolution topic over time.

Table 6
Example of extracted positive negative [English and Chinese] by CLSTOT

Extracted positive negative English and Chinese
Positive–EN	fight,treatment,effect, love, diseases, help,health, thank, life, supply
Negative–EN	mistake, aircraft, wreckage, fail, lawsuit, grieved, terrible, conspiracy, search, death
Positive–CN	UTF8gbsn中国, 坚守, 国家, 提供, 援助, 抗击, 疫苗, 防范, 埃博拉, 希望
Negative–CN	UTF8gbsn受伤, 最高, 疑似, 大众, 劫机者, 官场, 明确, 此案, 判刑, 拘留

Figure 5.

Average Precision @10, 20 for target positive and negative

5. Cross-lingual mechanism

This experiment aims to estimate the validity of the proposed cross-lingual mechanism. For this reason, we calculated the perplexity [2] of the test set under different $\gamma_{sc}^{tg}$ , which controls the degree of information transfer between the source and target language, i.e. the point that the knowledge learned from the dataset in the source language is used to learn knowledge for the dataset in the target language. According to [2], the lower the perplexity, is the better performance topic model will have. In this experiment, for each domain, 2000 reviews were used for model training and the rest for inference. Table 4 presents the perplexity values of the test set in Chinese under different domains. As seen in Table 4, with the increase of $\gamma_{sc}^{tg}$ the perplexity under different domains radically decreases in a monotonic manner. Therefore, we can infer that the performance of the model was improved by the introduction of the cross-lingual mechanism. The improvement mainly benefits from the knowledge transferred from the source language data. According to the experimental results, we can observe that within a specific range, the more significant the $\gamma_{sc}^{tg}$ , the more improvement it can be achieved. However, $\gamma_{sc}^{tg}$ can not be too large or it will cause the model to learn knowledge almost entirely from the source language and neglect the characteristics of the target language. We further evaluated the proposed cross-lingual framework on other languages. Taking English as the source language, Table 5 presents the perplexity values of the test sets in different languages under the total domain. It can be observed from Table 5 that with the increase of $\gamma_{sc}^{tg}$ , the perplexity monotonically decreases under different languages. Therefore, we can claim that the proposed cross-lingual mechanism is useful for different domains and for different languages.

6. Conclusions

In this paper, we proposed a novel unsupervised CLSTOT for sentiment analysis. The CLSTOT model requires neither similar corpora nor labeled sentiment reviews. Through the cross-lingual mechanism, our model can significantly improve sentiment analysis in a target language by leveraging data and knowledge available in the source language. Mainly, CLSTOT can obtain bilingually aligned news reviews from the same topic distribution. Experimental results demonstrated that compared to similar unsupervised baselines, CLSTOT can significantly improve the supervised baseline. CLSTOT also exhibits comparable performance which has convincingly validated its effectiveness and highlighted its value. Moreover, we used seven different topics. We have demonstrated that CLSTOT outperforms four other state-of-the art models in extracting accurate topic-sentiment time associations. We further illustrated the efficiency of the CLSTOT model for homogeneous topics, associated sentiments and their evolution over time using both academic and real-world datasets collected from the web. Moreover, analyzing topic-sentiment evolution on real-world data allowed us to discover and gain insight into hidden phenomena which was impossible with previous models. This last feature generates a wide range of promising applications. For future research , we will try to adapt this strategy to additional languages. We will also consider expanding our model by using word embedding.

References

Blei

D.M.

and Lafferty

J.D.

, Dynamic topic models, in Proceedings of the 23rd international conference on Machine learning, ACM, 2006, pp. 113–120.

Blei

D.M.

A.Y.

and Jordan

M.I.

, Latent dirichlet allocation, Journal of Machine Learning Research 3 (2003), 993–1022.

Boyd-Graber

and Blei

D.M.

, Multilingual topic models for unaligned text, in Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, AUAI Press, 2009, pp. pp. 75–82.

Boyd-Graber

and Resnik

, Holistic sentiment analysis across languages: Multilingual supervised latent dirichlet allocation, in Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2010, pp. 45–55.

Chen

and Li

, Modeling language discrepancy for cross-lingual sentiment analysis, in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, ACM, 2017, pp. 117–126.

Chen

and Parsons

, A sentence-level sparse gamma topic model for sentiment analysis, in Advances in Artificial Intelligence: 31st Canadian Conference on Artificial Intelligence, Canadian AI 2018, Toronto, ON, Canada, May 8–11, 2018, Proceedings 31, Springer, 2018, pp. 316–321.

Dermouche

Velcin

Khouas

and Loudcher

, A joint model for topic-sentiment evolution over time, in 2014 IEEE International Conference on Data Mining (ICDM), IEEE, 2014.

Erosheva

Fienberg

and Lafferty

, Mixed-membership models of scientific publications, Proceedings of the National Academy of Sciences 101 (2004), 5220–5227.

Farra

Challita

Assi

R.A.

and Hajj

, Sentence-level and document-level sentiment mining for arabic texts, in Data Mining Workshops (ICDMW), 2010 IEEE International Conference on, IEEE, 2010, pp. 1114–1119.

10.

Fukumasu

Eguchi

and Xing

E.P.

, Symmetric correspondence topic models for multilingual text analysis, in Advances in Neural Information Processing Systems, 2012, pp. 1286–1294.

11.

Griffiths

T.L.

and Steyvers

, Finding scientific topics, Proceedings of the National academy of Sciences 101 (2004), 5228–5235.

12.

Hofmann

, Probabilistic latent semantic analysis, in Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence, Morgan Kaufmann Publishers Inc., 1999, pp. 289–296.

13.

and Oh

A.H.

, Aspect and sentiment unification model for online review analysis, in Proceedings of the fourth ACM international conference on Web search and data mining, ACM, 2011, pp. 815–824.

14.

Kaminka

et al., A joint model for sentiment-aware topic detection on social media, in ECAI 2016: 22nd European Conference on Artificial Intelligence, 29 August-2 September 2016, The Hague, The Netherlands-Including Prestigious Applications of Artificial Intelligence (PAIS 2016), vol. 285, IOS Press, 2016, p. 338.

15.

Zhang

Sun

J.-T.

and Chen

, Sentiment topic model with decomposed prior, in Proceedings of the 2013 SIAM International Conference on Data Mining, SIAM, 2013, pp. 767–775.

16.

Huang

and Zhu

, Sentiment analysis with global topics and local dependency, in AAAI, vol. 10, 2010, pp. 1371–1376.

17.

Lin

and He

, Joint sentiment/topic model for sentiment analysis, in Proceedings of the 18th ACM conference on Information and knowledge management, ACM, 2009, pp. 375–384.

18.

Lin

Everson

and Ruger

, Weakly supervised joint sentiment-topic detection from text, IEEE Transactions on Knowledge and Data Engineering 24 (2012), 1134–1145.

19.

Lin

Jin

Wang

Cheng

Wang

and Meng

, An unsupervised cross-lingual topic model framework for sentiment classification, IEEE/ACM Transactions on Audio, Speech, and Language Processing 24 (2016), 432–444.

20.

Ling

Xiang

Dyer

Black

and Trancoso

, Microblogs as parallel corpora, in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, 2013, pp. 176–186.

21.

Mei

Ling

Wondra

and Zhai

, Topic sentiment mixture: modeling facets and opinions in weblogs, in Proceedings of the 16th international conference on World Wide Web, ACM, 2007, pp. 171–180.

22.

Newman

Chemudugunta

and Smyth

, Statistical entity-topic models, in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, 2006, pp. 680–686.

23.

Nguyen

T.H.

and Shirai

, Topic modeling based sentiment analysis on social media for stock market prediction, in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), vol. 1, 2015, pp. 1354–1364.

24.

Pham

D.-H.

and Le

A.-C.

, Learning multiple layers of knowledge representation for aspect based sentiment analysis, Data & Knowledge Engineering 114 (2018), 26–39.

25.

Ramirez

E.H.

Brena

Magatti

and Stella

, Topic model validation, Neurocomputing 76 (2012), 125–133.

26.

Rohit

and Jagdale

, An approach to sentiment analysis, (2018).

27.

Schouten

and Frasincar

, Survey on aspect-level sentiment analysis, IEEE Transactions on Knowledge & Data Engineering (2016), p. 1.

28.

Song

Lin

C.-Y.

Tseng

B.L.

and Sun

M.-T.

, Modeling and predicting personal information dissemination behavior, in Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, ACM, 2005, pp. 479–488.

29.

Swan

and Jensen

, Constructing timelines with statistical models of word usage, in Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2000.

30.

Wang

Blei

and Heckerman

, Continuous time dynamic topic models, arXiv preprint arXiv:12063298, (2012).

31.

Wang

and McCallum

, Topics over time: a non-markov continuous-time model of topical trends, in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, 2006, pp. 424–433.

32.

Xiong

Wang

and Wang

, A short text sentiment-topic model for product reviews, Neurocomputing 297 (2018), 94–102.

33.

Yessenalina

Yue

and Cardie

, Multi-level structured models for document-level sentiment classification, in Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2010, pp. 1046–1056.

34.

Zhang

Mei

and Zhai

, Cross-lingual latent topic extraction, in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2010, pp. 1128–1137.

35.

Zhang

and Rettinger

, X-lisa: cross-lingual semantic annotation, Proceedings of the VLDB Endowment 7 (2014), 1693–1696.

36.

Zhang

Liu

Zhao

et al., Cross lingual entity linking with bilingual topic model, in IJCAI, 2013, pp. 2218–2224.

37.

Zhou

and Song

, Aspect-level sentiment analysis based on a generalized probabilistic topic and syntax model, in FLAIRS Conference, 2015, pp. 241–245.

A cross-lingual sentiment topic model evolution over time

Abstract

Keywords

1. Introduction

2. Related work

2.1 Joint topic-sentiment modeling

2.2 Topic evolution over time

2.3 Cross-lingual topic models

3. Proposed method

3.1 Cross-lingual model-sentiment evolution over time

3.3.1 Joint distribution

4.1 Experimental setting

4.1.1 Data set and preprocessing

Table 3 The statistics for the data set

4.1.3 Baselines

4.2 Topic coherence

4.3.1 Topic labeling

Table 6 Example of extracted positive negative [English and Chinese] by CLSTOT

6. Conclusions

References

Table 3
The statistics for the data set

Table 6
Example of extracted positive negative [English and Chinese] by CLSTOT