Topic mining of tourist attractions based on a seasonal context aware LDA model

3.1 LDA Model

Latent Dirichlet Allocation (LDA) [6] is a generative probabilistic model that tries to capture the implicit topic structure from a collection of documents. It specifies a probabilistic procedure that depicts how the words in documents are generated. The basic idea is that each document is represented by a specific topic distribution and each topic is characterized by a probability distribution over words. The LDA model is a three-level hierarchical Bayesian model, where topics are associated with documents and words are associated with topics. There is a clear hierarchy followed by the document layer, topic layer and word layer.

1.

Word layer: A word is the basic unit of discrete data, defined to be an item from a vocabulary of size V denoted by V = { w 1 , w 2 , … , w V } .

2.

Topic layer: A topic z k , k ∈ { 1 , 2 , … , K } is associated with a multinomial φ k over the V -word vocabulary and can be denoted by φ k = ⟨ p k , 1 , p k , 2 , … , p k , V ⟩ , where p k , j refers to the probability that word w j is generated from topic z k .

3.

Document layer: A document is a sequence of N m words denoted by d m = { w 1 , w 2 , … , w N m } . Likewise, each document is associated with a multinomial θ m over K topics and can be represented as θ m = ⟨ p m , 1 , p m , 2 , … , p m , K ⟩ , where p m , z refers to the probability that topic z is generated from document d m .

Figure 2a shows the graphical model representation of the LDA. In this graphical notation, nodes are random variables and arrows indicate conditional dependencies between two variables. The shaded and unshaded circles represent observed and latent variables respectively, while boxes refer to repeated sampling with the number of samples in the lower right corner of the boxes. It is well known that the Dirichlet distribution is the conjugate prior of the multinomial distribution. Therefore, a Dirichlet prior with parameter α for document-topic multinomial distribution θ m and a Dirichlet prior with parameter β for topic-word multinomial distribution φ k are chosen respectively. Given a corpus consisting of M documents, LDA makes assumptions that each word w is connected with a latent topic z . Each of topic z k , k ∈ { 1 , 2 , … , K } is related to a multinomial distribution φ k defined on the V -word vocabulary, and each φ k is chosen from a Dirichlet prior distribution with parameter β . Similarly, each document d m is defined as a multinomial distribution θ m over topics, drawn from a Dirichlet prior distribution with parameter α . The full generative process for each document d m in a corpus is defined as follows:

1.

For each topic z k , k ∈ { 1 , 2 , ⋯ , K }

a.

Draw a topic-word multinomial distribution φ k ∼ 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡 ⁢ ( β )

2.

For each document d m , m ∈ { 1 , 2 , … , M }

a.

Draw a document-topic multinomial distribution θ m ∼ 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡 ⁢ ( α )

b.

For each word w m , n , n ∈ { 1 , 2 , … , N m } in document d m

i.

Draw a topic z m , n ∼ 𝑀𝑢𝑙𝑡𝑖𝑛𝑜𝑚𝑖𝑎𝑙 ⁢ ( θ m )

ii.

Draw a word w m , n ∼ 𝑀𝑢𝑙𝑡𝑖𝑛𝑜𝑚𝑖𝑎𝑙 ⁢ ( φ z m , n )

Given the parameters α and β , the joint distribution over the random variables ( 𝐰 𝐦 , 𝐳 𝐦 , φ k , θ m ) then can be derived from Fig. 2a, which is given by:

p ( 𝐰 𝐦 , 𝐳 𝐦 , θ m , φ k | α , β ) = p ( θ m | α ) p ( φ k | β ) ∏ n = 1 N m p ( z m , n | θ m ) p ( w m , n | φ z m , n )(1)

Integrating over θ m and φ k , summing over z m , n , the marginal distribution of a document can be obtained:

p ( 𝐰 𝐦 | α , β ) = ∫ θ m ∫ φ k p ( θ m | α ) p ( φ k | β ) ( ∏ n = 1 N m ∑ z m , n p ( z m , n | θ m ) p ( w m , n | φ z m , n ) ) d θ m d φ k(2)

Finally, taking the product of the marginal probability of every document in the corpus, the generative probability of a corpus is defined as follows:

p ( D | α , β ) = ∏ m = 1 M ∫ θ m ∫ φ k p ( θ m | α ) p ( φ k | β ) ( ∏ n = 1 N m ∑ z m p ( z m , n | θ m ) p ( w m , n | φ z m , n ) ) d θ m d φ k(3)

In LDA, there are two sets of parameters that need to be estimated from a collection of documents, one is the topic distribution in each document and the other is the word distribution in each topic. In reality, only the documents can be observed, while the topic structure including topics and topic probability proportions is hidden. The key issue of LDA model is to use the observed documents to infer the latent topic structure. Therefore, some statistical approaches have been fully utilized for inferring the latent variables that can generate the observed collection of documents best. The exact inference for posterior estimation is intractable in general, thus a wide variety of approximate inference algorithms are considered for LDA, including Expectation-Maximization [23], Gibbs Sampling [19, 35] and Variational approximation [4].

3.2 STLDA: A new probabilistic model for attractions

STLDA is a novel probabilistic topic model with the aim to extract topics from a collection of attraction documents by taking advantage of information of documents as well as the intrinsic seasonal characteristic in each document. STLDA is an expanded model of LDA by adding an additional season layer between the document layer and the topic layer. Therefore, STLDA is a four-level hierarchical Bayesian model, where seasonal features are correlated with documents, under which topics are associated with seasonal characteristics and words are related to topics. The STLDA model has a crucial enhancement that can clearly identify the meaningful topics corresponding to various seasonal contexts for each tourist attraction. As a result, the tourist attractions are described more comprehensively and precisely on the season level of fine-grained, which can benefit the further analysis. In this paper, by using the intrinsic seasonal characteristic in each tourist attraction, we assume that the words in attraction documents have distinct seasonal tendencies. The STLDA model is represented as a probabilistic graphical model in Fig. 2b.

OPEN IN VIEWER

Assume that we have a corpus with a collection of M documents denoted by { d 1 , d 2 , … , d M } , each document in the corpus is a sequence of N m words represented by d m = ( w 1 , w 2 , … , w N m ) , and each word in the document is an entity from a vocabulary with V distinct words denoted by { w 1 , w 2 , … , w V } . The number of season segments is S and the total number of topics is K . In our probabilistic generative model, we assume that each word w is related to one of latent topics z , just like the model of LDA does. Each topic z k , k ∈ { 1 , 2 , … , K } is defined as a multinomial distribution φ k over the V -word vocabulary, and each φ k is chosen from a Dirichlet prior distribution with parameter β . Each document d m is modeled by S different multinomial distribution θ m , s over the K topics with respect to different season labels s , s ∈ { 1 , 2 , ⋯ , S } , all drawn from a Dirichlet prior distribution with parameter α , which significantly distinguish STLDA from the original LDA model that each document d m is defined as just one multinomial distribution θ m over the K topics. Besides, another distribution π m is defined for each document d m , m ∈ { 1 , 2 , … , M } over the S season segments, drawn from a Dirichlet prior distribution with parameter γ . The process for generating a word w m , n in document d m under STLDA has three steps. First, a season label s is chosen from the document’s specific season distribution π m . Then a topic is sampled from the topic distribution θ m , s conditioned on both the document and the selected season label. Finally, a word is drawn from distribution over words defined by the topic. The notations of STLDA model to be used throughout the paper are summarized with brief descriptions in Table 1. The full generative process of STLDA model for each document d m in a corpus is defined as follows:

[H] Generative process of STLDA.[1] each topic z k , k ∈ { 1 , 2 , … , K } Draw a topic-word multinomial distribution φ k ∼ 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡 ⁢ ( β ) ; document d m , m ∈ { 1 , 2 , … , M } Draw a document-season multinomial distribution π m ∼ 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡 ⁢ ( γ ) ; each season label s , s ∈ { 1 , 2 , … , S } under document d m Draw a document-season-topic multinomial distribution θ m , s ∼ 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡 ⁢ ( α ) ; each word w m , n , n ∈ { 1 , 2 , … , N m } in document d m Draw a season label s m , n ∼ 𝑀𝑢𝑙𝑡𝑖𝑛𝑜𝑚𝑖𝑎𝑙 ⁢ ( π m ) ; Draw a topic z m , n ∼ 𝑀𝑢𝑙𝑡𝑖𝑛𝑜𝑚𝑖𝑎𝑙 ⁢ ( θ m , s m , n ) ; Draw a word w m , n ∼ 𝑀𝑢𝑙𝑡𝑖𝑛𝑜𝑚𝑖𝑎𝑙 ⁢ ( φ z m , n ) ;

OPEN IN VIEWER

As previously mentioned, STLDA considers both general description of attraction and seasonal features exisiting in attraction document in a unified manner and can detect the meaningful topics with respect to different seasons for each attraction. Figure 3 is a running example of STLDA model. As can be seen from this figure, there is a clear hierarchy followed by the attraction document layer, season layer, topic layer and word layer. The words constitute a number of topics and the tourist attraction corresponds to various topics in different seasons, where the weights labeled in the corresponding edges indicate the topic occurrence probabilities. For example for the attraction in spring, the detected topics are T7 with probability value 0.635 and T20 with probability value 0.208, while in winter the attraction corresponds to topics T16, T4 and T26 and the probability values are 0.613, 0.184 and 0.136 respectively.

Now, the likelihood function for the observed textual data of tourist attractions can be formulated according to our proposed probabilistic generative model. Given the hyperparameters α , β and γ , the joint distribution over the random variables ( 𝐰 𝐦 , 𝐳 𝐦 , 𝐬 𝐦 , φ k , θ m , s , π m ) can be derived from Fig. 2b, which is given by:

p ( 𝐰 𝐦 , 𝐳 𝐦 , 𝐬 𝐦 , φ k , θ m , s , π m | α , β , γ ) (4) = ∏ n = 1 N m p ( w m , n | φ z m , n ) p ( z m , n | θ m , s m , n ) p ( s m , n | π m ) ⏟ word plate p ( π m | γ ) ⏟ season plate p ( θ m , s | α ) ⏞ document plate ⁢ p ( φ k | β ) ⏟ topic plate

where θ m , s , φ k and π m denote model parameters to be estimated, 𝐳 𝐦 and 𝐬 𝐦 are hidden variables, 𝐰 𝐦 refer to known variable. α , β and γ are the Dirichlet priors for θ m , s , φ k and π m respectively. Furthermore, the clear hierarchy struture of the model is also maked in the above formula.

By integrating out the distributions of θ m , s , φ k and π m and summing over z m , n and s m , n , the likelihood of a document 𝐰 𝐦 can be obtained:

p ( 𝐰 𝐦 | α , β , γ ) = ∫ Θ ∫ Φ ∫ Π p ( θ m , s | α ) p ( φ k | β ) p ( π m | γ ) ⋅ ∏ n = 1 N m ∑ z m , n ∑ s m , n p ( w m , n | φ z m , n ) p ( z m , n | θ m , s m , n ) p ( s m , n | π m ) d Θ d Φ d Π(5)

where Θ , Φ and Π denote the set of θ m , s for all m ∈ { 1 , 2 , … , M } ⁢ s ∈ { 1 , 2 , … , S } , the set of φ k for all k ∈ { 1 , 2 , … , K } and the set of π m for all m ∈ { 1 , 2 , … , M } respectively.

Finally, the likelihood of the complete corpus D is determined by the product of the likelihoods of the independent documents:

p ( D | α , β , γ ) = ∏ m = 1 M ∫ Θ ∫ Φ ∫ Π p ( θ m , s | α ) p ( φ k | β ) p ( π m | γ ) ⋅ ∏ n = 1 N m ∑ z m , n ∑ s m , n p ( w m , n | φ z m , n ) p ( z m , n | θ m , s m , n ) p ( s m , n | π m ) d Θ d Φ d Π(6)

There are three sets of latent variables that need to be estimated in our model, including: the topic distribution of the corresponding per document-season pair θ m , s , the per topic-word multinomial distribution φ k and the per document-season multinomial distribution π m .

4.1 Model inference based on Gibbs sampling

Given a collection of documents, the posterior distribution of the latent variables including θ m , s , φ k and π m can be computed when the likelihood function of the whole corpus is maximized. The procedure of variable inference is to invert the document generative process and infer the latent variables from the observed corpus. While the exact inference for posterior estimation is intractable in general, a wide variety of approximate inference algorithms are considered to estimate these parameters. In this paper, we adopt the Gibbs sampling algorithm to perform approximate inference. Gibbs sampling is a simple and widely applicable Markov chain Monte Carlo algorithm for obtaining a sequence of observations which are approximated from a specified multivariate probability distribution, when direct sampling is difficult [1]. In Gibbs sampling, each parameter is sequentially sampled according to the full conditional distribution of the parameter conditioned on the observed data and all the other parameters. Since this method can get global optimum solution in theory and can be easily implemented while other inference algorithms such as variational EM algorithm is only guaranteed to find a local optimum [16], it is commonly used in Bayesian inference domain, especially for topic modeling.

In the case of our model, the target of inference is the posterior distribution of the hidden variables 𝐳 and 𝐬 , which is defined as follows:

p ( 𝐳 , 𝐬 | 𝐰 ) = p ⁢ ( 𝐳 , 𝐬 , 𝐰 ) p ⁢ ( 𝐰 ) = ∏ i = 1 W p ⁢ ( z i , s i , w i ) ∏ i = 1 W ∑ k = 1 K ∑ j = 1 S p ( z i = k , s i = j , w i )(7)

where W is the number of all the words that appear in corpus, 𝐳 and 𝐬 are vector of topics that appear with the words in corpus and season labels that assigned to the words of corpus respectively. Moreover, the hyperparameters are omitted for simplicity. This posterior distribution covers a large space of discrete random variables, causing intractable in exact computation. At this point, the Gibbs sampling procedure comes into play. In our setting, the desired Gibbs sampler runs a Markov chain that exploits the full conditional distribution p ( z i , s i | 𝐳 - 𝐢 , 𝐬 - 𝐢 , 𝐰 ) so as to simulate the posterior distribution p ( 𝐳 , 𝐬 | 𝐰 ) , where the subscript - i refers to a quantity that excludes word w i from the observed data.

Specifically, in our model, the full conditional probability distribution for a word w i based on the topics and season labels of all other variables as well as the observed data is defined as follows:

p ( z i , s i | 𝐰 , 𝐳 - 𝐢 , 𝐬 - 𝐢 , α , β , γ ) = p ( w i , z i , s i | 𝐰 - 𝐢 , 𝐳 - 𝐢 , 𝐬 - 𝐢 , α , β , γ ) p ( w i | 𝐰 - 𝐢 , 𝐳 - 𝐢 , 𝐬 - 𝐢 , α , β , γ )(8)

where 𝐰 - 𝐢 , 𝐳 - 𝐢 and 𝐬 - 𝐢 represent vector of all words that appear in corpus excluding word w i , vector of topic assignments for all words except word w i and season labels vector for all words in corpus excluding word w i respectively.

To apply a Gibbs sampling algorithm, the joint probability distribution of the observed words, topics and season labels assignments of the whole corpus is first derived by dividing this joint distribution into three parts, which is given by:

p ( 𝐰 , 𝐳 , 𝐬 | α , β , γ ) = p ( 𝐰 | 𝐳 , β ) p ( 𝐳 | 𝐬 , α ) p ( 𝐬 | γ )(9)

since the first part p ( 𝐰 | 𝐳 , β ) in the above equation is independent of α and γ , the second part p ( 𝐳 | 𝐬 , α ) is irrelevant to β and γ , and the third part p ( 𝐬 | γ ) is unrelated to α and β , these elements of the joint probability distribution can now be handled separately.

The first part p ( 𝐰 | 𝐳 , β ) can be derived from a multinomial on the observed word counts given the associated topics:

p ( 𝐰 | 𝐳 , Φ ) = ∏ i = 1 W p ( w i | z i ) = ∏ i = 1 W φ z i , w i(10)

where Φ indicates the set of φ k for all k ∈ { 1 , 2 , … , K } and φ z i , w i refers to the probability that word w i is generated from topic z i . This formula shows that the W words of the corpus are observed according to independent multinomial trials with parameters conditioned on the corresponding topic indice z i . By splitting the product over words into one product over topics and one over the vocabulary, we obtain the following equation:

p ( 𝐰 | 𝐳 , Φ ) = ∏ k = 1 K ∏ t = 1 V p ( w i = t | z i = k ) = ∏ k = 1 K ∏ t = 1 V φ k , t n k ( t )(11)

where n k ( t ) denotes the count of word t assigned to topic k in corpus.

Then, the target distribution p ( 𝐰 | 𝐳 , β ) can be obtained by integrating over Φ :

p ( 𝐰 | 𝐳 , β ) = ∫ Φ p ( 𝐰 | 𝐳 , Φ ) p ( Φ | β ) d Φ = ( Γ ⁢ ( ∑ t = 1 V β t ) ∏ t = 1 V Γ ⁢ ( β t ) ) K ∏ k = 1 K ∏ t = 1 V Γ ⁢ ( β t + n k ( t ) ) Γ ⁢ ( ∑ t = 1 V β t + n k ( t ) )(12)

where β = ( β 1 , β 2 , … , β V ) denotes the hyperparameters for topic-word multinomial distribution φ k and Γ refers to the gamma function.

Analogous to p ( 𝐰 | 𝐳 , β ) , the second part of the joint probability distribution p ( 𝐳 | 𝐬 , α ) can be derived from the following formula:

p ( 𝐳 | 𝐬 , Θ ) = ∏ i = 1 W p ( z i | s i , d i ) = ∏ m = 1 M ∏ k = 1 K ∏ j = 1 S p ( z i = k | s i = j , d i = m ) = ∏ m = 1 M ∏ k = 1 K ∏ j = 1 S θ m , j , k n m , j ( k )(13)

where the notation d i refers to a document in the collection of { d 1 , d 2 , … , d M } that the word w i belongs to, θ m , j , k denotes the probability of topic k appeared with season label j in document m and n m , j ( k ) is the count of words assigned to topic k and season label j in document m . By integrating out Θ , the second element p ( 𝐳 | 𝐬 , α ) can be formulated as follows:

p ( 𝐳 | 𝐬 , α ) = ∫ Θ p ( 𝐳 | 𝐬 , Θ ) p ( Θ | α ) d Θ = ( Γ ⁢ ( ∑ k = 1 K α k ) ∏ k = 1 K Γ ⁢ ( α k ) ) M * S ∏ m = 1 M ∏ j = 1 S ∏ k = 1 K Γ ⁢ ( α k + n m , j ( k ) ) Γ ⁢ ( ∑ k = 1 K α k + n m , j ( k ) )(14)

where α = ( α 1 , α 2 , … , α K ) is the hyperparameters for document-season-topic multinomial distribution θ m , j .

Likewise, the third part p ( 𝐬 | γ ) can also be obtained by integrating out Π :

p ( 𝐬 | γ ) = ∫ Π p ( 𝐬 | π ) p ( π | γ ) d Π = ( Γ ⁢ ( ∑ j = 1 S γ j ) ∏ j = 1 S Γ ⁢ ( γ j ) ) M ∏ m = 1 M ∏ j = 1 S Γ ⁢ ( γ j + n m ( j ) ) Γ ⁢ ( ∑ j = 1 S γ j + n m ( j ) )(15)

where γ = ( γ 1 , γ 2 , … , γ S ) refers to the hyperparameters for document-season multinomial distribution π m , π m , j and n m ( j ) indicate the probability of season label j associated with document m and the count of words assigned to season label j in document m respectively. Therefore, the joint probability distribution of the observed words, topics and season labels assignments of the whole corpus will be obtained by multiplying the results of Eqs (12), (14) and (15) according to Eq. (9).

After deriving the joint probability distribution, the topic assignment z i and season label assignment s i for each word w i in the corpus can be sampled with the following full conditional probability distribution:

p ( z i = k , s i = j | 𝐰 , 𝐳 - 𝐢 , 𝐬 - 𝐢 , α , β , γ ) = p ⁢ ( 𝐰 , 𝐳 , 𝐬 , α , β , γ ) p ⁢ ( 𝐰 - 𝐢 , w i , 𝐳 - 𝐢 , 𝐬 - 𝐢 , α , β , γ ) = p ⁢ ( 𝐰 , 𝐳 , 𝐬 , α , β , γ ) p ( 𝐰 - 𝐢 , 𝐳 - 𝐢 , 𝐬 - 𝐢 , α , β , γ ) p ( w i | 𝐰 - 𝐢 , 𝐳 - 𝐢 , 𝐬 - 𝐢 , α , β , γ ) ∝ p ( 𝐰 , 𝐳 , 𝐬 | α , β , γ ) p ( 𝐰 - 𝐢 , 𝐳 - 𝐢 , 𝐬 - 𝐢 | α , β , γ )(16)

The above formula can then be factored, which is shown as follows:

p ( 𝐰 , 𝐳 , 𝐬 | α , β , γ ) p ( 𝐰 - 𝐢 , 𝐳 - 𝐢 , 𝐬 - 𝐢 | α , β , γ ) = ( Γ ⁢ ( ∑ t = 1 V β t ) ∏ t = 1 V Γ ⁢ ( β t ) ) K ⁢ ∏ k = 1 K ∏ t = 1 V Γ ⁢ ( β t + n k ( t ) ) Γ ⁢ ( ∑ t = 1 V β t + n k ( t ) ) ( Γ ⁢ ( ∑ t = 1 V β t ) ∏ t = 1 V Γ ⁢ ( β t ) ) K ⁢ ∏ k = 1 K ∏ t = 1 V Γ ⁢ ( β t + n k , - i ( t ) ) Γ ⁢ ( ∑ t = 1 V β t + n k , - i ( t ) ) ⋅ ( Γ ⁢ ( ∑ j = 1 S γ j ) ∏ j = 1 S Γ ⁢ ( γ j ) ) M ⁢ ∏ m = 1 M ∏ j = 1 S Γ ⁢ ( γ j + n m ( j ) ) Γ ⁢ ( ∑ j = 1 S γ j + n m ( j ) ) ( Γ ⁢ ( ∑ j = 1 S γ j ) ∏ j = 1 S Γ ⁢ ( γ j ) ) M ⁢ ∏ m = 1 M ∏ j = 1 S Γ ⁢ ( γ j + n m , - i ( j ) ) Γ ⁢ ( ∑ j = 1 S γ j + n m , - i ( j ) ) ⋅ ( Γ ⁢ ( ∑ k = 1 K α k ) ∏ k = 1 K Γ ⁢ ( α k ) ) M * S ⁢ ∏ m = 1 M ∏ j = 1 S ∏ k = 1 K Γ ⁢ ( α k + n m , j ( k ) ) Γ ⁢ ( ∑ k = 1 K α k + n m , j ( k ) ) ( Γ ⁢ ( ∑ k = 1 K α k ) ∏ k = 1 K Γ ⁢ ( α k ) ) M * S ⁢ ∏ m = 1 M ∏ j = 1 S ∏ k = 1 K Γ ⁢ ( α k + n m , j , - i ( k ) ) Γ ⁢ ( ∑ k = 1 K α k + n m , j , - i ( k ) ) = α k + n m , j ( k ) - 1 ∑ k = 1 K α k + n m , j , - i ( k ) ⁢ β t + n k ( t ) - 1 ∑ t = 1 V β t + n k , - i ( t ) ⁢ γ j + n m ( j ) - 1 ∑ j = 1 S γ j + n m , - i ( j )(17)

Noting that, the above derivations exploited one important property of Gamma function Γ ⁢ ( n + 1 ) = n ⁢ Γ ⁢ ( n ) .

Finally, the full conditional probability distribution for each variable z i and s i can be equally separated from the joint sampling distribution of topic assignment and sesaon label indicator for word w i , which are given by:

p ( z i = k | 𝐰 , 𝐳 - 𝐢 , 𝐬 , α , β , γ ) ∝ ( α k + n m , j ( k ) - 1 ) β t + n k ( t ) - 1 ∑ t = 1 V β t + n k , - i ( t )(18)

and

p ( s i = j | 𝐰 , 𝐬 - 𝐢 , 𝐳 , α , β , γ ) ∝ ( γ j + n m ( j ) - 1 ) α k + n m , j ( k ) - 1 ∑ k = 1 K α k + n m , j , - i ( k )(19)

Gibbs sampling will serially draw each variable of z i and s i from the full conditional distribution of each variable conditioned on remaining variables and the observations according to Eqs (18) and (19). The algorithm iterates repeatedly until the Markov chain has reached a stationary state, where the stationary distribution of the Markov chain is our desired posterior distribution. According to Gibbs sampling rule, a bunch of draws that are approximately from our desired posterior distribution are obtained once the Markov chain has converged. Then, the model parameters Θ , Φ and Π corresponding to the stationary state of the Markov chain can be estimated based on these samples. The pseudocode of Gibbs sampling procedure for our model is presented in Algorithm 2, which runs over the three periods: initialisation period, burn-in period and sampling period.

[h] Gibbs sampling procedure of STLDA. Corpus of attraction documents, α , β , γ , K , S topic assignment and season label assignment for all words in the corpusInitialize M × S × K matrice Θ , K × V matrice Φ , M × S matrice Π Zero all count variables: n m , j ( k ) , n m , j , n k ( t ) , n k , n m ( j ) , n m each document d m , m ∈ { 1 , 2 , ⋯ , M } each word w m , n , n ∈ { 1 , 2 , … , N m } in document d m sample topic index z m , n = k ∼ M ⁢ u ⁢ l ⁢ t ⁢ i ⁢ n ⁢ o ⁢ m ⁢ i ⁢ a ⁢ l ⁢ ( 1 / K ) sample season label indice s m , n = j ∼ M ⁢ u ⁢ l ⁢ t ⁢ i ⁢ n ⁢ o ⁢ m ⁢ i ⁢ a ⁢ l ⁢ ( 1 / S ) increment counts and sums: n m , j ( k ) + 1 , n m , j + 1 , n k ( t ) + 1 , n k + 1 , n m ( j ) + 1 , n m + 1 not finished each document d m , m ∈ { 1 , 2 , ⋯ , M } each word w m , n , n ∈ { 1 , 2 , … , N m } in document d m decrement counts and sums: n m , j ( k ) - 1 , n m , j - 1 , n k ( t ) - 1 , n k - 1 , n m ( j ) - 1 , n m - 1 sample a new topic index k ∼ ∼ p ( z i = k | 𝐰 , 𝐳 - 𝐢 , 𝐬 ) according to Eq. (18) sample a new season label j ∼ ∼ p ( s i = j | 𝐰 , 𝐬 - 𝐢 , 𝐳 ) using Eq. (19) increment counts and sums: n m , j ∼ ( k ∼ ) + 1 , n m , j ∼ + 1 , n k ∼ ( t ) + 1 , n k ∼ + 1 , n m ( j ∼ ) + 1 , n m + 1 the Markov chain has converged every 100 iterations update matrices Θ , Φ and Π with new sampling results output matrices Θ , Φ and Π according to Eqs (23), (24) and (25)

By applying Bayes’ rule, the multinomial distributions θ m , j , φ k and π m with their Dirichlet prior α , β and γ can be redefined as follows:

p ( θ m , j | D , α ) = 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡 ( θ m , j | 𝐧 𝐦 , 𝐣 + α )(20) p ( φ k | D , β ) = 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡 ( φ k | 𝐧 𝐤 + β )(21) p ( π m | D , γ ) = 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡 ( π m | 𝐧 𝐦 + γ )(22)

where D denotes the observed corpus, 𝐧 𝐦 , 𝐣 , 𝐧 𝐤 and 𝐧 𝐦 refer to vector of topic observation counts for document m and season label j , term observation counts for topic k and season label observation counts for document m respectively. Specifically, 𝐧 𝐦 , 𝐣 , 𝐧 𝐤 and 𝐧 𝐦 are denoted by ( n m , j ( 1 ) , n m , j ( 2 ) , … , n m , j ( K ) ) , ( n k ( 1 ) , n k ( 2 ) , … , n k ( V ) ) and ( n m ( 1 ) , n m ( 2 ) , … , n m ( S ) ) respectively.

The probability of topic k asscociated with season label j in document m now can be approximated by using the expectation of Dirichlet distribution, which is given by:

θ m , j , k = n m , j ( k ) + α k ∑ k = 1 K n m , j ( k ) + α k(23)

For the remaining parameters φ k and π m using the same method, the approximate probability of word t in a topic k is formulated as follows:

φ k , t = n k ( t ) + β t ∑ t = 1 V n k ( t ) + β t(24)

The approximate probability of season label j in a document m is presented as follows:

π m , j = n m ( j ) + γ j ∑ j = 1 S n m ( j ) + γ j(25)

Note that the model parameter Θ is a M × S × K matrice, where each document season-topic distribution corresponds to a S × K matrice such that the value in the j th row and the k th column of each matrice is θ m , j , k . Similarly, Φ is a K × V matrice such that the value in the k th row and the t th column of Φ is φ k , t and Π is a M × S matrice such that the value in the m th row and the j th column of Π is π m , j .

5.1 Data collection and pre-processing

We employ English database of Wikipedia (http://www.wikipedia.org) as the primary source of the experimental data from which attraction description information is retrieved. Meanwhile, attraction information is also collected from their official websites and other tourism websites such as Wikitravel (http://wikitravel.org) and TravelChinaGuide (https://www.travelchinaguide.com). Then a comprehensive attraction description document is generated by integrating all these related information, which contains abundant knowledge for topic detection.

We construct an attraction corpus that consists of attractions description documents written in English. Each document in the corpus is associated with a single famous tourist attraction in China, covering 160 unique attractions in total. The selected attractions including natural landscape and cultural landscape are mainly 5A or 4A tourist attractions evaluated by China National Tourism Administration, where 5A represents the highest level of tourist attraction. Table 2 shows the summary of our data collection.

Since attraction textual information acquired from the Internet is unstructured and usually contains much disturbance, it is necessary to perform preprocessing on the original attractions textual data. Firstly, punctuations, numbers and other non-alphabet characters are removed. Secondly, all words are lowercased, stop words are removed based on a stop word list from natural language toolkit (NLTK) [3]. Thirdly, for the purpose of reducing the vocabulary size, the low frequency words that appear less than twice in corpus are also filtered out. Then, we use the spell-check function of word processor to examine our data collection. After preprocessing of the textual information in each attraction, the word distribution of a document can be obtained. Finally, the corpus is further expressed with a data format that can be identifiable by STLDA and LDA model.

Perplexity, widely used in the natural language modeling fields, is an important indicator to demonstrate the predictive power of a model [30]. A lower perplexity value means that a higher likehood is achieved on a test dataset, thus indicates a better generalization performance of a model. Given a test dataset D of M documents, the perplexity value can be calculated as follows:

perplexity(D) = 𝑒𝑥𝑝 ⁢ { - ∑ m = 1 M 𝑙𝑜𝑔𝑝 ⁢ ( 𝐰 𝐦 ) ∑ m = 1 M N m }(26)

where p ⁢ ( 𝐰 𝐦 ) denotes the generative probability of document m and N m is the number of words in document m .

OPEN IN VIEWER

In our experiments, we use perplexity to measure the generalization performance of the proposed model and the results are shown in Fig. 4. For the attraction corpus, 10-fold cross-validation scheme is adopted for enhancing the reliability of our results. The 10-fold cross-validation is a technique of dividing the original dataset into ten disjoint subsets with approximately equal size, and every subset is in turn used for prediction while the remaining nine subsets are used to train the model [21]. As shown in Fig. 4, STLDA presents lower perplexity value than LDA with different number of topics, which indicates STLDA owns a better predictive power for unseen documents than the original LDA model. Further analysis shows that the perplexity performance is improved about 24.09% on average. This is due to the ability of STLDA can well represent the content of new attractions documents by taking the intrinsic seasonal features of attractions into consideration. From Fig. 4, we can find that the perplexity values of these two models decrease rapidly with the number of topics increasing from 10 to 30, while the performances of these two models become worse when further increasing the latent topic number from 30 to 100. The experimental results reveal that the optimum number of topics for the attraction corpus is 30.

The statistical significance of the difference between the STLDA and LDA model regarding the perplexity performance is further assessed by using the Wilcoxon signed ranks test. The Wilcoxon test is a nonparametric test method that is used when overall distribution is unknown [37]. According to the test result, the value of Z statistics is - 2.803 and the concomitant probability α is 0.005, less than the significance level of 0.05, which indicates the perplexity performance of STLDA is significantly better than that of LDA at the 95% confidence level. The statistical analysis demonstrates that seasonal contextual information contribute positively to the performance of topic modeling.

OPEN IN VIEWER

To evaluate the time complexity of our proposed model on attraction corpus, we summarize the running time of STLDA and LDA for different number of topics K in Fig. 5. From this figure, it can be easily observed that the running time of STLDA are all longer than LDA with different values of K . STLDA adds an additional season layer on the basis of basic LDA model and this leads to its higher computational complexity. However, from Fig. 5 we can find that just like the LDA model, STLDA still has the linear time complexity and its running time grows linearly as the number of topics increases.

In this section, we train STLDA and LDA model on the whole attraction corpus to learn topics respectively. The number of topics is set empirically to 30 according to Section 5.2. Since the representative topical words generated from the STLDA and LDA model are very close, we present the 22 topics that sharing the same meaning by these two models in Table 3, where the topic number j denotes the j th topic discovered by the model. The representative ten words of each topic are also shown in Tabel 3. Note that here we use representative words in each topic rather than the common practice that words with the highest probabilities. The reason for this is that the topic features of attractions often reveal high-level concepts, which are highly correlated to travel-related words. The words irrelevant with travel are removed for better illustrating the extracted topics.

As can be seen from Table 3, the extracted topics apparently characterize some features of attractions, including both natural styles like woods (topic 7), snowscape (topic 16) and cultural styles like playground (topic 21). The representative words in these different topics are quite informative and coherent. For example for topic 4, words such as relaxing, entertainment, comfort, fun, vacation and resort are related to each other and semantically coherent, conveying the meaning about leisure and entertainment, and thus we name the topic accordingly. In addition to the meaning of these topical words, we also refer to classification criteria of Chinese national tourism resources [12] and the related study of paper [17, 24, 32, 45] to name all the topics that extracted from attraction corpus.

Next, we illustrate how STLDA can accurately capture season-dependent topic clusters and improve topic representation of tourist attractions. In our experiments, topics whose occurrence probability greater than 0.1 are selected for each tourist attraction. Three tourist attractions, namely Nalati scenic spots, Yuntai Mountain and Zhangjiajie National Forest Park, are chosen as typical examples to compare with LDA model. These three attractions are all national 5A tourist attractions and located in Northwest of China, middle east of China and Central of China respectively. Table 4 summarizes the obtained topics and topic probability distributions of these tourist attractions using STLDA and LDA model. The detected topics for each tourist attraction are arranged in descending order according to their probability values and the probability value corresponding to each topic is shown in parentheses.

From Table 4, we can clearly see that the topics and topics’ occurrence probability of all three tourist attractions change significantly with the alternation of seasons. Specifically, we take Nalati scenic spots for example. In spring, the detected topics from STLDA are woods and blossom and the corresponding probability value are 0.635 and 0.208 respectively. Obviously, the representative topic features of Nalati scenic spots are woods and flowers in spring, which is consistent with the common sense that the beautiful natural scenery is prominent in spring of Nalati scenic spots. The topic generated from STLDA with highest probability in summer is cultural activity (0.494). By accessing the relevant information, we see that the temperature is agreeable in summer of Nalati scenic spots, and the local hospitable Kazakhs often hold a variety of folk activities to show their colorful ethnic culture. These observations reveal that topics generated from STLDA are capable of reflecting the features of attraction with respect to different seasons in real life. For LDA model, the detected topic with highest probability is snowscape, followed by woods, village, entertainment and blossom. The topic representation of Nalati scenic spots obtained from two models are also visually shown in Fig. 6.

OPEN IN VIEWER

Further comparison and analysis of the results generated from different models, three conclusions can be made. Firstly, the topics found by STLDA and LDA for the same tourist attraction indeed have a certain degree of similarity, but the topic probability distributions are prominently different. For instance, the results of STLDA and LDA for Yuntai Mountain both have topics such as woods and blossom. The corresponding probability value obtained from STLDA are 0.648 and 0.121, while in LDA are 0.220 and 0.176 respectively. Secondly, STLDA explicitly identifies topic clusters corresponding to various seasonal contexts, while the topic representation of LDA tend to be more general and less coherent. Taking Zhangjiajie National Forest Park as an example, STLDA clearly detects and localizes the snowscape topic in winter and the maple leaves topic in autumn, but these topics are confusingly merged by LDA. Not modeling time can confound co-occurrence topic patterns and result in unclear topic representation for tourist attractions. Finally, STLDA clearly detects some other topics that are ignored by LDA model. For Nalati scenic spots, topics found by STLDA model such as maple leaves, ice sports and cultural activity may be the representative features in specific season, but are neglected by LDA model. If the topics whose occurrence probability less than 0.1 are examined from the results of LDA, we can see that the probability values of these ignored topics are 0.088, 0.043 and 0.0002 respectively. Fortunately, the probability value of cultural activity topic increases from 0.0002 to 0.494 in STLDA, which makes the cultural activity topic prominent in summer of Nalati scenic spots. This difference comes from STLDA’s assumption that takes the intrinsic seasonal features of each tourist attraction into consideration. Therefore, STLDA can capture the potential season-dependent topics on a season level of fine-grained, while some of meaningful topics are filtered out in LDA due to their extremely low probability value on a coarse-grained level.

To show the dominancy of STLDA over the basic LDA model more intuitively, we evaluate the statistical properties of obtained topics and topic probability distributions of all 160 tourist attractions. We choose five statistical indicators, namely Richness, Coincidence, Diversity, Significance and Volatility, and the results are shown in Table 5. Richness indicator refers to the average number of detected topics for each tourist attraction. Coincidence indicator denotes the average coincident number of topics generated from two models for each tourist attraction. Diversity indicator reflects the average number of extra topics generated from one model over the other model for each tourist attraction. Significance indicator represents the average highest topic probability value of each tourist attraction. Volatility indicator indicates the average standard deviation of topic probability distribution corresponding to each tourist attraction. The SP, SU, AU and WI in Table 5 denote spring, summer, autumn and winter respectively.

According to Table 5, the average number of topics generated from STLDA is 7.838, while the value is 3.313 in LDA, indicating that STLDA is capable of detecting more topics than LDA. The value of Coincidence indicator is 2.856, comparing with the Richness value 3.313 in LDA, we can see that the topics identified by STLDA can mostly cover the topics generated from LDA. Further referring to the Diversity indicator, the values corresponding to STLDA and LDA are 4.981 and 0.456, which implies that STLDA can capture 4.981 topics more than LDA for each tourist attraction on average, while fail to detect only 0.456 topics. The Significance indicator of STLDA is calculated in each season. Their values are 0.564, 0.537, 0.536 and 0.563 respectively, all larger than the value 0.355 in LDA. The result reveals that topic with the highest probability is dominant in topics generated from STLDA for most attractions in specific season. In other words, the results of STLDA can clearly give the prominent topic that a tourist attraction belongs to. Finally, we investigate the Volatility indicator. The Volatility indicator values of STLDA all larger than that of LDA in different seasons, indicating that the topic probability values generated from STLDA have remarkable difference. But the topics found by LDA tend to have relatively uniform probability value and consequently there is no obvious discrimination between these topics.

Further deeply investigate the topic probability distributions of all 160 tourist attractions generated from STLDA model, we find that not all tourist attractions have clearly seasonality in every season, but all exhibit seasonal characteristics to some extent. For example, Wuyuan, the most beautiful village in China, have prominent topics in spring, summer and autumn. However, in winter, it presents a uniform probability distribution over the topics and the corresponding probability values all less than 0.1. Similarly, Thousand Islet Lake shows seasonal characteristics in spring, autumn and winter and it does not present distinct seasonality in summer.

These comparative results indicate that the seasonal contextual information captured by STLDA plays a very important role in forming better topic representation of tourist attractions. By including seasonal contextual information, STLDA can model the variations of topic occurrence that reflect the changing seasonal contexts, which helps us understand more precisely what remarkable topic features are corresponding to specific season for a given tourist attraction. The most straightforward application is to apply STLDA to the field of personalized attraction recommendation. For example, a tourist plans to travel during the Labour Day. The holiday lasting for three days is one of the peak travel seasons every year. Besides, the Labour Day is on May 1st which coincides with spring in most areas of China. At this time, the tourist intend to see flowers. Taking the data shown in Table 4 as example, the probability values of blossom topic in Nalati scenic spots, Yuntai Mountain and Zhangjiajie National Forest Park generated from LDA are 0.102, 0.176 and 0.179 respectively and there is no distinct difference between these probability values. When providing recommendation for the tourist based on the results of LDA, the tour operator would lack obvious pertinence. But according to the results of STLDA, it is apparently that Zhangjiajie National Forest Park should be recommended to him since the probability value of blossom topic up to 0.678 in spring. Recommending different tourist attractions to users exposed at specific seasonal contexts on the basis of different seasonal topic features of attractions can achieve higher satisfaction for tourists and realize more profit for tour operators undoubtedly.