Multi-label text classification based on the label correlation mixture model

4.1 Label correlation network

We define a label correlation network employing an undirected graph, in which each vertex represents a single label, each edge between two vertexes indicates the correlation of the two corresponding labels, and the weight on each edge represents a certain measure for the correlation. Our basic assumption is that labels are generated based on the label correlation network, and P ( y l | 𝐲 , Ξ ) in the generative process S1-(c) can be calculated via the label correlation network. Here, we assume that the label correlation network has been set up. In Section 4.3, we discuss how to create a label correlation network and how to learn its parameters.

Denote g ⁢ ( y , y ′ ) as the edge between the two vertexes which corresponds to label y and label y ′ , and denote ξ ⁢ ( y , y ′ ) as the weight on g ⁢ ( y , y ′ ) . Define Ξ as all the weights in the label correlation network:

Ξ = { ξ ⁢ ( y , y ′ ) | y ≠ y ′ , y ∈ 𝐘 , y ′ ∈ 𝐘 } .

𝐲 ¯ denotes the complementary set of a label subset y,

𝐲 ¯ = 𝐘 ∖ 𝐲 .

𝐲 * denotes the candidate set of y,

𝐲 * = { y ′ | ∃ g ⁢ ( y , y ′ ) , y ∈ 𝐲 , y ′ ∈ 𝐲 ¯ } .

Given a subset y, suppose that another label y ′ will be sampled. Our basic idea is that the label y ′ should be chosen from the “candidate set” 𝐲 * , because 𝐲 * includes all the labels that connect to at least one label in y. Denote P ( y ′ | 𝐲 , Ξ ) as the conditional probability of choosing a label y ′ when a subset y is given. This is another form of P ( y l | 𝐲 , Ξ ) in the generative process S1-(c), and we suppose it can be calculated by:

P ( y ′ | 𝐲 , Ξ ) = κ ⁢ ( 𝐲 , y ′ ) ∑ y ′′ ∈ 𝐲 * κ ⁢ ( 𝐲 , y ′′ ) = ∑ y ∈ 𝐲 ξ ⁢ ( y , y ′ ) ∑ y ′′ ∈ 𝐲 * ∑ y ∈ 𝐲 ξ ⁢ ( y , y ′′ ) , y ′ ∈ 𝐲 *(5)

where κ ⁢ ( 𝐲 , y ′ ) = ∑ y ∈ 𝐲 ξ ⁢ ( y , y ′ ) . According to Eq. (5), ∑ y ′ ∈ 𝐲 * P ( y ′ | 𝐲 , Ξ ) = 1 , which means that y ′ is randomly generated under a multinomial distribution with parameters determined by Eq. (5). Figure 2 shows two examples of how to calculate the conditional probability of sampling a new label according to Eq. (5) based on the label correlation network.

OPEN IN VIEWER

We denote P ^ ( 𝐲 | φ , Ξ ) as the second term of the RHS of Eq. (3):

P ^ ( 𝐲 | φ , Ξ ) = ∑ S L ∈ Φ 𝐲 P ( S L | φ , Ξ )(6)

This is the key point in Eq. s(3). sIn theory, P ^ ( 𝐲 | φ , Ξ ) should be computed by summing over all the L ! possible label orders’ probabilities, in which each order has L - 1 times of multiplication operation. According to the Stirling’s approximation:

L ! ≈ 2 ⁢ π ⁢ L ⋅ ( L / e ) L

the total computational requirements are on the order of L 3 / 2 ⋅ ( L / e ) L operations. This complexity greatly influences the computational efficiency, which is unaffordable. As a result, we propose an equivalent algorithm for calculating P ^ ( 𝐲 | φ , Ξ ) , which can easily alleviate the exorbitant computational requirements.

We appeal to finite state machine (FSM) to illustrate the algorithm. In an FSM associated with y, a state is defined to represent a subset of y. Each subset maintains a one-to-one correspondence to a state. There are three kinds of states:

Initial state: corresponds to the empty subset, ϕ ;

Suppose u is a subset of y, denote st(u) as the state which corresponds to u. Furthermore, we denote 𝐮 𝑝𝑟𝑒 as the precursor set of u, and denote st( 𝐮 𝑝𝑟𝑒 ) as the precursor state of st(u). 𝐮 𝑝𝑟𝑒 meets the following condition:

| 𝐮 𝑝𝑟𝑒 | + 1 = | 𝐮 | , 𝐮 𝑝𝑟𝑒 ⊂ 𝐮

where | ⋅ | represents the subset’s size, and there are | 𝐮 | precursor sets of u. A transition in the FSM refers to the probability of transiting from one state to another. A transition to st(u) can only comes from one of its precursor states. In addition, the transition probability is defined as the conditional probability of generating u when 𝐮 𝑝𝑟𝑒 is given, i.e. P ( 𝐮 - 𝐮 𝑝𝑟𝑒 | 𝐮 𝑝𝑟𝑒 , Ξ ) , which can be calculated by Eq. (5). We also define the state score of st(u) as P ^ ( 𝐮 | φ , Ξ ) , which is displayed in Eq. (6).

According to the above description, the FSM utilized in this paper can be represented by a directed acyclic graph. Figure 3 displays an example of an FSM associated with four labels. In the FSM, a complete path from the initial state st( ϕ ) to the final state st(y) corresponds to an order of all the labels in y. We define the path score as the product of all the transition probabilities, which is equal to the probability of generating the corresponding order of labels, i.e. P ( S L | φ , Ξ ) in Eq. (4). P ^ ( 𝐲 | φ , Ξ ) , the state score of st(y), can be calculated by summing over all of the path scores in the FSM without complexity constraints.

Reducing the complexity of calculating P ^ ( 𝐲 | φ , Ξ ) to an acceptable order presents a challenge. Fortunately, the paradigm of forward algorithm[19, 20] can be applied. The forward algorithm is widely utilized for hidden Markov model. We call our algorithm the forward sampling algorithm, due to its similar properties to the forward algorithm. Figure 3 shows that the state score of st(u) can be calculated by:

P ^ ( 𝐮 | φ , Ξ ) = ∑ 𝐮 𝑝𝑟𝑒 P ^ ( 𝐮 𝑝𝑟𝑒 | φ , Ξ ) ⋅ P ( 𝐮 - 𝐮 𝑝𝑟𝑒 | 𝐮 𝑝𝑟𝑒 , Ξ ) .(7)

This indicates that u’s state score involves two parts. The first one is the state score of st( 𝐮 𝑝𝑟𝑒 ), and the second one is the transition probability from st( 𝐮 𝑝𝑟𝑒 ) to st(u). This also holds true for y,

P ^ ( 𝐲 | φ , Ξ ) = ∑ 𝐲 𝑝𝑟𝑒 P ^ ( 𝐲 𝑝𝑟𝑒 | φ , Ξ ) ⋅ P ( 𝐲 - 𝐲 𝑝𝑟𝑒 | 𝐲 𝑝𝑟𝑒 , Ξ ) .(8)

From the perspective of the label rather than that of subset, Eq. (8) also has an equivalent formula:

P ^ ( 𝐲 | φ , Ξ ) = ∑ y ∈ 𝐲 P ^ ( 𝐲 ∖ { y } | φ , Ξ ) ⋅ P ( y | 𝐲 ∖ { y } , Ξ ) .(9)

The calculation of the state score does not require computing all of the path scores. Instead, this calculation only involves the state scores of its precursor sets and the transition probabilities. This divide-and-conquer strategy significantly alleviates computational complexity. The operation requirement can be reduced to L 3 / 2 ⋅ 2 L , which is an upper bound order. Although this is an exponential order, it is acceptable in most cases. On the one hand, the actual number of labels for a document may not be very large. On the other hand, the size of y, i.e. L , can be limited in a reasonable range, which is relevant to the classification strategy discussed in Section 6 that can further improve the efficiency of the multi-label classification.

For parameter learning, we first set up and initialized the label correlation network based on the observed multi-label classes in the training data set. Since the frequency of co-occurrence is a typical representation for the correlation between two labels, in the current paper, we directly employ the raw counts of co-occurrence for the weights’ initialization. If two labels co-occur in the training set, the weight on the corresponding edge is set to the co-occurrence count. In addition, if two labels do not co-occur in the training set, an edge exists between the two corresponding vertexes in the label correlation network, and we set the initial weight is set to one.

To obtain a more reasonable estimation for the prior probability P ( 𝐲 | λ , φ , Ξ ) in Eq. (2), all the weights in the label correlation network can be further optimized via utilizing certain criteria. In this paper, we choose the likelihood of all of the training data as the objective function, which maintains the following form:

ℒ ( Ξ ) = ∑ m = 1 M P ( 𝐲 m | λ , φ , Ξ ) .(10)

ℒ ⁢ ( Ξ ) involves three kinds of parameters, λ , φ and Ξ . Here, we only consider the weight parameter Ξ , while we regard λ and φ as constant parameters whose estimation approaches are discussed in Section 4.4. The weights could be optimized by maximizing the objective function ℒ ⁢ ( Ξ ) . However, since the objective function is highly nonlinear for each weight, identifying the close form solution is difficult. The generalized probability descent (GPD) algorithm [21] is employed to seek the optimal Ξ * . Denote ∇ ⁡ ℒ ⁢ ( Ξ ) as the gradient of ℒ ⁢ ( Ξ ) . The iterative GPD can be implemented as follows:

Ξ t + 1 = Ξ t + ε t ⋅ ∇ ⁡ ℒ ⁢ ( Ξ ) | Ξ t(11)

where t is the iteration number, Ξ t represents the parameters at t t ⁢ h iteration; ε t is the learning rate; ε t = ε 0 ⁢ ( 1 + t / T ) ; ε 0 is the initial learning rate; and T is the predefined maximum number of iterations.

We assign Ξ another equivalent definition for the sake of simplicity:

Ξ = { ξ ⁢ ( p , q ) | p ≠ q , 1 ⩽ p ⩽ K , 1 ⩽ q ⩽ K }

where ξ ⁢ ( p , q ) represents the weight on the edge between the two vertexes, which correspond to label y p and label y q , respectively. Suppose the weight ξ ⁢ ( p , q ) will be optimized. The gradient of ξ ⁢ ( p , q ) can be written as:

∂ ⁡ ℒ ⁢ ( Ξ ) ∂ ⁡ ξ ⁢ ( p , q ) = ∑ m = 1 M ∂ P ( 𝐲 m | λ , φ , Ξ ) ∂ ⁡ ξ ⁢ ( p , q ) = ∑ m = 1 M P ( L | λ ) ⋅ ∂ P ^ ( 𝐲 m | φ , Ξ ) ∂ ⁡ ξ ⁢ ( p , q ) .(12)

We focus on how to calculate the partial derivative of the state score P ^ ( 𝐲 m | φ , Ξ ) . Figure 4 displays four cases relevant to this calculation. The details of each case are described as (the subscript m is ignored for brevity hereafter):

If , then:

∂ P ^ ( 𝐲 | φ , Ξ ) ∂ ⁡ ξ ⁢ ( p , q ) = 0

OPEN IN VIEWER

In Eq. (13), P ( y | 𝐲 ∖ { y } , Ξ ) and P ^ ( 𝐲 ∖ { y } | φ , Ξ ) can be calculated utilizing Eqs (5) and (9), respectively. ∂ P ^ ( 𝐲 ∖ { y } | φ , Ξ ) ∂ ⁡ ξ ⁢ ( p , q ) can be deduced by Eq. (13) itself recursively. Therefore, from Eq. (13) we can observe that the paradigm of the forward algorithm can be reapplied to obtain the partial derivative. This resembles the inference process described in Section 4.2. We call this learning algorithm the forward gradient algorithm.

λ and φ are two vector parameters of multinomial distribution employed in the generative process S1. λ is a K dimensional vector ( λ 1 , λ 2 , … , λ K ) , in which λ k ( 1 ⩽ k ⩽ K ) represents the probability of containing k labels for a subset, and ∑ k = 1 K λ k = 1 . λ k can be easily estimated by c k λ / ∑ k = 1 K c k λ , where c k λ is the count of subsets that contain k labels in the training data set. φ is also a K dimensional vector ( φ 1 , φ 2 , … , φ K ) , where φ k ( 1 ⩽ k ⩽ K ) is the probability of attaining the k t ⁢ h label in the generative process for the first label in S1-(b), and ∑ k = 1 K φ k = 1 . φ k in the current paper is estimated by c k φ / ∑ k = 1 K c k φ , where c k φ is the count of the k t ⁢ h single label in the training data set. In our experiments, λ and φ are both smoothed via the laplace smoothing method [22].

5.1 Different choices

As described in Section 3, the label mixture model corresponds to S2, which formulates the generative process of words in a document with given labels. Many traditional generative mixture models can be chosen and embedded into S2, such as the mixture of language models, PLSA, LDA, etc. Different models maintain different generative process assumptions. In the current paper, we adopt LDA as the label mixture model. Figure 1d displays the overall graphical model. Since labels are visible in the training data, parameters of these models should be learned in a supervised manner, in which the labels of a document are restricted to belonging to a known label subset. This varies from the traditional models. The parameter learning procedure can be the same as that of the L-LDA. After parameter training, only { β k | 1 ⩽ k ⩽ K } , the parameters of labels’ multinomial distribution over words, are utilized in multi-label classification stage.

5.2 Fold-in process

Given a new document 𝐝 ′ , a label subset y for 𝐝 ′ , and the model displayed in Fig. 1d, the fold-in process can be applied to obtain the optimal joint probability of 𝐝 ′ and the vector variable θ , which is a multinomial mixture distribution over labels. This optimization problem can be written as:

θ * = argmax θ P ( 𝐝 ′ , θ | 𝐲 , α , β ) argmax θ P ( 𝐝 ′ | θ , 𝐲 , β ) P ( θ | 𝐲 , α ) .(14)

This fold-in process resembles that of PLSA [13, 10, 23], except for two differences. The first difference is that, in Eq. (14) the labels involved in θ are limited within y, whereas in PLSA, all of the labels are taken into account. The second difference is that the term P ( θ | 𝐲 , α ) should be calculated, while in PLSA only the term P ( 𝐝 ′ | θ , 𝐲 , β ) is considered. Regarding the optimization method, Gibbs sampling [24] or EM algorithm can be chosen to obtain the optimal θ * .

7.1 Data sets and evaluation measures

In our experiments, we utilize three original text classification data sets downloaded from the Internet to evaluate the performance of the LCMM: Reuters-21578 [25], TMC2007 [26] and OHSUMED.1 Reuters-21578 is a collection of articles that appeared in the 1987 Reuters newswire, a standard text classification benchmark. We utilize the Mod-Apte split, as descried in the “README” file accompanying the original set. This data set is named Reuters-all90 in this paper, and contains 90 labels. Two subsets of Reuters-all90 are also employed in these experiments. The subsets only utilize 10 and 36 labels [6, 15, 17], which are called Reuters-top10 and Reuters-top36, respectively. TMC2007 is a data set regarding aviation safety reports, comprising records of the problems that occurred during flights. OHSUMED includes medical abstracts from the MeSH categories of the year 1991. Table 2 presents the details of the data sets.

Various kinds of evaluation measures are employed: accuracy, precision, recall, F-score, micro/macro F-score. The details of these evaluation measures are described in [1, 27].

Simple preprocessing is conducted on each data set. Meaningless tags and numbers are removed. All words are converted to lowercase, and the words on a standard stop word list of about 400 words are removed. Stemming is performed by Porters stemmer [28]. In addition, term selection [29, 30] is performed utilizing the χ 2 (Chi-square) function with a reduction factor of 0.9 .

The label correlation model is trained according to the learning approach described in Section 4.3. The label mixture model is trained directly utilizing the JGibbLabeledLDA package2 with the default parameter configuration. Regarding the two-step multi-label classification strategy, the binary relevance method (BM) [1] is employed in the coarse selection step. The approach learns a set of binary classifiers for each label y k ( 1 ⩽ k ⩽ K ) and gives the 0 / 1 prediction of each label for a document. Here 0 means the label does not belong to the document, and 1 has the opposite meaning. Various kinds of traditional models can be employed to build the binary classifiers. In our experiments, the naive Bayes (NB) model [1] and the SVM model [29, 31, 32] are implemented. For the naive Bayes model, the binary classifier can be formulated as:

$$\log P(\text{𝐝}|y_k^1)P(y_k^1)-\log P(\text{𝐝}|y_k^0)P(y_k^0)>\tau ,\mathrm{\hspace{0.25em}1}⩽k⩽K$$

where τ is a constant that controls the classification boundary. When τ = 0 (default), this is the traditional naive Bayes binary classifier. Regarding SVM, the classifiers is implemented via the SVM light toolkit3 [32]. For the features utilized in the SVM model, we adopt the term weighting policy:

t ⁢ f ⁢ - ⁢ χ 2 ⁢ ( w n , 𝐝 ) = t ⁢ f ⁢ ( w n , 𝐝 ) ⋅ χ 2 ⁢ ( w n )

where χ 2 ⁢ ( w n ) is the chi-square of word w n , and

$tf(w_n,\text{𝐝})=\{\begin{array}{cc}1+\log \mathrm{\#}(w_n,\text{𝐝}),\mathrm{\#}(w_n,\text{𝐝})>0\hfill & \\ 0,\text{otherwise}\hfill & \end{array}$

# ⁢ ( w n , 𝐝 ) is the frequency of word w n in document d. This resembles the t ⁢ f ⁢ - ⁢ i ⁢ d ⁢ f term weighting approach.

In the coarse selection step, we should try to guarantee that the actual label are included in 𝐘 CS , so the configuration of the approach is different from the default configuration. For the NB classifiers, τ is set to - 80.0 to enhance the recall performance. For the SVM classifiers, the trade-off parameter (“-c” option) is set to 5.0 , the cost-factor (“-j” option) is set to 1.5 . All of the parameters are set empirically on the training data set.

We first investigate the learning procedure for the weights of label correlation network, Fig. 5 shows different convergence curves of objective function values corresponding to different ε 0 on the TMC2007 data set. The maximum iteration number T is set to 300 . We can observe that objective functions for different ε 0 increase during the iterative training. All these curves reach stable stages after approximately 100 iterations. These stable values increase with the ε 0 . However, when the ε 0 is larger than 0.8 , the corresponding objective functions almost converge to the same value. We can also see that the larger the ε 0 is, the more fluctuant the convergence curve is, especially at the beginning of the iterative learning. Similar phenomena can be also observed from the convergence curves on the Reuters and OHSUMED data sets, which are not displayed in this paper. In our experiments, we obtain the optimal weights of the label correlation network on different data sets utilizing the same learning configuration: ε 0 = 1.6 and 120 iterations.

Based on the optimized label correlation network, we implement two approaches denoted by BMNB CS + LCMM GPD and BMSVM CS + LCMM GPD , following the classification strategy described in Section 6. The BMNB CS represents the BM approach based on the NB model employing the coarse selection (CS) configuration, whereas BMSVM CS denotes the SVM-based coarse selection. In both approaches, the LCMM is adopted in the second refined assessment step, and we utilize the superscript “GPD” to denote the use of the optimized label correlation network. We compare the two approaches with other reported multi-label classification methods, including: Mixture Model (Mix-Model) [15]; CoL-Model [17]; ML-KNN [8, 17, 33]; Multinomial Model (Mul-Model) [34]; Boostexter [7, 17]; MFoM-Bin [5]; MFoM-MC [6]; Rocchio method (Roc-Method) [29]; KNN [29]; Ensemble-CC [3]; Ensemble-BM [3]; Ensemble-PS [3, 4]; RAKEL [3, 35].

OPEN IN VIEWER

Tables 3–5 presents the experimental results. Since different reported approaches were evaluated via different measures, and the data sets were also different, we list these reported experimental results and the corresponding ones of our approaches in the tables. The superscript “*” in the tables indicates the best reported results for a certain evaluation measure. We utilize the boldfaced value to denote the best result in its column.

From the three tables, we observe that BMNB CS + LCMM GPD achieves good overall performance, BMSVM CS + LCMM GPD outperforms all of the other methods. For Reuters data sets (Reuters-top10, Reuters-top36 and Reuters-all90), take the F-score measure as an example, The relative improvements are 61.8%, 47.6% and 38.8%, respectively. For the data sets TMC2007 and OHSUMED, performance is compared on the accuracy measure and the macro F-score measure; the relative improvements on TMC2007 are 3.8% and 15.0%, respectively; and for OHSUMED, the relative improvements achieve 28.3% and 37.0%, respectively.

OPEN IN VIEWER

We further analyze our proposed approaches via more comparison. For BMNB CS + LCMM GPD , we employ four methods:

BMNB: the BM approach based on NB model with the default NB configuration.

Similarly, we implement four approaches based on SVM methods to compare with BMSVM CS + LCMM GPD . The comparison is conducted on micro/macro F-score evaluation measures, and Table 6 of Appendix A presents the complete experimental results.

Since phenomena in NB-based methods and SVM-based methods are similar, this section only discusses NB-based methods, for illustration purposes. For the sake of clarity, we employ the performance of BMNB as the baseline, and compare all of the BMNB CS -based approaches with the BMNB. Figures 6 and 7 display the relative micro/macro F-score improvements. From the two figures, we can see that BMNB CS maintains the poorest performance in all of the data sets. Although it retains the best recall performance, its precision performance decreases significantly compared with BMNB. BMNB CS + LMM achieves slightly better performance than BMNB CS , which demonstrates the effectiveness of the second refined assessment step based on the label mixture model. The label correlation network significantly improves classification ability, which can be observed from the performance of BMNB CS + LCMM INI . On the whole, compared with BMNB CS + LCMM INI , the disadvantage of BMNB, BMNB CS and BMNB CS + LMM is that they do not model the correlation between labels. The BMNB and BMNB CS directly assume that labels are independent of each other. For the label mixture model in BMNB CS + LMM, labels are conditional independent when a label subset is given, which is similar to traditional topic models. The independency assumption for labels in these models limits the classification performance. Therefore, the performance comparison demonstrates the effectiveness of the label correlation model. In addition, the performance of BMNB CS + LCMM GPD illustrates that the parameter-learning algorithm can further enhance performance in most cases. However, the performance on Reuters-top10 and TMC2007 does not increase obviously, and even declines for the micro F-score measure on Reuters-top10. We analyze the data in the two sets. On the one hand, each multi-label class in the training data set maintains high frequency on average, and the distribution of multi-label classes determined by the initial label correlation network is, to some extent, consistent with that of the training data set. On the other hand, the two distributions of multi-label classes in the training and testing data set are also relatively close to one another. Therefore the parameter-learning procedure cannot significantly optimize the label correlation network, and the final performance of BMNB CS + LCMM GPD will not improve significantly on the testing data set compared with BMNB CS + LCMM INI .

OPEN IN VIEWER

From the perspective of Bayes decision theory, both the prior probability P ⁢ ( 𝐲 ) and the conditional probability P ( 𝐝 ′ | 𝐲 ) in Eq. (1) should be accurately calculated for classification. In LCMM, P ⁢ ( 𝐲 ) and P ( 𝐝 ′ | 𝐲 ) correspond to the label correlation model and the label mixture model, respectively. Some other methods [15, 16] only focused on the conditional probability P ( 𝐝 ′ | 𝐲 ) , which limited the multi-label classification performance. Compared with other approaches, one primary advantage of LCMM is that it can provide a reasonable estimation for the prior probability P ⁢ ( 𝐲 ) of an arbitrary label subset based on the label correlation model, which guarantees further performance improvement.

According to Eq. (3), if every multi-label class maintains only one single label, the multi-label classification problem reduces to a single label classification problem, and Eq. (3) is equivalent to direct estimation of the prior probability for each single label. However, for general multi-label classification problems, directly calculating the prior probability of a label subset is challenging, since the number of multi-label classes may be significantly larger than the number of observed samples. The label correlation model does not directly calculate the prior probability of a label subset. Instead, it estimates the probability of a label subset based on the correlation between pairs of labels, which are encoded in a label correlation network. The approximate computation can be implemented through a recursive algorithm according to Eq. (9), in which large, complicated problems are decomposed to a set of small, simple problems. The correlation between labels is involved in the process of estimating a prior probability, which is necessary for the multi-label classification.

According to Sections 3 and 4, if the labels in a multi-label class are connected in the label correlation network, the generative probability of this multi-label class can be calculated. For another case, in which a multi-label class does not exist in the training data set, if its labels are connected in the label correlation network, its prior probability can also be calculated. For this reason, as described in Section 4.3, although two labels do not co-occur in the training data set, an edge and a weight will be set between them, and the weight can be optimized by the learning algorithm. Therefore, the proposed label correlation model maintains a smoothing capacity for the unseen data.

We consider the LCMM to represent a basic framework of the generative model for multi-labeled data and the classification. Different configurations and submodels can be adopted in this framework. Regarding the labels’ generative process, S1, described in Section 3, it may not be the only generative solution to sample labels based on the label correlation network. We utilize the variable L to directly set the size of the label subset in advance. This might contradict the generative process under the condition that the label correlation network is an unconnected graph. Although the smoothing method discussed previously can avoid the unconnected case, it is probably more reasonable to incorporate a binary variable instead of L to control whether or not to finish the generative process. This binary variable can be independent of other variables, which represents the simplest case, in which only one parameter should be estimated. The variable can also be dependent on the current generated label subset, in which relevant parameters will significantly increase. The variable L employed in this paper can be regarded as a compromise between the previous two choices of the binary variable. Regarding the label mixture model, many other text generative models besides the mixture of language model, PLSA and LDA, can also be adopted. The model selection is intended to be task-oriented. These models can be easily integrated with the label correlation model to form an entire generative process for both labels and words.

In addition, the label correlation model and the label mixture model are relatively independent. In our experiments, we train the two models on the same training data set. However, this is unnecessary in practice, and the two models can be trained on different data sets. For the label correlation model, the training data set can only contain the multi-label classes. Regarding the label mixture model, both the documents and the corresponding multi-label classes are necessary.