Robust fused hypergraph neural networks for multi-label classification

2.1 Multi-label classification with deep learning

Zhang and Zhou [35] proposed the BP-MLL method which is the first multi-label classification method based on DNN. BP-MLL transforms the multi-label classification problem into several binary classification problems and utilizes the loss function to mine the label correlations. Gong et al. [9] trained deep convolutional neural networks for multi-label classification with a ranking-based strategy. The weighted approximated-ranking loss is shown to work particularly well for multi-label annotation problems. Wang et al. [22] proposed the CNN-RNN method for multi-label classification. CNN-RNN uses networks to convert both images and labels into a common latent space, while the label correlations can be mined by LSTM. Yeh et al. [31] proposed a multi-label embedding method based on deep learning and canonical correlations analysis. It can solve the multi-label classification problem with large-scale data well by using the deep neural network for spatial transformation. Chen et al. [2] proposed a deep architecture that consists of two parts, the spatial transformer and the LSTM sub-network. The two components perform alternately to locate the attentional region and capture the dependency on them. Wang et al. [24] proposed a model based on recurrent neural network which consists of a visual attention model and a confidence-ranked LSTM. This method can get proper label sequence automatically and has great robustness in label prediction. Chen et al. [3] proposed the ML-GCN based on graph convolutional network. The method utilizes the word-embedding representations to learn a classifier, and then second-order label correlations is mined by the correlations matrix, which is built in a data-driven way. Wu et al. [25] proposed a high-order semantic learning model based on adaptive hypergraph neural networks (AdaHGNN). The adaptive hypergraph is constructed based on the label embedding, and the hypergraph neural network can improve the ability of the multi-label classification model. Tan et al. [20] proposed a hypergraph-induced convolutional network (HI-GCN) which is inspired by the nature of hypergraph coding high-order connections. The model can effectively mine higher-order label correlations with high adaptivity and scalability. Xu et al. [29] proposed the TSGCN, which learns from both input and output spaces for multi-label image classification. TSGCN is the multi-label classification model conducted by mapping spatial object correlations and semantic label correlations.

2.2 Multi-label classification with noise in label space

The noise that exists in the label space includes false-positive noise and false-negative noise. Multi-label classification with false positive-noise alone is equivalent to partial multi-label learning, and multi-label classification with false-negative noise alone is equivalent to weak-label learning. We will briefly explain these two aspects here.

Partial multi-label learning is extended from partial label learning [26, 34], but the difficulty of partial multi-label learning is far greater than partial label learning, because there are multiple correct labels in the candidate label set for each instance in the partial multi-label learning framework. So mining the information of the complex label space effectively is the key to improving the performance of partial multi-label learning. For partial multi-label learning, not all the relevant labels in the given training label set are the really relevant labels in absolute truth. Xie and Huang [27] first introduced the concept of partial multi-label learning. They gave the new learning framework and its solutions. The method calculates the confidence for all the labels in the candidate set for each instance when performing the multi-label classification operation. Wang et al. [21] proposed the method of DRAMA which can be divided into two steps to solve the problem of partial multi-label learning. Firstly, the confidence of the labels corresponding to each instance is calculated, and then the confidence matrix and gradient superposition algorithm are used to build and optimize the model. Fang and Zhang [6] first used the propagation of label information to solve the problem of the confidence of the label matrix, and then they used the pairwise loss between the labels to build the model. Afterwards, the two methods named P-VLS and P-MAP are proposed based on different predictors.

For weak-label learning, the labels corresponding to some instances are incomplete, as not all irrelevant labels in the given training label set are the really irrelevant labels in absolute truth. Sun et al. [19] proposed a weak-label learning method named WELL which adopts the hypothesis of low-rank similarity matrix to mine the correlations among instances, and then completes the dissemination of the label information. Lin et al. [16] proposed a sparse reconstruction-based weak label learning method named LSR. This method gives an incomplete correlations matrix between instances and labels, then it uses instance correlations, label correlations, and correlations between instances and labels simultaneously with sparse constraints to complete the relation matrix between instances and labels. Xu et al. [30] proposed a weak-label learning algorithm named Maxide based on the low-rank matrix completion algorithm. This algorithm uses side information to reduce the observation information required for matrix completion, and can effectively complete missing label matrices in the state of weak-label. Dong et al. [4] proposed a weak-label learning method named SSWL based on ensemble learning, which considers feature space similarity and label space similarity to make up missing labels. This method integrates multiple different models through the collaborative regularization framework, which can improve the robustness of the model when the label information is insufficient.

Based on the above analysis, it can be found that the existing multi-label classification algorithms have paid attention to the label correlations from different aspects. However, most of the existing methods fail to effectively integrate and utilize the high-order label correlations and high-order instance correlations. Besides, how to flexibly improve the robustness of the model with the help of the above-mentioned higher-order correlations is also a key issue. Our proposed method can effectively utilize the high-order correlations of instances and high-order correlations of labels from the perspective of alleviating label noise to improve the accuracy and robustness of multi-label classification models, which is detailed in Section 3.

2.3 Hypergraph preliminaries

Let V denote a finite set of instances, and let E be a family of subsets e of V such that ∪_e∈E = V. G = (V, E) is then called a hypergraph with the vertex set V and the hyperedge set E. A hyperedge that contains just two vertices is just a simple graph edge. A weighted hypergraph is a hypergraph that has a positive number ω (e) which is associated with each hyperedge e called the weight of hyperedge e. We denote a weighted hypergraph by G = (V, E, ω). A hyperedge e is defined to be incident with a vertex v when v ∈ e. The degree of each vertex v ∈ V is defined as: d ( v ) = ∑ { e ∈ E rvert v ∈ e } ω ( e )(1)

Let rvertSrvert denote the cardinality of a given arbitrary set S. The degree of a hyperedge e ∈ E is defined as: δ (e) = rvertervert. A hypergraph G can be represented by a rvertVrvert × rvertErvert matrix H with entries h (v, e) =1, if v ∈ e and 0 otherwise, called the incidence matrix of G. Based on matrix H, the degree of each vertex and each hyperedge can be calculated as: d ( v ) = ∑ e ∈ E ω ( e ) h ( v , e ) , δ ( e ) = ∑ v ∈ V h ( v , e )(2)

Let D _v and D _e denote the diagonal matrices containing the vertex and hyperedge degrees respectively, and let W denote the diagonal matrix containing the weights of hyperedges [37].

The hypergraph structure is more complicated than the ordinary graph structure, and it is more difficult to use the hypergraph structure properly. At present, most of the work with the hypergraph structure are based on the hypergraph laplacian matrix. The derived hypergraph laplacian matrix is a positive semi-definite matrix, which integrates the information of the hypergraph well and is widely used. Following the random walk model, Zhou et al. [37] proposed the following normalized hypergraph Laplacian L: L h = I - D v - 1 / 2 HWD e - 1 H T D v - 1 / 2(3)

A hypergraph can describes the high-order correlations among instances. Feng et al. [7] proposed a hypergraph neural network framework which can deal with the the high-order correlations with the hypergraph for representation learning. Specifically, when there is a hypergraph signal X ∈ R^n×C₁ with n nodes and C₁ dimensional features, Y ∈ R^n×C₂ can be obtained after the following hyperedge convolution formulation: Y = D v - 1 / 2 HW D e - 1 H ⊤ D v - 1 / 2 X Θ(4) where W = diag (w₁, w₂, …, w _n ). Θ ∈ R^C₁×C₂ is the parameter to be learned during the training process. The filter Θ is applied over the nodes in the hypergraph to extract features.

3.1 Fused hypergraph neural networks (Module in the blue dashed box)

Here we introduce the fused hypergraph neural networks module in our model. It is used is to merge the hypergraph networks which are based on the feature space information and the label space information. Mining and using the structure information in the feature space and the label space can make the constructed model more accurate and discriminative. The construction of the fused hypergraph networks neural module can be divided into two steps as described below.

Firstly, we construct hypergraphs in both feature space and label space. In feature space, different from the simple graph where each edge represents the vertex-to-vertex correlation, the incidence matrix H ^f of a hypergraph describes the vertex-to-hyperedge correlation. To achieve this, we regard each instance as one vertex and try to generate a hyperedge for each vertex by following the method in [37]. More specifically, we generate the hyperedge e _i by the following formulation [40]: e i = { v j rvert θ ( x i , x j ) ≥ 0.1 σ i } , i , j = 1 , … , n(5) where θ (x _i , x _j ) indicates a similarity measurement between x _i and x _j while σ _i is the average similarity between x _i and each of the other instances (For example, in Fig. 1, the images with black, red, yellow, and blue grids have the similar feature, so they share the same hyperedge).

For the instance structure in the feature space, we use a hypergraph matrix H ^f to represent the high-order correlations among instances in the feature space. Based on the hypergraph matrix, we propose the corresponding hypergraph neural network. The main function of this hypergraph network is to use the high-order structure of the instance to learn better image features. Here we adopt X ∈ R^m×d with m nodes and d dimensional features to represent the initial deep feature extracted by CNN, then the hyperedge convolution can be formulated by: X k + 1 = σ ( D v - 1 / 2 H f WD e - 1 ( H f ) ⊤ D v - 1 / 2 X k Θ f k )(6) where W = diag (w₁, w₂, …, w _n ), D _v , D _e are the corresponding variables according to H ^f . Θ f k ∈ R d × d is the parameter to be learned during the training process, and σ (·) is a non-linear activation. After convolution, we can obtain the image feature X ∈ R^n×d that incorporates the high-order structural information of the feature space.

In label space, a hypergraph is built where each vertex corresponds to one training instance and each hyperedge for one label includes all the training instances that are relevant to the same label. (In Fig. 1, the black and red instances have the same label ’beach’, so they share the same hyperedge ’beach’). For the instance structure in the label space, we use a hypergraph matrix H ^l  = Y to represent the high-order correlations among instances in the label space. Based on the hypergraph matrix, we propose the corresponding hypergraph neural network. The main function of the hypergraph network here is to effectively use the high-order correlations of instances in the label space to improve the overall discrimination of the model. The hypergraph network that we propose here mainly acts on the results after classification. As a result, the high-order correlations of the instances in the label space will carry out effective label information dissemination. That is, the instances that are similar in the label space can get similar label sets, which will improve the prediction accuracy of our model. Here we adopt Y ˆ ∈ R m × n with m nodes and n dimensional features to represent the predicted labels, then the hyperedge convolution can be formulated by: Y ˆ k + 1 = σ ( D v - 1 / 2 H l W D e - 1 ( H l ) ⊤ D v - 1 / 2 Y ˆ k Θ l k )(7) where W = diag (w₁, w₂, …, w _n ), D _v , D _e are the corresponding variables according to H ^l . Θ l k ∈ R n × n is the parameter to be learned during the training process, and σ (·) is a non-linear activation. After convolution, we can obtain the prediction results Y ˆ ∈ R m × n that incorporates the high-order structural information of the label space.

Secondly, based on these two hypergraphs, we give the corresponding hypergraph neural networks. These two hypergraphs can make full use of the high-order correlations of the instances in the feature space and the label space. These two high-order correlations will act on the process of extracted deep features and the classified prediction labels, and lay the foundation for the following noise correction module (Section 3.3).

In this module, we first utilize Apriori algorithm to mine the second-order and high-order label correlations. Then we combine the label correlations with the word-to-vector representation of the label text to learn an effective classifier [3], which can improve the accuracy of classification while also providing assistance for the following noise correction in effect.

How to mine label correlations is very important in multi-label classification, as label correlations mined by the existing methods sometimes don’t have strong universality. This may decrease the robustness of multi-label classification. So we introduce the association rule mining algorithm in data mining to mine the second-order and high-order label correlations, which can alleviate the above problem. The association rule mining algorithm is an important branch of data mining. It generally uses the ’if-then’ logic rules to describe the rules and patterns of certain attributes in items from a large amount of data.

Our innovation focuses on the frequent item mining methods in data mining to mine co-occurrence correlations between labels and find the subsets of labels that often appear at the same time. We believe that there is a strong correlations among the labels that often appear at the same time. As a result, we create a second-order or high-order correlations for these labels to connect them. The Apriori algorithm is one of the most influential and widely used algorithms for mining frequent itemsets and association rules. This algorithm can generate strong association rules with frequent itemsets. When searching for frequent itemsets, the algorithm requires that all non-empty itemsets must be frequent. This step can filter out some co-occurrence noise in the multi-label datasets in advance, so it can make the second-order and high-order label correlations that we find be more robust and effective.

We take each label set for each instance as a data record, so we can mine the corresponding frequent co-occurrence label subset from these data, and then find the association rules. We set up a certain degree of confidence, and then generate some strong association rules. We regard these strong association rules as second-order or higher-order correlations, and then generate corresponding hypergraphs. Specifically, for the label matrix Y, we regard label sets for instances such as y₁, y₂, ⋯ , y _m as itemsets, and mine the frequent co-occurrence label subset that they contain. For example, (l₂, l₃, l₅) is a frequent label subset and then we can achieve a strong association rule based on it. As we repute that there is a high-order correlation among the three labels, we enclose the three labels with a hyperedge e₁. ( l 2 , l 3 , l 5 ) ⟶ e 1(8) We can achieve a hypergraph H ^c after the above process. Then we use the hypergraph neural network to incorporate such second-order and high-order correlations into our model, which can greatly improve the performance of our model. Different label correlations have different importance in the model. Here we use the confidence degree of association rules corresponding to the higher-order correlations to reflect this importance. As a result, we need to add the confidence matrix C ∈ R^n×n to the convolution operation.

The labels are represented by Z ∈ R^{n×d _wev} (n is the number of label attributes and d _wev is the dimensionality of word-embedding vector). The convolution operation can be written as follows. Z k + 1 = σ ( D v - 1 / 2 H c W D e - 1 CH c D v - 1 / 2 Z k Θ c k )(9) where C = diag (c₁, c₂, …, c _n ) is the confidence matrix of association rules, which is also regarded as the weight of hyperedges. σ (·) is a non-linear activation. W = diag (w₁, w₂, …, w _n ), D _v , D _e are the corresponding variables according to H ^c . Θ c k ∈ R d 1 × d 2 is the parameter to be learned during the training process. d₁ and d₂ are the dimensions of the middle layers of the networks. With the help of a multi-layer hypergraph neural network, we can get Classifer ∈ R^n×d from Z ∈ R^{n×d _wev} .

In multi-label application problems, there is often a certain amount of noise in the label space. They will affect the distribution of the label space, thus interfering with the training of the model. Therefore, effectively distinguishing and correcting the noise will greatly improve the accuracy of the learned model. This section designs a noise correction module (the module in the red dashed box in Fig. 1). The specific ideas are described as follows:

This section mainly designs the noise correction method based on the label transformation matrix T ∈ R^n×n, and combines it with the above two modules (Section 3.1 and 3.2). It is assumed here that the noise in the label space is class-conditional multi-label noise, that is, there will be a certain conversion correlations between the two labels (The probability of converting from y to y ˜ can be expressed as p ( y ˜ rvert y ) ), and this transformation correlations is independent of instance features [17]. This assumption is consistent with the noise that actually exists in the label space of multi-label datasets, because most labels in multi-label training datasets are manually annotated, and similar objects (label) are prone to be mislabeled in the manual labeling process.

Here we suppose that y^i* = 1 represents the instance x with the i class label, and y ˜ j = 1 represents the instance x is labeled with the j class label under noise interference, and the conversion between the two can be done by the label transformation matrix T ji * = p ( y ˜ j = 1 ∣ y i * = 1 ) to achieve. The label conversion matrix here T ∈ R^n×n is an asymmetric matrix. p ( y ˜ j = 1 ∣ x ) = ∑ i p ( y ˜ j = 1 ∣ y i * = 1 ) p ( y i * = 1 ∣ x ) = ∑ i t ji * p ( y i * = 1 ∣ x )(10) Suppose that the probability of the true label predicted by the classifier trained by the model is expressed as p ˆ ( y i * = 1 ∣ x , θ ) . Here the model is modified with the help of the label transformation matrix T, which contributes to make the predicted distribution of the model match the label distribution of the noisy data. p ˆ ( y ˜ j = 1 ∣ x , θ , T ) = ∑ i t ji p ˆ ( y i * = 1 ∣ x , θ )(11) The model with the added label transformation matrix consists of two parts: the basic prediction model (the parameter is θ) and the noise transformation matrix (the parameter is T), both of which need to be obtained by training the model. The transformation matrix T is directly introduced into the cross-entropy loss calculation, and the model is trained by maximizing the cross-entropy between the noisy label y ˜ and the prediction value from the model.

The noise transition matrix T is the core of the noise correction module, and it is important to learn an accurate noise transition matrix. T can be approximated by the following two steps: x ¯ i = argmax x ∈ X p ˆ ( y ˜ i = 1 rvert x )(12) T ij = p ˆ ( y ˜ j = 1 rvert x ¯ i )(13)

There are two issues that need to focus on here. The first point is that there is a hypergraph neural network before and after the noise correction module in the model separately. They have been described in detail in Section 3.1. These two hypergraph neural networks can effectively use the high-order correlations of instances in the feature space and the label space, and the high-order correlations of the instances in the feature space can effectively deal with noise in the label space. The second point is that we learn the noise transition matrix on the basis of the great classifier achieved in Section 3.2. We adopt the hypergraph neural network and apriori algorithm to obtain the classifier. On one hand, such a good classifier can help us get a better noise transition matrix, and on the other hand, it can also directly improve the accuracy of the classification of our proposed model.

From the above three parts, we can see that these three parts can be combined well with hypergraph neural networks, thereby they can greatly improve the classification accuracy, discriminability, and robustness of the model. These three parts are combined based on the hypergraph neural network to form an end-to-end model, and the first two parts (Section 3.1 and 3.2) lay the foundation for the noise correction part (Section 3.3). The most significant difference between our model and the existing models is that we make full use of the high-order correlations in the hypergraph neural networks, including the high-order correlations of the instances in the feature space and the label space, and the high-order correlations between the labels. We use the information in the training set as much as possible, which also determines that we will achieve a more powerful multi-label classification model.

For the instance x _i , if the instance belongs to the jth label, then y _ij  = 1; otherwise y _ij  = 0. The entire network will be trained based on the cross-entropy loss function, and the specific loss function is expressed as follows: L ( y i , y i ˆ ) = ∑ j = 1 n ( y ij log ( σ ( y ˆ ij ) ) + ( 1 - y ij ) log ( 1 - σ ( y ˆ ij ) ) )(14) where σ (·) is the sigmod function.

The general steps of model training are mainly divided into three steps. First, the noise correction module (Section 3.3) is closed, and only the fused hypergraph neural network module (Section 3.1) and the multi-label classification module (Section 3.2) are trained. The main reason for doing so is that the training of the noise correction module needs to be based on a relatively mature classifier. Then, the parameters of the fused hypergraph neural network module and the multi-label classification module are fixed, and the parameters of the noise correction module are learned. Finally, all modules are turned on at the same time for training and fine-tuning to obtain the best classification result.

4.1 Datasets and settings

To validate the proposed Robust Fused Hypergraph Neural Networks for Multi-Label Classification (RFHNN), we download four benchmark datasets for experiments, i.e., MS-COCO [15], VOC2007 [5], IAPRTC-12 [10] and ESPGame [11]. The statistics of the four real world datasets are summarized in Table 1.

Here we need to introduce the general structure of our experimental part. Our experiments can be mainly divided into three parts. (1) The first part is about the mining and verification of high-order label correlations. The high-order label correlations mining and utilization is one of the key parts in our proposed method. In this section, we verify that the Apriori algorithm can indeed mine the high-order correlations between labels. (2) The second part is the experiments on datasets without noise in the label space. Our proposed RFHNN will be compared with the latest multi-label classification algorithms based on deep learning. The experiments in this part are based on two datasets MS-COCO and VOC2007, and we don’t add additional noise to the label space of these two datasets. (3) The third part is the experiments on datasets with noise in the label space. For the fairness of comparison and to reflect the advantages of our method in noise correction, we conduct experiments on the training datasets with noise in the label space.

In our experiments, we follow ML-GCN [3] to set the experiments parameters, and the CNN module uses the ResNet-101 which is pre-trained on ImageNet. A two-layer HGNN is adopted with output dimensions of 1024 and 2048 to generate classifiers in our model, while the hidden layer is set as 1024 in FHConv. For word-to-vector of labels, we follow ML-GCN to choose 300-dim GloVe which is trained on the Wikipedia dataset. We choose LeakyReLU as the nonlinear activation function. For network optimization, we adopt SGD as the optimizer with a learning rate of 0.01. The weight decay is set to be 0.0001 and the momentum is 0.9. We implement the network based on PyTorch, and all the experiments are performed on a 64-Bit Linux workstation with an Intel E5-2650 CPU, an NVIDIA Titan X Pascal card, and 256GB memory.

In the multi-label learning problem, since each instance may have multiple category labels at the same time, the single-label evaluation metrics that are commonly used in traditional supervised learning, such as accuracy, precision, and recall, cannot be directly used for the performance evaluation of the multi-label learning system. Therefore, researchers have successively proposed a series of multi-label evaluation metrics. Here we consider four evaluation metrics, i.e., Macro-F1, Micro-F1, Average precision, and Hamming loss, which are widely used in multi-label learning to evaluate the prediction performance. Based on the symbolic representation in the definition of the problem, we denote Y _i as the set of related labels belonging to the instance x _i , then in order to characterize the binary classification performance of the predictors on each label, four basic quantities related to the test instance are usually used: TP _j (true positive), FP _j (false positive), TN _j (true negative) and FN _j (false negative). Macro - P ( CP ) = 1 n ∑ j = 1 n TP j TP j + FP j(15) Macro - R ( CR ) = 1 n ∑ j = 1 n TP j TP j + FN j(16) Macro - F 1 ( CF 1 ) = 2 × CP × CR CP + CR(17) Micro - P ( OP ) = ∑ j = 1 n TP j ∑ j = 1 n TP j + ∑ j = 1 n FP j(18) Micro - R ( OR ) = ∑ j = 1 n TP j ∑ j = 1 n TP j + ∑ j = 1 n FN j(19) Micro - F 1 ( OF 1 ) = 2 × OP × OR OP + OR(20) Average precision = 1 m ∑ i = 1 m 1 ∥ Y i ∥ ∑ y ∈ Y i ∥ y ′ ∥ rank f ( x , y ′ ) ≤ rank f ( x i , y ) , y ′ ∈ Y i ∥ rank f ( x i , y )(21)

Here, rank _f is the ranking function corresponding to the real-valued function f (·), and h (·) is the multi-label classifier we learned.

The mean average precision (mAP) [22] adopted in Section 4.3 is different from the Average precision adopted in Section 4.4. The mean average precision (mAP) is widely adopted in the evaluation of multi-label image classification and image retrieval, which is detailed as follows: AP ( l i ) = 1 m l i ∑ j = 1 m P l i ( j ) × ( R l i ( j ) - R l i ( j - 1 ) )(22) mAP = 1 n ∑ i = 1 n AP ( l i )(23) where m _{l i} represents the number of all instances associated with the label l _i , m represents the total number of instances in the dataset, j represents the serial number of the real-valued output of all instances under the classifier sorted in descending order, P _{l i}  (j) and R _{l i}  (j) represent the precision and recall corresponding to the first j instances.

For Macro-F1, Micro-F1, Average precision and mean average precision (mAP), the larger the values, the better the performance [36].

From the previous analysis, we can see that the high-order label correlations are crucial to the improvement of the accuracy of the multi-label classification algorithm. Our proposed method adopts the mature Apriori algorithm to indirectly realize the mining of label correlations by mining frequent itemsets, and then integrates the high-order label correlations into the learning process with the hypergraph neural network. In the whole process, the key point is whether we can successfully mine accurate high-order label correlations by Apriori algorithm.

In order to demonstrate that our method can effectively mine high-order correlations, here we use Apriori algorithm to mine the correlations among the labels in dataset IAPRTC-12. On different support levels, we can see that the high-order correlations mined are different in Table 2. For example, when we set support as 0.01, we can get the freqitemsets {bike, cycling, helmet, jersey, short, cyclist} and the six labels often appear at the same time. To some extent, this explains that there is a wide range in high-order correlations among labels. These high-order correlations contain rich information and are of great help to multi-label classification. In addition, we introduce the association rules generated under different confidence levels in Table 3. From our experiments, we can see that the Apriori algorithm can indeed mine the label correlations from label space, and from the semantic information of the labels, the label correlations correlations mined are reasonable.

To validate the proposed Robust Fused Hypergraph Neural Networks for Multi-Label Classification (RFHNN) on datasets without noise in label space, we consider the following evaluation metrics, i.e., average per-class precision (CP), recall (CR), F1 (CF1), average overall precision (OP), recall (OR), F1 (OF1) and mean average precision (mAP), which are detailed in Section 4.1.

In this part of the experiment, we compare our proposed approach with the following state-of-art multi-label classification methods: A Unified Framework for Multi-Label Image Classification (CNN-RNN) [22], Order-Free Rnn with Visual Attention for Multi-Label Classification (Order-Free RNN) [2], Multi-Label Image Recognition by Recurrently Discovering Attentional Regions (RNN-Attention) [23], Multi-Label Zero-Shot Learning with Structured Knowledge Graphs (ML-ZSL) [13], Learning Spatial Regularization with Imagelevel Supervisions for Multi-Label Image Classification (SRN) [39], Multi-Evidence Filtering and Fusion for Multi-Label Classification, Object Detection and Semantic Segmentation Based on Weakly Supervised Learning (Multi-Evidence) [8], Multi-Label Image Recognition with Graph Convolutional Networks (ML-GCN) [3] (both Binary and Re-weighted schemes) and Joint Input and Output Space Learning for Multi-Label Image Classification (TSGCN) (both Binary and Re-weighted schemes) [29].

Table 4 illustrates and compares the performances of the above methods on MS-COCO [15]. From the experimental results, we can draw the following interesting observations. The proposed RFHNN outperforms most of the state-of-art methods under the seven metrics, which validates our theoretical results. We can see that the proposed method is significantly better than the current CNN based methods, which proves that our proposed RFHNN can mine high-order label correlations accurately and improve the performance of classification. Especially when comes to the comparison between our method and ML-GCN, we can dig out and use more high-order information by hypergraph structure in the training dataset. This high-order information fits well with the characteristics of label correlations in multi-label learning, which can better help us perform multi-label classification.

Table 5 illustrates and compares the performances of the above methods on MS-COCO [15]. Following [3, 22], we use the training set to train our model, and evaluate the recognition performance on the test set. In order to compare with other state-of-the-art methods, we compare the results of average precision (AP) and mean average precision (mAP). The results on VOC2007 are presented in Table 5, and the results of many previous works on VOC2007 are based on the VGG model. For fairness, we also report the results that use VGG models as the base model. It is clear to see that our proposed method obtains improvements over the previous methods. Concretely, the proposed RFHNN obtains 94.6% mAP, which outperforms ML-GCN (Re-weight) by 0.3% and outperforms TSGCN (Re-weight) by 0.6%. Also, it achieves improvement on 9 out of the 20 labels. The dataset VOC2007 contains only 20 labels, and the number of tags is relatively small, so the high-order information among labels is relatively insufficient. Nevertheless, the existing high-order information also make help for our results, which fully illustrates the importance of high-order information.

To validate the proposed Robust Fused Hypergraph Neural Networks for Multi-label Classification (RFHNN) on datasets with noise in label space, we consider three evaluation metrics, i.e., Macro-F1, Micro-F1 and Average precision, which are widely-used in multi-label learning to evaluate the prediction performance of all the methods. The definitions of the three metrics can be found in Section 4.1 [36].

In this part of the experiment, we compare our proposed approach with the following state-of-art multi-label classification methods: Multi-Label Image Recognition with Graph Convolutional Networks (ML-GCN) [3], Multi-Label Learning with Global and Local Label Correlations (GLOCAL) [41], Partial Multi-Label Learning with Noisy Label Identification (PML-NI) [28], Multi-label Manifold Learning (ML2) [12] and Learning Deep Latent Spaces for Multi-Label Classification (C2AE) [31]. We also report the results of the baseline algorithm, Binary Relevance (BR) [1]. Besides, we have added a set of ablation experiments here to verify the ability of the anti-noise of our method. Classifier based on Hypergraph neural networks (CHNN) is proposed by omitting Fused Hypergraph Neural Networks (Module in the blue dashed box, Section 3.1) and Hypergraph Construction Noise Correction (Module in the red dashed box, Section 3.3) on the basis of RFHNN.

For the Robust Fused Hypergraph Neural Networks for Multi-Label Classification (RFHNN) that we proposed, there are currently no ready-made datasets with label noise for us to use, so we designed the datasets to verify the effectiveness of our proposed method [14, 32]. We selected two commonly used natural datasets i.e., IAPRTC-12 and ESPGame. These datasets have been divided into training and test sets in advance. For each dataset, to simulate false-positive noise and false-negative noise at the same time, we will randomly add some labels to the label matrix (randomly set negative labels (0) to positive (1)) to simulate the case false-positive noise, while we will randomly remove some labels from the label matrix (randomly set positive labels (1) to negative (0)) [32] to simulate the case of false-negative noise. Specifically, we make noise datasets according to the proportion of the total number of labels. In almost all commonly used multi-label datasets, the number of relevant labels for each instance is much smaller than the number of irrelevant labels [19]. In the experiment, the proportion of false-positive noise we added was 10%, 20%, and the proportion of false-negative noise was 10%, 30%. We used them in pairs so the experimental verification was carried out in four different situations (20% -50%). The dataset we designed is close to the actual scenario and can effectively verify the performance of our proposed method. For datasets of image, we extract 4096-dimensional deep features by using the ResNet-101 which is pre-trained on ImageNet, we didn’t perform the fine-tuning for fairness and computational efficiency.

Table 6 gives the illustrations of multi-label classification performance under the situation of noisy labels on the two datasets. The table can be divided into three parts longitudinally, which corresponds to three different evaluation indicators, i.e., Macro-F1, Micro-F1, and Average precision. The Noise column in the table refers to the scale of the noise of training datasets. From the experimental results, we can draw the following interesting observations:

OPEN IN VIEWER

Fig.2

Several multi-label image annotation examples on IAPRTC-12 datasets. For each image, we show the ground-truth annotations, and the labels predicted by the proposed RFHNN. The labels in black are those that match with ground-truth annotation. The blue labels denote the correctly predicted ones, while the red labels are those that are wrongly predicted. Besides, we use green labels to represent the labels that are correctly predicted but missing in the ground-truth annotations.

(1) The proposed RFHNN outperforms most of the state-of-art methods on the four datasets. For example, under the situation of noise = 20%, on dataset ESPGame, our method improves the best results of the baselines by 4.33% (Micro-F1), 3.41% (Macro-F1), which validates our theoretical results. Through the overall analysis of Table 6, we can find that when the noise increase, the performance of the method we choose to compare shows a sharp decline. In contrast, our method can still maintain relatively acceptable performance, which reflects the high accuracy and strong anti-noise ability of our method.

(2) An important characteristic of multi-label learning evaluation is that the results obtained under different evaluation metrics are quite different, so here we select four representative evaluation metrics, which include instance-based metrics and label-based metrics. We can find from the experimental results that there are some differences in the results between different methods under these four indicators. For example, the method that we proposed works well on Micro-F1 and Macro-F1, and the performance of GLOCAL method on Average precision is very prominent. Our method has generally good results on these four indicators, and some of the poor results are also within the acceptable range, which shows that our method has good stability.

(3) Label correlations is a very important topic in multi-label learning. Most of the existing multi-label classification methods have taken this aspect into consideration, but most methods just consider the second-order correlations, which will cause the loss of information. However, in the case of datasets with noise in label space, if the method is too sensitive to the correlations of the labels, the performance may even be worse than the simple method that does not consider the label correlations. This means that when we deal with noisy datasets, if we use an effective label correlations mining method, we must use an effective anti-noise module at the same time. Otherwise, the label correlations will play a negative role.

(4) Through the comparison of RFHNN with CHNN, we can find that the high-order correlations mining module (Section 3.1) and the anti-noise module (Section 3.3) play important roles in dealing with noisy datasets. CHNN is a degenerate version of our model, but it still has better performance than ML-GCN with the help of higher-order label correlations (Section 3.2). The performance of RFHNN on IAPRTC-12 and ESPGame is better than that on MS-COCO and VOC2007 in the comparison with ML-GCN, because our method makes better use of the information in the label space, and IAPRTC-12 and ESPGame have more kinds of labels and their label space information is more abundant.

To visually demonstrate the effectiveness of the proposed RFHNN, we present a case for further study in which RFHNN is applied to IAPRTC-12 dataset. The annotation results of several instance images are shown in Fig. 1 annotiation. The proposed RFHNN correctly predict most of the labels of these images, and RFHNN can even find the missing labels in ground-truth. For example, our method tags the image in row 1 and column 4 with the missing label ’building’. The performance shows that our method can work well in real-world image annotation applications.

From the above experimental results and analysis, we can find that our proposed method is superior to the state-of-the-art methods in most cases, which well demonstrates its effectiveness.