Content and structure based attention for graph node classification

Abstract

Graph-structured data is ubiquitous in real-world applications, such as social networks, citation networks, and communication networks. Graph neural network (GNN) is the key to process them. In recent years, graph attention networks (GATs) have been proposed for node classification and achieved encouraging performance. It focuses on the content associated on nodes to evaluate the attention weights, and the rich structure information in the graph is almost ignored. Therefore, we propose a multi-head attention mechanism to fully employ node content and graph structure information. The core idea is to introduce the interactions in the topological structure into the existing GATs. This method can more accurately estimate the attention weights among nodes, thereby improving the convergence of GATs. Second, the mechanism is lightweight and efficient, requires no training to learn, can accurately analyze higher-order structural information, and can be strongly interpreted through heatmaps. We name the proposed model content- and structure-based graph attention network (CSGAT). Furthermore, our proposed model achieves state-of-the-art performance on a number of datasets in node classification. The code and data are available at https://github.com/CroakerShark/CSGAT.

Keywords

Graph neural network graph attention network node classification graph-structured data

1 Introduction

Graph-structured data [37 , 47] expresses the relational model of real things. Thus, it received a growing amount of attention in recent years, due to the ubiquity of graph-structured data in real-world applications, including molecular structures [2, 42], social networks [28, 48], citation networks [8, 25] and protein interactions [19, 30]. Graph-structured data plays an important role in analyzing real-world models. Graph neural networks (GNNs) [18, 60], as generic deep learning models for graph structured data, have demonstrated great superiority on network analysis. GNNs aim to learn effective vector representation for each node by taking into account the graph structure and aggregating information from its neighbors. The learned representations will be further utilized for the classification task. It is worth noting that graph-structured data is usually only partially known, that is, network analysis focuses on predicting unknown information [26 , 46]. Semi-supervised node classification on graphs [9] means at training time we only know the labels of a subset of the graph nodes, and its task aims at predicting the labels for the unlabeled nodes. An efficient GNNs model is the key to successfully predict unknown labels, which means that GNNs need to make full use of the known information in the graph, including node content and topology [16, 61].

Message passing neural network(MPNN) [11, 58] is a general framework for the working mechanism of graph neural network. The power of current GNNs largely relies on their message-passing mechanisms [24], which recursively update the features of a node by aggregating the received features from its neighbor nodes. In some research, messages are passed along edges uniformly without accounting for priority of either graph structure or node attributes [55]. However, each neighbor node’s impact is distinctive to the center node in the node classification task. Thus, attention-based GNNs are proposed to further evaluate how the contribution of neighbors to the center node varies. GAT [35] is the first attempt to introduce attention module into graph node classification, in which the weighting coefficients for each neighbor are calculated based on a self-attention strategy. Despite the numerous success, GAT computed the attention weighting coefficients mainly based on node attributes. Those strategies may be suboptimal because they neglect the importance of the structure information in the graph [20]. Note that graph consists of two pieces of information, node content and network structure. For example, Fig. 1 depicts the node content and network structure of a graph. Imaging that ww need to update the features of node A, the features of node B and C should be aggregated. In Fig. 1(a), it can be found through observation that node A and C have higher similarity in node content, which means that they are more likely to be in the same category. This information is useful for graph learning. However, due to the tradeoff between efficiency and performance, the content based attention mechanism may be accompanied by the loss of important information when mapping content, which interferes with the weight allocation. In Fig. 1(b), you can intuitively see that node C is more impactive than node B, because node A and C belong to the same community, while A and B do not. This structural information cannot be extracted by content based attention. However, it has important guiding significance for neural network to focus on important nodes.

Fig. 1

Presentation of a graph data example.

The example mentioned above demonstrates that content and structure information have important impact on achieving a more discriminative node representation. However, the existing neural network models cannot take into account both content and structure information, so there are great limitations. In this paper, we propose a novel attention-based algorithm for semi-supervised node classification. The algorithm includes a hybrid attention mechanism, in which node content and structure information are all considered to compute the attention weights. This method can make up for the defects of the existing model effectively and make full use of the information in the graph. Specifically, node features among nodes are employed for calculating the content based coefficient, while structure in adjacent matrix are considered to compute structure based coefficient. Then, two types of coefficients are combined to form attention weights. These weighs are used to linearly combine the feature vectors of neighbor nodes, then the features of the central node are updated. Note that friendly neighbor nodes are better discovered through this hybrid attention mechanism. Extensive experiments verify that combining those two types of information deduce the more effective performance for semi-supervised node classification. Importantly, the mechanism is lightweight and efficient. It can accurately calculate the weight based on the high-order structure without complicated learning tasks, which makes the proposed model further improve the performance in the task while maintaining the efficiency. Moreover, our model is more explanatory, and ablation analysis and heatmap are used to support the model in the experimental part.

2 Related works

2.1 Graph neural network

Convolutional neural networks (CNNs) were used to extract features from images [15 , 51]. Motivated by the effectiveness of CNN, the graph convolution operation is designed to extract information from graphs, which generally falls into two categories: spectral and spatial methods [6, 43]. Bruna et al. [14] utilized Fourier base vector to perform convolution in the spectral domain. ChebNet [27] introduced a well-approximated K-order Chebyshev polynomials for convolution operations, in which the computationally expensive Laplacian eigen decomposition is removed. Spatial methods are defined by a message-passing mechanism operating on the target node and its topological neighbors. The most representative early work is the algorithm GCN [45]. It conducts the layer-wise convolution operation iteratively in order to expand a node’s neighborhood, so as to aggregate the neighbor nodes’ feature to update its representation. Different from GCN, GraphSAGE [49] generates a node’s representation by aggregating features from a fixed number of neighbors, which can deal with inductive learning for unseen nodes or entirely new graphs. Further, GNNs have been widely studied from various perspectives, including over-smoothing problem [7], the adversarial attack and defense [50], and the model explainability [13]. Simplifying graph convolutional network (SGC) [10] considers that GCN may inherit unnecessary complexity and computation. Therefore, it removes nonlinearities between GCN layers, resulting in a linear model for repeated feature transformation and propagation. Labeling real data has a huge cost, but when labels are limited, GNN performance suffers a severe drop. Contrastive graph poisson networks (CGPN) [40] improves model performance by efficiently propagating limited labels throughout the graph. K-hop graph neural networks (k-hopGNNs) [12] update the representation of nodes by aggregating information from its k-hop neighborhood and immediate neighbors, which solves the problem that GNNS cannot recognize basic graph properties. The framework recognizes connectivity and triangular degrees of freedom. Two-level graph neural network (TL-GNN) [52] proposes a novel subgraph counting method based on dynamic programming algorithm that combines subgraph level and node-level information to enrich the features captured by the graph neural network, such as high-level information and node-level information. Furthermore, it overcomes the characterization limitation of existing neural networks, which is caused by local permutation problem. Multiresolution reservoir graph neural network (MRGNN) [23] reduces the high computational cost in the process of model training by generating explicit k-hop unsupervised graph representation processing, and excessive smoothing will occur in reservoir calculation when iterating graph structure. In general, this algorithm has absolute efficiency advantage over traditional algorithms, and maintains the performance. Weighted feature fusion of convolutional neural network and graph attention network (WFCG) [53] integrates convolutional neural network and graph neural network with the goal of complementary advantages, so as to fully explore high-dimensional features and achieve excellent results in hyperspectral images. Meta-learning spatial-temporal graph attention network (MetaSTGAT) [32] captures the dynamic changes of nodes in the graph based on the graph neural network, and then performs dynamic weighting.Furthermore, the model also takes advantage of the potential correlation between spatial and temporal information through graph neural network to improve the effectiveness of the model.

2.2 Attention mechanism

It is intuitive that the impact of each neighbor node is different when aggregating its features to the central node. To deal with this problem, the algorithm GAT utilizes a self-attention mechanism to learn the different contributions from each neighbor node and aggregates information from these neighbors based on the learned weights. Graph attention mechanisms have been used in many directions, such as targeted drug research [41, 59] and molecular property prediction [4]. Li et al. [29] demonstrated that GAT is dataset-dependence and the distributions across heads and layers are uniform. They adapted GAT by proposing margin-based constraints to reduce information propagation among nodes belonging to different classes. However, in most current attention-based approaches, the learned aggregation weights are based on node features, and they neglect other useful information inheriting in graph structure. Multi-view graph attention networks (MGAT) [56] explore an attention-based model that learns node representation from each view, and design a novel regularization term constraint for this algorithm, in order to cooperatively integrate multiple types of relationships in different views. To improve the generalization performance of the model. It can capture various relationships in the multi-view network, and then update the low-dimensional representation of nodes, and finally characterize the complex types of proximity relationships in the graph. Graph conjoint attention networks (CATs) [1] have proposed combining content and structure intervention to calculate attention score, and good results are achieved. However, fatally, it requires a large number of heavy and complex learning tasks to obtain additional representations of nodes, and the experiment lacks a convincing explanation. Structural attention network (SAN) [1] uses transition matrix to extract structural information among nodes. In order to distinguish the structure of multi-order in the graph, the output features of nodes in the graph are expressed as series of multi-order features. Although structural information is beneficial to the weighting process of the model, the node content information of the model is ignored in the weighting process, so there are limitations. Sparse graph attention networks (SGATs) [57] found that Gats were prone to overfitting and could not avoid noise interference when dealing with a large number of graph structure data. Therefore, SGATs sets activation state for connections by generating edge sparse maps, thus eliminating noise connections and maintaining model classification accuracy. Path reliability-based graph attention networks (PRGATs) [54] use multi-hop adjacent context information in the calculation of attention scores, which further expands the receptive domain of the network and captures the dependence relationships in a farther range. Solved the problem that GAT could not capture multi-hop neighbor information. Secondly, the model also uses an attention layer based on path reliability to further improve the performance of the graph against attacks. However, these models hardly fully exploit the potential of attention mechanism. Therefore, we propose to use both content and structure information to learn better attention patterns for node classification. We use a lightweight attention mechanism that greatly reduces the learning stress of the model, is more interpretive, and yields better results.

3 The proposed algorithm

How to learn a new feature representation for each node over multiple layers is an important issue for graph node classification. In this section, we introduce the proposed incorporated attention mechanism to do it, in which two primary components are involved for attention computation: graph structure and node attributes. In other words, we believe that the weight required for the information aggregated by a central node from its neighbors depends on these two components. Figure 2 depicts the architecture of the proposed incorporated attention mechanism. The upper part of Fig. 2 illustrates the generation of content coefficient, while the lower part illustrates the generation of structural coefficient. Then, those two type of coefficients are combined to be used in the message passing step to update node features. The downstream task-node classification is done by means of a softmax classifier based on the final representation.

Fig. 2

A simple architectural diagram of the proposed model.

For convenience, mathematical preliminaries and notations used in the paper are firstly illustrated. Given a graph G (V, E) with a set of nodes V and a set of edges E. Each node i has an original feature vector x _i and a true label y _i. The matrix X = [ x ₁, x ₂, …, x _n] ^T stacks all node vectors on top of one another. Furthermore, let A = [ a ₁, a ₂, …, a _n] ^T denote the adjacency matrix. For node i, N_i contains the set of indices for its 1-hop neighbors. We only know the labels of a subset of the nodes, and the proposed algorithm aims to learn a model $Z = f (X, A) \in ℝ^{n \times l}$ that predicts the unknown labels. For each message passing step l, let $h_{i}^{l - 1} \in ℝ^{F}$ denote the input features for node i, then its output vector is . Naturally, the initial node representations are X. We elaborate our algorithm in the following.

3.1 Content coefficient generation

We follow the method introduced in GAT for content coefficient generation. For a L-layer model, the content coefficient f_ij is calculated as Equation (1). f_ij describes the relevance between two adjacent nodes i and j, which is normalized among its neighbourhood.

$f_{ij} = \frac{\exp (σ (α^{T} \cdot [W^{l} h_{i} | | W^{l} h_{j}]))}{\sum_{k \in N_{i}} \exp (σ (α^{T} \cdot [W^{l} h_{i} | | W^{l} h_{k}]))},$ (1) where ∥ is the concatenation operation, and σ (·) is a non-linear activation function, such as a LeakyReLU. and $α \in ℝ^{2 F}$ are parameters which need to be tuned in the training. The bigger the value of f_ij, the more important the relevance between node i and j, relative to other neighbors.

3.2 Structure coefficient generation

Considering the adjacent matrix A that describes the structural information, we generates the structure coefficients from it. Please note that A needs to add a self-connection, because the node’s own information is the key to node feature update. For node i, a_i ∈ A denotes the adjacency information between it and the other nodes. Therefore, a intuitive way to evaluate the structure coefficient between node i and node j is shown as Equation (2).

$s_{ij} = \sum_{k = 1}^{V} (\frac{a_{ik}}{\sum_{m = 1}^{V} a_{im}} \frac{a_{jk}}{\sum_{n = 1}^{V} a_{jn}}) .$ (2)

Note that the adjacency matrix A needs to be normalized, which guarantees the value of s_ij is between 0 and 1. The bigger the value of s_ij, the more similar their structure is. For example, if node i and node j share lots of neighbors or they allocate in the same community, s_ij will be large.

3.3 Incorporating content coefficients and structure coefficients

In order to incorporate content coefficients and structure coefficients, we introduce two learnable parameters g_f and g_s to adjust the relative importance between s_ij and f_ij. r_f and r_s are normalized results of g_f and g_s respectively, as shown in Equations (3) and (4).

$r_{f} = \frac{\exp (g_{f})}{\exp (g_{f}) + \exp (g_{s})} .$ (3)

$r_{s} = \frac{\exp (g_{s})}{\exp (g_{f}) + \exp (g_{s})} .$ (4)

Then, the attention weights are computed as

$e_{ij} = \frac{\exp (r_{f} \cdot f_{ij} + r_{s} \cdot s_{ij})}{\sum_{k \in N_{i}} \exp (r_{f} \cdot f_{ik} + r_{s} \cdot s_{ik})} .$ (5)

In Equation (5), r_f and r_s are used to control their proportions between f_ij and s_ij. e_ij measures the influence of node j on node i. Note that e_ij is different from e_ji, because node i and node j have different neighbors.

3.4 Feature updating

After getting attention weights from the center’s neighbors, it features can be updated according to the messaging passing mechanism.

$h_{i}^{l + 1} = σ (\sum_{j \in N_{i}} e_{ij} W^{l} h_{j}^{l}) .$ (6)

To make the self-attention calculation more stable, we applied multi-head attention for feature transformation and concatenated the resulting features as the returned output:

$h_{i}^{l + 1} = \begin{matrix} K \\ ‖ \\ k = 1 \end{matrix} σ (\sum_{j \in N_{i}} e_{ij}^{k} W^{l} h_{j}^{l}) .$ (7)

We set K = 8. Also, we used the Dropout mechanism and set it to 60 percent. This means that during model training, the activation value of the node stops working with a certain probability. Because this mechanism alleviates the over-dependence of some local features, it improves the model generalization. Secondly, this measure will reduce the complexity of the model in the forward propagation process and improve the training efficiency of the model. In addition, it can effectively alleviate the occurrence of overfitting and achieve the regularization effect to a certain extent.

After updating each node through information aggregation, we can obtain the corresponding class label of each node. In general, the dimension of the node representation obtained through the neural network is the same as the number of categories of the dataset. After processing by softmax, each element in the node representation represents the probability that it belongs to the corresponding class, and the class with the highest probability is the predicted result.

In the semi-supervised node classification task, we use standard cross entropy as the objective function, and use backpropagation to optimize the parameters in the model. Our goal is to minimize the inconsistency between the ground truth label and the prediction label. The specific formula is as follows:

$L = - \sum_{i \in T} \sum_{k \in C} y_{ik} log {\tilde{y}}_{ik} + λ {∥ W ∥}^{2},$ (8) among them, y and $\tilde{y}$ represent true label and predicted result, respectively. C represents the collection of categories, T is the set of training nodes. We obtain the optimal model through multiple epochs of learning. λ is a regularization parameter, which controls the complexity of model parameters, so as to prevent overfitting and improve the generalization ability of the model.

3.5 The pseudocode

For the node classification, the cross entropy loss is adopted, in which the inconsistence between the ground true labels and the predicted labels is minimized. In the training state, we construct the two-layer network structure for the proposed approach. The first layer utilizes multi-head attention. The output of the second layer is used to predict, so multi-head attention is no longer sensible. The whole process is described in Algorithm 1. Note that it would be run for multiply epochs to achieve convergence.

4 Experiments

We have validated our model against a wide variety of state-of-the-art baselines and previous approaches, and achieved the best performance across all of them. The dataset, comparison algorithms, results and qualitative analysis are included in this section.

4.1 Datasets

We conducted experiments on three widely-used datasets, including Cora, Cite and Pubmed [21, 34]. They are standard citation network, in which nodes represent documents and edges represent citations. The original node features correspond to elements of a bag-of-words representation of a document. And, each node has a class label. All datasets are splitted to three parts: training, validation and testing. Considering the huge amount of nodes and edges in Pubmed, a connected subgraph is selected for our experiments. Table 1 showns the statistics of these datasets.

Table 1
Datasets statistics

Datasets Nodes Edges Features Classes Training Validation Testing

Cora 2708 5429 1433 7 140 500 1000

Cite 3327 4732 3703 6 120 500 1000

Pubmed 3356 4278 500 3 60 500 1000

Datasets	Nodes	Edges	Features	Classes	Training	Validation	Testing
Cora	2708	5429	1433	7	140	500	1000
Cite	3327	4732	3703	6	120	500	1000
Pubmed	3356	4278	500	3	60	500	1000

4.2 Comparison algorithms

We compared out method with nine typical algorithms, including multilayer perceptron (MLP), Deepwalk, text-associated deep walk (TADW), deep graph infomax (DGI), GCN, edge enhanced graph neural network (EGNN), CAT, heat kernel graph convolutional network (HKGCN) and GAT. For convenience, we name the proposed model as content- and structure-based graph attention network (CSGAT).

MLP is a three-layer artificial neural network, including input layer, hidden layer and output layer. Different layers of MLP neural network are fully connected. It is the first neural network proposed to solve classification problems. We set the number of hidden units to 64.

Deepwalk 1 is a representative and successful work of early network representation learning. Deepwalk learns a social representation of a network from truncated random walks. It is also a method that only considers graph structure information when learning graph representation.

TADW 2 [5] takes advantage of the extra attribute information of nodes by designing related algorithms to simultaneously consider and embed the text information of nodes in the graph network into the final low-dimensional distributed representation.

DGI 3 [36] is a node level graph representation learning architecture that enables unsupervised learning. In order to obtain node representations, it trains an encoder model to maximize the mutual information for capturing the global information content of the entire graph.

GCN 4 [45] is a semi-supervised graph convolutional network, which introduces a convolution kernel that can be optimized to optimize the convolution parameters in the training process and achieve accurate feature extraction. It encode a localized approximation model for spectral graph convolutions by terms of layer-wise feature propagation, linear transformation, and pointwise nonlinear activation.

EGNN 5 [22] is a novel graph convolution framework, which solves the problem that GCN does not make full use of multiple edge features. It uses edge features across network layers to make more efficient use of one-dimensional edge features, and modifies the existing model to make use of multi-dimensional edge features using double random normalization.

CAT-I-MF 6 [44] is a conjoint attentions network. It incorporates the feature-based attention with structural interventions which may describe the global relation among nodes. It uses matrix decomposition to solve the objective function approaching adjacency matrix to obtain the structure representations. The resulting representations are used to capture the weighted mean attention with feature-based correlation.

HKGCN 7 [17] generalizes the SGC as a linear model by thermonuclear. SGC simplifies the model by removing the nonlinearity and weight matrix of all continuous layers in GCN. In this way, HKGCN can be used as a low-pass filter, which is conducive to the fast smooth convergence of feature propagation, and then the aggregation of local information. The performance of the model is improved while maintaining the efficiency advantage of the model.

GAT 8 [35] is a semi-supervised neural network which considers the attention mechanism. To compute the hidden representations of a center node, it employ a self-attention mechanism for assigning weights to the neighbors. However, the weight distribution is based on the content of low-order nodes, and the structure information is completely ignored, so the model has limitations.

4.3 Parameter settings

Our uses a two-layer model. The first layer is multi-head convolution, which consists of eight graph convolutions. Before entering the second layer, we concatenate the output of the first layer. The second layer consists of a single graph convolution, followed by a softmax activation, and its output is used for the classification task, and the element corresponding to the maximum value of the final output is the prediction result. To prevent overfitting, regularization is used. During training, we use λ = 0.0005 for Cora and Cite datasets, λ = 0.001 for Pubmed. Also, Dropout [31] p = 0.6 is applied to the input of each layer. All the above measures can effectively prevent overfitting and improve the generalization performance of the model.

The training objective is to minimize the cross-entropy on the training nodes, and the learning rate is 0.005 for all datasets except Pubmed which is 0.01. During the training process, we used an early stop strategy for the cross-entropy loss of the validation nodes, with a patience of 100 epochs. That is, if the validation loss does not decrease within 100 epochs, we stop training.

4.4 Evaluation

The experimental results of all methods are evaluated by Micro-F1, Macro-F1 and Weighted-F1. In those evaluations, precision P and recall R will be used in the calculation process.

$P = \frac{TP}{TP + FP} .$ (9)

$R = \frac{TP}{TP + FN} .$ (10)

Among them, TP is the number of instances which are positive and predicted correctly. FP is the number of instances which are negative but predicted to be positive. And, FN is the number of instances which are positive but predicted to be negative.

Equations (9) and (10) are calculated over all tested instances, which means that they do not need to distinguish between categories. Micro-F1 directly uses the precision p and recall R to calculate the F1-score:

$Micro - F 1 = \frac{2 PR}{P + R} .$ (11)

If Equations (9) and (10) are calculated for each category, Macro-F1 can be calculated based on them. Specifically, denote P_e and R_e as precision and recall for class label e, respectively. Then, Macro-F1 can be calculated:

$Micro - F 1 = \frac{\sum_{e \in C} \frac{2 P_{e} R_{e}}{P_{e} + R_{e}}}{| C |} .$ (12)

Weighted-F1 considers the importance of different categories. The ratio a_e of the number of instances in each category e to the total number of instance is used as the weight to calculate it:

$Weighted - F 1 = \sum_{e \in C} a_{e} \frac{2 P_{e} R_{e}}{P_{e} + R_{e}} .$ (13)

4.5 Result analysis

For each set of testing data, all aproaches are run ten times to obtain the statistically steady performance. Table 2 records the summary of results in terms of Micro-F1, Macro-F1 and Weighted-F1 over three datasets. For each measure, we highlight the best results over each dataset in bold. MLP is a network embedding method based on word vector information extraction. Deepwalk is a random walk-based network embedding method. We can see that graph weighted convolution based methods like GCN, GAT, etc. outperform MLP and Deepwalk. DGI is an unsupervised model, so its model parameters do not perform as well as semi-supervised models in the absence of effective optimization. Interestingly, EGNN exploits edge features but does not yield overall performance advantages over GCN. CAT uses structural interventions to learn new node representations in model training. However, the lack of accurate information extraction mechanism makes the performance greatly limited. HKGCN, as a simplified version of the GCN model, not only improves the model efficiency, but also achieves stronger performance in some experimental structures. It is worth noting that GCN based on topological weighting and GAT that introduces node content attention mechanism usually have good performance. However, our method effectively improves GAT, making its graph structure information and node content a dual evaluation metric for the attention mechanism. This enables our model to discover meaningful neighbor nodes better than traditional method assignments. In addition, the structure-based attention used by the model is lightweight and efficient. It does not need to be trained in the process of model training, but only needs to simply traverse the graph structure before training, so that the structural information can be effectively extracted accurately. This mechanism can effectively extract higher-order structure information. In the following chapters, we will use heatmap to visualize it to prove its effectiveness. CSGAT algorithm demonstrates the best performance being achieved across all three datasets. It is worth mentioning that because we adopt more advanced attention mechanism compared with GAT, we can optimize the model parameters more efficiently, which speeds up the convergence rate of the model, which means that the training time is effectively shortened. In conclusion, the experimental results verify that the mixture of content-based and structure-based attention is effective.

Table 2
Experimental results in terms of F1-Score

Cora Cite Pubmed

Micro-F1

MLP 56.0±0.1% 62.8±0.1% 66.5±0.1%

Deepwalk 60.6±0.8% 40.9±0.8% 53.2±0.5%

TADW 46.2±0.7% 50.1±0.9% 66.1±0.7%

DGI 75.3±0.4% 71.7±0.4% 71.1±0.2%

GCN 82.1±0.6% 71.8±0.5% 74.2±0.7%

EGNN 81.3±0.4% 69.7±0.3% 75.7±0.3%

CAT-I-MF 83.5±0.5% 72.0±0.3% 75.5±0.7%

HKGCN 82.8±0.1% 66.2±0.2% 74.3±0.2%

GAT 83.8±0.2% 71.5±0.2% 74.8±0.1%

CSGAT 84.6±0.2% 72.7±0.2% 76.7±0.3%

Macro-F1

MLP 50.5±0.1% 56.4±0.1% 64.0±0.1%

Deepwalk 58.0±0.6% 36.3±0.9% 46.9±0.6%

TADW 56.1±0.8% 58.3±0.8% 68.7±0.7%

DGI 69.8±0.7% 61.6±0.5% 67.7±0.3%

GCN 79.5±0.7% 63.2±0.4% 71.6±0.8%

EGNN 78.3±0.4% 61.2±0.3% 74.0±0.3%

CAT-I-MF 82.3±0.4% 65.0±0.4% 74.1±0.8%

HKGCN 80.2±0.1% 60.2±0.2% 73.1±0.2%

GAT 82.4±0.2% 64.4±0.2% 73.6±0.1%

CSGAT 83.1±0.1% 66.0±0.3% 75.8±0.4%

Weighted-F1

MLP 55.8±0.1% 61.0±0.1% 65.5±0.1%

Deepwalk 60.9±0.7% 39.3±0.8% 51.3±0.5%

TADW 50.1±0.7% 55.1±0.6% 65.8±0.5%

DGI 74.2±0.5% 68.8±0.5% 69.7±0.2%

GCN 82.1±0.6% 69.6±0.6% 73.4±0.8%

EGNN 81.4±0.4% 67.4±0.3% 75.2±0.2%

CAT-I-MF 83.4±0.4% 70.6±0.3% 75.0±0.7%

HKGCN 83.0±0.1% 65.1±0.2% 74.0±0.2%

GAT 83.8±0.1% 70.0±0.2% 74.5±0.1%

CSGAT 84.7±0.1% 71.3±0.2% 76.4±0.3%

	Cora	Cite	Pubmed
		Micro-F1
MLP	56.0±0.1%	62.8±0.1%	66.5±0.1%
Deepwalk	60.6±0.8%	40.9±0.8%	53.2±0.5%
TADW	46.2±0.7%	50.1±0.9%	66.1±0.7%
DGI	75.3±0.4%	71.7±0.4%	71.1±0.2%
GCN	82.1±0.6%	71.8±0.5%	74.2±0.7%
EGNN	81.3±0.4%	69.7±0.3%	75.7±0.3%
CAT-I-MF	83.5±0.5%	72.0±0.3%	75.5±0.7%
HKGCN	82.8±0.1%	66.2±0.2%	74.3±0.2%
GAT	83.8±0.2%	71.5±0.2%	74.8±0.1%
CSGAT	84.6±0.2%	72.7±0.2%	76.7±0.3%
		Macro-F1
MLP	50.5±0.1%	56.4±0.1%	64.0±0.1%
Deepwalk	58.0±0.6%	36.3±0.9%	46.9±0.6%
TADW	56.1±0.8%	58.3±0.8%	68.7±0.7%
DGI	69.8±0.7%	61.6±0.5%	67.7±0.3%
GCN	79.5±0.7%	63.2±0.4%	71.6±0.8%
EGNN	78.3±0.4%	61.2±0.3%	74.0±0.3%
CAT-I-MF	82.3±0.4%	65.0±0.4%	74.1±0.8%
HKGCN	80.2±0.1%	60.2±0.2%	73.1±0.2%
GAT	82.4±0.2%	64.4±0.2%	73.6±0.1%
CSGAT	83.1±0.1%	66.0±0.3%	75.8±0.4%
		Weighted-F1
MLP	55.8±0.1%	61.0±0.1%	65.5±0.1%
Deepwalk	60.9±0.7%	39.3±0.8%	51.3±0.5%
TADW	50.1±0.7%	55.1±0.6%	65.8±0.5%
DGI	74.2±0.5%	68.8±0.5%	69.7±0.2%
GCN	82.1±0.6%	69.6±0.6%	73.4±0.8%
EGNN	81.4±0.4%	67.4±0.3%	75.2±0.2%
CAT-I-MF	83.4±0.4%	70.6±0.3%	75.0±0.7%
HKGCN	83.0±0.1%	65.1±0.2%	74.0±0.2%
GAT	83.8±0.1%	70.0±0.2%	74.5±0.1%
CSGAT	84.7±0.1%	71.3±0.2%	76.4±0.3%

4.6 Algorithm complexity analysis

We perform complexity analysis on the proposed algorithm CSGAT and compare it with GAT. As shown in Table 3, we record the running time of the algorithm on the Cora dataset to represent the complexity of the algorithm. It can be seen that our algorithm takes slightly more time than GAT in a single epoch because our algorithm makes full use of the graph information. However, thanks to the complementary advantages of content and structure information, our algorithm can converge faster in graph learning, which makes CSGAT reach the optimal solution with less number of epochs. In other words, CSGAT can achieve better performance in less total time. Therefore, CSGAT achieves better results in both running time and performance.

Table 3
Time taken to run the algorithm on the Cora dataset

GAT CSGAT

Single epoch time 275±25ms 323±15ms

Number of epochs 812±40 596±30

Total time 225±20s 191±10s

	GAT	CSGAT
Single epoch time	275±25ms	323±15ms
Number of epochs	812±40	596±30
Total time	225±20s	191±10s

4.7 Parameter sensitivity

In this section, we analyze the parameter sensitivity through the Cite data set and report the performance of ours with different number of multi-head attention. The results are shown in Fig. 3. Note that when the number of multi-head attention is set to 1, the multi-head attention will be removed. It can be seen that with the increase of the number of multi-head attention, the overall performance of ours shows a trend of improvement. Therefore, to obtain the best performance on a certain dataset, we recommend searching for this parameter value first.

Fig. 3

Parameter analysis of different number of multi-head attention K.

4.8 Ablation analysis

To further explore the key component of our proposed hybrid attention mechanism, we design two variants of CSGAT method for ablation analysis, named CSGAT-content and CSGAT-structure. Note that CSGAT-content is the same as GAT in fact. CSGAT-content only utilizes the content-based attentions, while CSGAT-structure only utilizes the structure-based attentions. For convenience, all variants are single-layered, including no dropout and multi-head. In this way, other factors that affect the experimental results are removed. Table 4 reports the results of the comparing algorithms. It clearly indicates that the hybrid attention mechanism achieves the best performance. And, the content-based attention achieves the worst performance, which indicates the structure-based attention is superior to the content-based attention to some extend.

Table 4
Experimental results for ablation analysis

Cora Cite Pubmed

Micro-F1

CSGAT-content 54.1% 63.1% 59.8%

CSGAT-structure 58.8% 66.5% 62.4%

CSGAT 60.0% 66.6% 62.5%

Macro-F1

CSGAT-content 46.8% 52.4% 49.5%

CSGAT-structure 49.0% 55.9% 51.6%

CSGAT 50.4% 56.1% 51.7%

Weighted-F1

CSGAT-content 52.9% 59.2% 55.1%

CSGAT-structure 55.8% 63.0% 57.4%

CSGAT 57.9% 63.2% 57.5%

	Cora	Cite	Pubmed
	Micro-F1
CSGAT-content	54.1%	63.1%	59.8%
CSGAT-structure	58.8%	66.5%	62.4%
CSGAT	60.0%	66.6%	62.5%
	Macro-F1
CSGAT-content	46.8%	52.4%	49.5%
CSGAT-structure	49.0%	55.9%	51.6%
CSGAT	50.4%	56.1%	51.7%
	Weighted-F1
CSGAT-content	52.9%	59.2%	55.1%
CSGAT-structure	55.8%	63.0%	57.4%
CSGAT	57.9%	63.2%	57.5%

4.8.1 Heatmap analysis

We test the ability of CSGAT-structure to discover dominant neighbors under different graph structures, and visualize it as a heatmap in Fig. 4. In Fig. 4(a) we design a large community, and through CSGAT-structure, we can find important neighbor nodes within 2-hop neighbors. In Fig. 4(b)-(d), we design several special subgraphs, all of which contain communities. CSGAT-structure can well judge whether the central node is in the community or not, and further confirm whether the neighbor nodes belong to the same community or not. We eliminate the community in Fig. 4(e), (f), as we can see from the heatmap, the weights of the neighbor nodes have changed. Nodes that originally belonged to different communities from the central node gained relatively greater weights due to the elimination of the community structure. This is very beneficial.

Fig. 4

The heatmap of the attention mechanism based on topology structure under different graph structures. Left: A simple artificially constructed graph, the yellow node is the center node, and the blue node is the neighbor node, and the two are connected by edges; Middle: Visualization of weights; Right: Contours of weights.

4.8.2 Parameter sensitivity

As shown in Equation (5), r_f and r_s adjust the content-based scores and structure scores, leading to a standard weighted average. The two variable are tuned in the training. It is interesting to explore the importance of them in the proposed model. We visualized these those two weight coefficients in CSGAT model over the datasets Cora, Cite and Pubmed, as shown in Fig. 6. According to Fig. 6, the content-based scores are more important over dataset Cora, while the structure-based scores are more important over datasets Cite and Pubmed. Because Cora has the densest network structure in all datasets, it also has the most node classes, which means that there are connections between a large number of nodes of different classes. Therefore, there is a lot of noise in the structural information in Cora, which degrades the performance of the structure-based attention mechanism. Then, thanks to the adaptive mechanism of the network, the weight distribution takes the content as the main index. This does not mean that structure-based attention mechanisms have completely failed. As mentioned earlier, our attention mechanisms are mixed. Under different datasets, the attention mechanism will adopt different strategies, that is, different proportions, to achieve the optimal solution of the model. To sum up, both weighted scores play a role in constituting the attention of the model, however, their importance is different in different application domains.

Fig. 5

Neighbors and their weights of node p408 in dataset Cite. (a), (b), (c) represent the neighbors, weights in CSGAT-Content and weights in CSGAT, respectively.

Fig. 6

An illustration of weight coefficients between content-based scores and structure-based scores.

4.8.3 Investigating the attention weights

Some important neighbors that are useful for specific tasks should have greater attention value. The key to improving the performance of node embeddings is how to discover important neighbor nodes and assign appropriate weights to the nodes. As mentioned earlier, take node P408 in the dataset Cite as an example. We enumerate the neighbors of node P408 and their weights achieving in CSGAT-content and CSGAT, as shown in Fig. 5. Here are two neighbors for discussion. Among them, P408 belongs to ’label1’ and P637 belongs to ’label4’. The CSGAT-content assigns maximum weight to P637, which is obviously unreasonable. Therefore, the unreasonable weight distribution of CSGAT-content makes them predict the p408 node as ’label4’. which obviously reduces the performance of it. It is worth noting that CSGAT gives the most reasonable weight combination after comprehensively considering the graph structure and node content. We assigned the highest weight to the meaningful node P408 and finally made the correct prediction. Based on the above analysis, we can see that CSGAT has an excellent performance in discriminating the differences between neighbor nodes.

5 Conclusion

In this paper, we point out the defects of the existing neural network model in weight allocation, that is, it does not use both content and structure information. We propose a graph attention mechanism based on content and structure information to make full use of the information in the graph, so as to make up for the shortcomings of existing models. In this paper, we propose a new adaptive graph attention network structure for node classification. First, unlike traditional GAT, Our considers not only the similarity of node content, but also the similarity of structural information. Secondly, the structure-based attention mechanism is efficient and explainable, and it can accurately assign weight to neighbor nodes without additional training and learning. Furthermore, the attention mechanisms we use are mixed, which means they can extract more effective information from multiple dimensions. Finally, because the proposed model can find the key neighbor node for the central node more efficiently, the convergence rate of the model is improved, which means that our model not only has better performance, but also is more efficient than the existing model. In the experiment, we used ablation analysis and heatmaps to support and analyze the model. It can be seen from the experimental results that this improvement effectively improves the representation learning ability of graph neural network. In the future, we will further improve this method, for example, by removing the nonlinear transformation to further improve the efficiency while striving to improve its performance.

Footnotes

Acknowledgment

This work is supported by National Social Science Fund (Grant No. 21BTJ011). The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers, which have improved the presentation. Yong Chen proposed the methodology, performed the experiment and wrote the manuscript; Xiao-zhu Xie and Wei Weng revised the manuscript and helped perform the analysis with constructive discussions.

https://github.com/shenweichen/GraphEmbedding

https://github.com/dedekinds/Graph-Embedding

https://github.com/PetarV-/DGI

https://github.com/tkipf/pygcn

https://github.com/vietph34/Edge GNN

https://github.com/he-tiantian/cats

https://github.com/hazdzz/HKGCN

https://github.com/Diego999/pyGAT

References

Zhou

and Li

, Structural attention network for graph, Applied Intelligence 51(8) (2021), 6255–6264.

Chaluvaraju

and Shaikh

A.B.

, Different versions of atom-bondconnectivity indices of some molecular structures: applied for thetreatment and prevention of COVID-19, Polycyclic AromaticCompounds 42(6) (2022), 3748–3761.

Perozzi

, AI-Rfou

and Skiena

, DeepWalk: Online Learning of Social Representations, Processdings of the 20th ACM SIGKDDInternational Conference on Knowledge Discovery and Data Mining (2014), 701–710.

Tang

, Kramer

S.T.

, Fang

, Qiu

, Wu

and Xu

, Aself-attention based message passing neural network for predictingmolecular lipophilicity and aqueous solubility, Journal ofCheminformatics 12(1) (2020), 1–9.

Yang

, Liu

, Zhao

, Sun

, Chang

Network Representation Learning with Rich Text Information, Processings of the Twenty-fourth International Joint Conference on Artificial Intelligence, 2015, 2111–2117.

Spielman

D.A.

Spectral graph theory and its applications, Processdings of the 48th Annual IEEE Symposium on Foundations of Computer Science, 2007, 29–38.

Chen

, Lin

, Li

, Zhou

, Sun

Measuring and Relieving the Over-smoothing Problem for Graph Neural Networksfrom the Topological View, Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 3438–3445.

Ayala-Gómez

, Daróczy

, Benczúr

, Mathioudakisand

and Gionis

, Global citation recommendation using knowledgegraphs, Journal of Intelligent & Fuzzy Systems 34(5) (2018), 3089–3100.

, Zhu

, Wu

, Wang

, Tan

Hierarchical Graph Convolutional Networks for Semi-supervised Node Classification, Proceedings of the 28th International Joint Conference on Artificial Intelligence, 2019, 4532–4539.

10.

, Souza

, Zhang

, Fifty

, Yu

and WeinbergerSimplifying

, Graph Convolutional Networks, Proceedings of the 43rd International Conference on Machine Learning (2020), 6861–6871.

11.

Nikolentzos

, Tixier

, Vazirgiannis

Message passing attention networks for document understanding, Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 8544–8551.

12.

Nikolentzos

, Dasoulas

and Vazirgiannis

, K-hop graph neuralnetworks, Neural Networks 130 (2020), 195–205.

13.

Yuan

, Yu

, Gui

and Ji

, Explainability in graph neuralnetworks: A toaxonmic survey, IEEE Transactions on PatternAnalysis and Machine Intelligence 45(2) (2022), 5782–5799.

14.

Bruna

, Zaremba

, Szlam

and LeCun

, Spectral Networks andLocally Connected Networks on Graphs, Proceedings of the 2ndInternational Conference on Learning Representations (2014), 1–14.

15.

, Wang

, Kuen

, Ma

, Shahroudy

, Shuai

, Liu

, Wang

, Cai

and Chen

, Recent advances in convolutionalneural networks, Pattern Recognition 77 (2018), 354–377.

16.

Ding

J.Y.

, Cheng

R.H.

, Song

, Zhang

X.R.

, Jiao

L.C.

and WuGraph

J.S.

, label prediction based on local structure characteristicsrepresentation, Pattern Recognition 125 (2022), 108525.

17.

Zhao

, Dong

, Tang

, Ding

, Wang

Generalizing Graph Convolutional Networks Via Heat Kernel, Processings of the 9th International Conference on Learning Representations (ICLR), 2021, 1–14.

18.

, Hu

, Leskovec

, Jegelka

How Powerful are Graph Neural Networks?, Proceedings of the 7th International Conference on Learning Representations, 2019, 1–17.

19.

, Lung

P.Y.

, Zhao

, Tseng

Y.Y.

and Zhang

, Automaticextraction of protein-protein interactions using grammaticalrelationship graph, BMC Medical Informatics and DecisionMaking 18(2) (2018), 35–43.

20.

Zhang

, Zhu

, Wang

, Zhang

Adaptive Structural Fingerprints for Graph Attention Networks, Proceedings of the 8th International Conference on Learning Representations, 2020, 1–10.

21.

Getoor

Link-based classification, Advanced Methods for Knowledge Discovery from Complex Data (2005), 189–207.

22.

Gong

, Cheng

Exploiting Edge Features for Graph Neural Networks, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 9211–9219.

23.

Pasa

, Navarin

and Sperduti

, Multiresolution reservoir graphneural network, IEEE Transactions on Neural Networks andLearning Systems 33(6) (2021), 2642–2653.

24.

Ruiz

, Gama

and Ribeiro

, Gated graph recurrent neuralnetworks, IEEE Transactions on Signal Processing 68(2020), 6303–6318.

25.

Shi

, Tong

, Tang

and Lin

, Vegas: Visual influence graphsummarization on citation networks, IEEE Transactions onKnowledge and Data Engineering 27(12) (2015), 3417–3431.

26.

Baek

, DiMaio

, Anishchenko

, Dauparas

, Ovchinnikov

, Lee

G.R.

, Wang

, Cong

, Kinch

L.N.

and Baker

, Accurateprediction of protein structures and interactions using athree-track neural network, Science 373(6557) (2021), 871–876.

27.

Defferrard

, Bresson

and Vandergheynst

, Convolutional neuralnetworks on graphs with fast localized spectral filtering, Neural Information Processing Systems 29 (2016), 3844–3852.

28.

Newman

M.E.J.

, Watts

D.J.

, Strogatz

S.H.

Random graph models of social networks, Proceedings of the National Academy of Sciences, 2002, 2566–2572.

29.

, Zhang

, Shi

, Wang

, Zhang

A Statistical Characterization of Attentions in Graph Neural Networks, Proceedings of the 7th International Conference on Learning Representations, 2019, 1–10.

30.

Przulj

, , Protein-protein interactions: Making sense ofnetworks via graph-theoretic modeling, Bioessays 33(2) (2011), 115–123.

31.

Srivastava

, Hinton

, Krizhevsky

, Sutskever

and Salakhutdinov

, Dropout: a simple way to prevent neural networks fromoverfitting, Journal of Machine Learning Research 15(1) (2014), 1929–1958.

32.

Wang

, Wu

, Li

, Wu

, Shi

and Ma

, Meta-learningbased spatial-temporal graph attention network for traffic signalcontrol, Knowledge-Based Systems 250 (2022), 109166.

33.

Liben

N.D.

, Kleinberg

The Link Prediction Problem for Social Networks, Proceedings of the Twelfth International Conference on Information and Knowledge Management, 2003, 556–559.

34.

Sen

, Namata

, Bilgic

, Getoor

, Galligher

and Eliassi

T.R.

, Collective classification in network data, AIMagazine 29(3) (2008), 93–93.

35.

Velicković

, Cucurull

, Casanova

, Romero

, Lioand Yoshua Bengio

Graph Attention Networks, Proceedings of the 6th International Conference on Learning Representations, 2018, 1–12.

36.

Velicković

, Fedus

, Hamilton

W.-L.

, Lio

, Bengioand

, Hjelm

R.-D.

Deep Graph Infomax, Proceedings of the 7th International Conference on Learning Representations, 2019, 4–21.

37.

Kaushik

, Shenoy

, Bohannon

, Gudes

Exploiting Local Similarity for Indexing Paths in Graph-Structured Data, Proceedings 18th International Conference on Data Engineering, 2002, 129–140

38.

Albawi

, Mohammed

T.A.

, Al-Zawi

Understanding of a Convolutional Neural Network, 2017 Proceedings of the International Conference on Engineering and Technology (ICET), 2017, 1–6.

39.

Narang

S.K.

, Gadde

, Ortega

Signal Processing Techniques for Interpolation in Graph Structured Data, Proceedings of the 2013 IEEE International Conference on Acoustics Speech and Signal Processing, 2013, 5445–5449.

40.

Wan

, Zhan

, Liu

, Yu

, Pan

and Gong

, Contrastivegraph poisson networks: Semi-supervised learning with extremelylimited labels, Advances in Neural Information ProcessingSystems 34 (2021), 6316–6327.

41.

Zhang

, Jiang

, Wang

, Wei

and LiSAG-DTA:

, prediction of drugâŁ"target affinity using self-attentiongraph network, International Journal of Molecular Sciences 22(16) (2021), 8993.

42.

Zhang

, Liu

, Xie

Molecular Mechanics-Driven Graph Neural Network with Multiplex Graph for Molecular Structures, The 34th Conference on Neural Information Processing Systems, 2020.

43.

Danel

, Spurek

, Tabor

, Smieja

, Struski

L.A

, Slowikand Maziarka

Spatial graph convolutional networks, Proceedings of the International Conference on Neural Information Processing, 2020, 668–675.

44.

, Ong

Y.S.

and Bai

, Learning conjoint attentions for graphneural nets, Neural Information Processing Systems 34(2021), 2641–2653.

45.

Kipf

T.N.

, Welling

Semi-Supervised Classification with Graph Convolutional Networks, Proceedings of the 5th International Conference on Learning Representations, 2017, 1–14.

46.

Pham

, Tao

, Zhang

, Yong

, Li

and Xie

, Graph-basedmulti-label disease prediction model learning from medical data anddomain knowledge, Knowledge-Based Systems 235 (2022), 107662.

47.

Gatta

V.L.

, Moscato

, Postiglione

and Sperli

, Anepidemiological neural network exploiting dynamic graph structureddata applied to the covid-19 outbreak, IEEE Transactions on BigData 7(1) (2020), 45–55.

48.

Fan

, Ma

, Li Yuan He

, Zhao

, Tang

, Yin

Graph Neural Networks for Social Recommendation, Proceedings of theWorldWideWeb Conference, 2019, 417–426.

49.

Hamilton

, Ying

and Leskovec

, Inductive representationlearning on large graphs, Neural Information ProcessingSystems 30 (2017), 1025–1035.

50.

Jin

, Li

, Xu

, Wang

, Ji

, Aggarwal

and Tang

, Adversarial attacks and defenses on graphs, ACM SIGKDDExplorations Newsletter 22(2) (2021), 19–34.

51.

Rawat

and Wang

, Deep convolutional neural networks for imageclassification: A comprehensive review, Neural Computation 29(9) (2017), 2352–2449.

52.

, Sun

, Zhang

and Hancock

E.R.

, Two-level graph neuralnetwork, IEEE Transactions on Neural Networks and LearningSystems 10 (2022), 1–14.

53.

Dong

, Liu

, Du

and Zhang

, Weighted feature fusion ofconvolutional neural network and graph attention network forhyperspectral image classification, IEEE Transactions on ImageProcessing 31 (2022), 1559–1572.

54.

, Liang

and Jiang

, Path reliability-based graph attentionnetworks, Neural Networks 159 (2023), 153–160.

55.

Pand

, Huang

, Wang

, Li

, Hosseini

, Zhang

and Ai

, Graph decipher: A transparent dual-attention graph neural networksto understand the message-passing mechanism for the nodeclassification, International Journal of Intelligent Systems 37(11) (2022), 8747–8769.

56.

Xie

, Zhang

, Gong

, Tang

and Han

, Mgat: Multi-viewgraph attention networks, Neural Networks 132 (2020), 180–189.

57.

and Ji

, Sparse graph attention networks, IEEETransactions on Knowledge and Data Engineering 35(2) (2021), 905–916.

58.

Zhang

, Yu

, Cui

, Wu

, Wen

, Wang

Every Document Owns Its Structure: Inductive Text Classification Via Graph Neural Networks, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, 334–339.

59.

Cheng

, Yan

, Wu

F.X.

and Wang

, Drug-target interactionprediction using multi-head self-attention and graph attentionnetwork, IEEE/ACM Transactions on Computational Biology andBioinformatics 19(4) (2021), 2208–2218.

60.

, Pan

, Chen

, Long

, Zhang

and Philip

S.Y.

, Acomprehensive survey on graph neural networks, IEEETransactions on Neural Networks and Learning Systems 32(1) (2020), 4–24.

61.

Yang

, Cohen

, Salakhudinov

Revisiting semisupervised learning with graph embeddings, Proceedings of the International Conference on Machine Learning, 2016, 40–48..

Content and structure based attention for graph node classification

Abstract

Keywords

1 Introduction

2.1 Graph neural network

2.2 Attention mechanism

3 The proposed algorithm

4 Experiments

4.1 Datasets

Table 1 Datasets statistics Datasets Nodes Edges Features Classes Training Validation Testing Cora 2708 5429 1433 7 140 500 1000 Cite 3327 4732 3703 6 120 500 1000 Pubmed 3356 4278 500 3 60 500 1000

4.3 Parameter settings

4.4 Evaluation

Table 3 Time taken to run the algorithm on the Cora dataset GAT CSGAT Single epoch time 275±25ms 323±15ms Number of epochs 812±40 596±30 Total time 225±20s 191±10s

5 Conclusion

Footnotes

Acknowledgment

References

Table 1
Datasets statistics

Datasets Nodes Edges Features Classes Training Validation Testing

Cora 2708 5429 1433 7 140 500 1000

Cite 3327 4732 3703 6 120 500 1000

Pubmed 3356 4278 500 3 60 500 1000

Table 3
Time taken to run the algorithm on the Cora dataset

GAT CSGAT

Single epoch time 275±25ms 323±15ms

Number of epochs 812±40 596±30

Total time 225±20s 191±10s