Image semantic learning method based on social heterogeneous graph networks 1

Abstract

Image semantic learning techniques are crucial for image understanding and classification. In social networks, image data is widely disseminated thanks to convenient acquisition and intuitive expression characteristics. However, due to the freedom of users to publish information, the image has apparent context dependence and semantic fuzziness, which brings difficulties to image representation learning. Fortunately, social attributes such as hashtags carry rich semantic relations, which can be conducive to understanding the meaning of images. Therefore, this paper proposes a new method named Social Heterogeneous Graph Networks (SHGN) for image semantic learning in social networks. First, a heterogeneous graph is built to expand image semantic relations by social attributes. Then the consistent semantic space is reconstructed through cross-media feature alignment. Finally, an image semantic extended learning network is designed to capture and integrate the social semantics and visual feature, which obtains a rich semantic representation of images from a social context. The experiments demonstrate that SHGN can achieve efficient image representation, and favorably against many baseline algorithms.

Keywords

Social networks image representation learning heterogeneous graph social semantic aggregation

1 Introduction

As a form of expression for social network messages [1], images can attract users’ attention faster due to their intuitive visual nature. Image information is widely disseminated and shared in fast-paced entertainment and interactive platforms, leading it to encompass a wealth of social phenomena. Adequate semantic understanding of images can help to mine and discover public opinion events promptly. Thus, the feature representation of social network image data are crucial for tasks such as the detection and search of public events [2, 3]. However, due to the ubiquity of users, the shooting and uploading of pictures are random. This phenomenon leads to irregularity and semantic sparsity of images in social networks. Therefore, semantic representation learning for social network image face some challenges [4, 5].

Traditional image feature representation methods can be roughly divided into two categories, shallow and deep feature [6] extraction and learning methods. They design a deep learning framework to extract and mine the global characteristics of images, which shows significant advantages. However, these methods do not fully consider internal and external potential correlations of image data and do not use the additional knowledge to discover the relation between image contexts. Thus, they cannot effectively distinguish and judge the generalization of social network image information, resulting in poor quality of image representation. In this paper, the social relationship of social network images is semantically extended and mined, and high-quality image semantic representation methods are studied.

Due to the unique sharing and interactivity of social networks, multi-attribute features of social networks play a crucial role in message understanding. For example, topic hashtags marked with “#" are summary information that can quickly categorize events. A reference operation that points to or reminds a relevant user represented by the “@" symbol, a hyperlink information “URL" that points to the same external extension content with “http://" and the time-space trajectory of the news release and dissemination. These multi-attribute features can integrate the interaction between social network messages, which are ability to mine the latent semantics between seemingly unrelated messages. Given the relevant statistics, more than 10% of the microblog messages published contain different social attributes such as a topic hashtag, links, or “@". This result indicates that most additional information in social networks has been used as external metadata. The above multi-attribute features of social networks can reflect more semantic external information. Therefore, mining the potential associations and meanings of multi-attribute features in social networks can fully discover and represent the consistent semantics of image data. It can achieve effective semantic expansion and fusion of information to eliminate the incompleteness and semantic sparsity in social networks image descriptions.

Currently, research on graph convolutional neural networks [7, 8] has made advancements with the proliferation of graph data such as social networks and knowledge graphs [9, 10]. [11] propose a method of learning node embedding representation through random walk GCN. It constructs semantic associations of neighbor nodes to realize image recommendation. During the generation and dissemination of events, social attributes have strong semantic consistency in describing events. Therefore, benefiting from the various forms of attribute features within social networks, numerous studies have been devoted to exploiting the unique social context information [12, 13] for tasks such as efficient event detection and feature representation. In addition, there are existing GCN models [14, 15] for heterogeneous graphs, which perform fusion learning on heterogeneous feature spaces by reconstructing adjacency matrices [16]. Based on the implied social relations between social attributes, the social networks can be constructed as a graph model with rich semantics. We mine and analyze the latent semantics contained in social multi-attributes to effectively perform association discovery and semantic expansion of features on images.

In order to obtain higher-quality image semantic information, this paper proposes an image semantic representation learning algorithm based on the social heterogeneous graph network. With the help of social multi-attribute features, the image context information is associated, and a semantically related topic hashtag heterogeneous graph is constructed. The latent semantics of social attributes are incorporated into the expression of image features to establish relevance between images. In the process of node aggregation and convolution operation, more semantically rich image features are obtained. The proposed method associates corresponding text hashtags with visual objects and achieve the effective complement of the semantic features of event images. Extensive experiments fully demonstrate the effectiveness of the proposed algorithm SHGN. Compared with the benchmark methods of image feature representation, the proposed SHGN can obtain a higher quality semantic representation of social networks images. The main contributions of this paper are summarized as follows:

Towards fully understanding the meaning of social network images in public events, this paper proposes a novel social heterogeneous graph network to learn the rich semantics of images.

We adopt social attributes to mine the relationship between independent image data, which guides the semantic expansion in the association context.

The social heterogeneous graph model is established to reveal the abundant social semantics among images. The image representations are learned by designing SHGN based on semantic aggregation.

Extensive experimental results show that the proposed SHGN method performs favorably against many baseline image representation methods in classification tests.

In the following Section 2, the related work is reviewed. Section 3 provides the formal problem statement and the detailed design of the SHGN method. In Section 4, we report and display extensive experimental results. Finally, we conclude the work of this paper in Section 5.

2 Related work

2.1 Image representation

The traditional methods usually take learning more semantically discriminative features as the critical issues. At present, many researchers are committed to image understanding [17, 18] and image sentiment analysis [6], etc. Typical image semantic feature representation methods are divided into shallow visual feature acquisition methods [19, 20] and deep learning-based image representation methods [21, 22]. [23] integrates the social context and visual features according to the user search intent. The learned image features contain rich semantic information and the user search tendency. [24] proposes an image semantic acquisition method based on contextual information and a shallow spatial encoder-decoder network, which improves the quality of image semantics and shows superior performance in image segmentation.

In recent years, the deep learning methods have shown significant advantages in feature learning [25]. For example, deep neural network structures deep belief network (DBN) [26], AlexNet, and VGG [22] can deeply mine and learn the semantic representation of image features. It has been widely used in tasks such as image classification, annotation, and search. [27] proposes an image semantic information recognition method, which uses the low-level image features to assist CNN extract the high-level features. It realizes the image recognition of the stack sparse autoencoder network.

Excellent research for semantic understanding of social images has been proposed successively [28 –30]. [28] and [30] design a deep weakly-supervised model to discover image representations and improve the quality of tags. Li et al. [29] achieve image annotation by considering the visual similarity and semantic relevance. They address the problem of tags being noisy, irrelevant, or incomplete. This paper focuses on the issue of ambiguous semantic images and obvious reliance on the text on social platforms such as Sina Weibo and Twitter. We propose a novel social heterogeneous graph network to improve the image semantic representation by introducing social knowledge to mine the relation in image contexts.

2.2 Graph learning

Graph learning methods are currently applied in many tasks due to the rich correlations among data [31 –33]. They recognize group activities and social image retagging by constructing different relationship graphs. [31] and [32] propose a novel coherence constrained graph LSTM and graph LSTM-in-LSTM to effectively recognize group activities, respectively. Tang et al. [33] design a novel social anchor-unit graph regularized tensor completion method to refine the tags of social images. Graph Convolutional Network (GCN) has attracted the attention of a large number of researchers, which aggregates and learns nodes by using the structural information of the graph and the feature of the nodes. In a typical approach, [34] performs feature embedding learning on nodes by fusing node features and linking information of the graph structure. The GCN-based multi-label image recognition model ML-GCN [35] builds a label dependency graph and improves the ability of image representation recognition through learning of target classifiers. Text GCN method effectively solves the problem of sparse feature representation.

The existing research has obtained outstanding achievements in heterogeneous graph neural networks [36 –38]. They intelligently aggregate information from all sources to provide social recommendations by building a heterogeneous graph model. However, the heterogeneous graph model in this paper is to learn the image feature from different modalities in social networks. We use summary text as semantic guidance to integrate into the image feature representation. To some extent, it alleviates the semantic limitations of images and the dependence on text information. The visual and text attributes are connected through the co-occurrence of social characteristics. The edges in the graph also have rich social relationships. We can better fuse the associated nodes with different strengths in the graph convolution. [39] constructs a heterogeneous graph containing multiple node types and achieve accurate text classification by learning node representations based on GCN. The interrelated node information is aggregated during the convolution operation of GCN, which realizes the expansion of node features. [40] finds the hidden semantic spaces by mining context and content links, which maps multimedia data into consistent features. Therefore, in the representation of image features of the social networks, this paper makes full use of the nature of multi-attribute features of social networks. The topic hashtags and images are regarded as the nodes of a heterogeneous social network graph model which realizes image semantic learning and augment representations based on GCN.

3 Methodology

This paper proposes an image semantic representation algorithm based on the social heterogeneous graph network. It takes topic hashtags and images as two types of nodes in the graph structure. In the light of the potential social semantic association between the social multi-attribute information, a semantically rich graph model of social heterogeneity is constructed. The graph model fuses the contextual correction among image data and the propagation relationship of summative hashtags. Through the reconstruction of the heterogeneous feature space, we obtain the consistent node feature matrix for multimodality. The graph convolutional neural network (GCN) is adopt to represent and learn the features of the graph data. The hashtags are considered as an important guiding information in events, the semantic features of image information are aggregated, so as to achieve a high-quality representation of image semantics in social networks. The overall framework is shown in Fig. 1.

Fig. 1

The framework of SHGN.

The main steps of SHGN are: 1) Construction of social heterogeneous graph model. It includes two types of nodes of images and hashtags and forms associations by introducing multi-attribute features; 2) Heterogeneous feature reconstruction representation. We adopt the traditional image feature representation method VGGNet-19 to generate the initial feature of the image node. The visual features are aligned with text features (topic hashtags) to unify heterogeneous features; 3) Image extended semantic feature learning network. The joint learning is performed in the graph convolutional neural network. The neighbor aggregation of convolution operation makes the image features fuse the associated nodes with relevant social semantics so that it can obtain a certain semantic expansion. The SHGN enables deep representation and mining of semantics. It realizes effective semantic learning of social network images.

3.1 Problem statement

Given a social network image dataset M, each image element m ∈ M contains attribute domain {ids, image content, topic hashtags, external links, class label}. The id is the number of the message, and the hashtag and external link are the social attributes. According to the social relations of ids, hashtags, and external links, a social heterogeneous graph G is established (see Definition 1 below). The hashtags are regarded as important semantic nodes to establish connections with image nodes. We adopt different modal feature learning techniques to map the feature space of text (topic hashtags) and visual (image), denoted as X_H and X_V, respectively. The text and visual features are fused and reconstructed, obtaining length-aligned vector representations X. Combined with the graph structure feature matrix A formed by the association weights between the nodes, the node embedding learning is performed based on GCN. The predicted class labels L are output through the node classification training, and the semantic feature representation of the image nodes are obtained, i.e. H.

3.2 Construction of social heterogeneous graph model

To mine semantic associations between discrete image elements, hashtags and link information “URL” are extracted from social network messages. A social heterogeneous graph model is built to describe the contextual semantics of image data. The graph model contains two entity types, image nodes and hashtag nodes. The social multi-attribute features guide semantically consistent images to create associations. The formal definition of a social heterogeneous graph model is as follows.

Define 1. (Social Heterogeneous Graph). The social heterogeneous graph is a weighted undirected graph structure, which is expressed as G = (V, E, f). The node contains the visual image V_i and the hashtag text V_h, which are represented by V = {V_i ∪ V_h}. E = {E_i↔h ∪ E_i↔i ∪ E_h↔h} denotes a set of three types of edges formed by connecting relations co-mid (co-occurs within the same message), co-h (contains the same hashtag) and co-URL (contains the same link information). The feature space mapping function is f : V → R^d, where each node has a d-dimensional feature vector.

We construct the above social heterogeneous graph model with representative, abstract, and scalable social network multi-attribute features, i.e. hashtags and external link information. Through the co-occurrence relationship between these social multi-attribute features, the semantic relatedness of two types of nodes can be modeled. The following three connection rules are defined to establish connection relationships between nodes in a social heterogeneous graph:

Rule 1(image-hashtag). When the hashtag node and the image node contain the same id value, or the same “URL” information, a connection edge is established between the image and the hashtag node.

Rule 2(hashtag-hashtag). If the two hashtag nodes contain the same id, or the microblog message where the two hashtags are located contains the same “URL”, a connection relationship will be established between the two hashtag nodes.

Rule 3(image-image). If the message where the two image nodes are located contains the same hashtag or “URL”, a connection edge is formed between the two image nodes.

Fig. 2 shows an example of a social heterogeneous graph. This heterogeneous graph contains image nodes a, b, c, d and hashtag nodes e, f, g. Each node corresponds to a certain characteristic attribute. Since hashtag nodes f and g are both included in the message where image node d is located, according to Rule 1, f and d, g and d have a “co-mid” connection relationship. Based on Rule 2, nodes f and g have a “co-mid” connection relationship. In addition, the hashtag node g appears in the message where the image nodes c and d are at the same time, so according to Rule 3, the two image nodes c and d are connected by “co-h”.

Fig. 2

An example of social heterogeneous graph.

The connection rules of the graph model reveal explicit and implicit relationships between image nodes and hashtag nodes. The explicit association is directly connected relations, such as the co-occurrence relations of hashtag and hashtag (or images), which indicate that nodes have strong consistency in semantics. Other rules are an implicit association, including Rule 3 and the URL-directed relationships in Rule 1 and Rule 2. They indicate that nodes contain indirect dependencies of events. Therefore, the social heterogeneous graph combines rich semantic relationships and structured information, which lays the foundation for the semantic representation of the social network images.

3.3 Heterogeneous feature reconstruction representation

Nodes in the social heterogeneous graph have different attribute characteristics so that the heterogeneous graph contains different feature spaces. We set the consistent feature matrix X _U to represent the semantic feature space of all nodes in the heterogeneous graph. The attribute domain of the hashtag node is a sparse text feature. All hashtags in the dataset are tokenized. Based on the research of text word vectors, the top-2000 most frequent words in the corpus can cover the hashtag vocabulary. Thus, we select them to constitute a text feature dictionary. The feature of each hashtag node is converted into a d_h-dimensional vector. Each dimension represents the number of occurrences of the feature in the short text. The feature of the hashtag node is normalized as: $\hat{h_{ij}} = f_{h} (h_{ij}) = \frac{h_{ij}}{\sum_{n = 1}^{d_{h}} h_{in}}$ (1)

Where h_ij is the j-dimensional vector representation of the hashtag node h_i. The f_h mapping function generates the characteristic matrix of hashtag, expressed as X_H ∈ R^(N-n)*d_h.

Aiming at the feature representation of image nodes in the social heterogeneous graph model, an image original feature representation sub-network is designed. The notable image feature learning approach VGGNet-19 [22] is adopted to capture the global features of the social image. Due to the feature space heterogeneity of text and vision, we need to align the feature dimensions of nodes in the graph model. Thus, we set the feature dimension of the VGGNet-19 final output to 1000. The VGGNet-19 network is pre-trained to conduct feature extraction and dimensionality reduction conversion on the social image. By adding a fully connected layer at the end of VGGNet-19, the image original feature representation sub-network is formed. The representation of the last layer is shown as: $v_{i} = σ (W_{v} \times V_{vgg} + b_{v})$ (2)

Where V_vgg is the feature representation learned by the pre-trained VGGNet-19. W_v and b_v are the weight vector of the fully connected layer. The image feature matrix of the social heterogeneous graph is denoted as X_V ∈ R^n*d_v.

Under the heterogeneous feature formed by text and visual, this section constructs a unified feature matrix of graph data node. The feature space is reconstructed by fusing the image feature X_V with the hashtag feature X_H. The graph node feature matrix is expressed as: $X_{U} = [\begin{matrix} 0, & X_{H} \\ X_{V}, & 0 \end{matrix}] \in R^{N * (d_{h} + d_{v})}$ (3)

Among them, d_h and d_v are 1000 respectively, so X_U ∈ R^N*2000.

There are different degrees of semantic similarity between interconnected nodes. In order to avoid the mismatch of similarity calculation in various feature spaces, the condition numbers that satisfy different rules are utilized to denote the degree of association of different node pairs in the social heterogeneous graph. They are set as edge weights. According to the define of Rules, two nodes may satisfy multiple conditions at the same time. That is, two hashtags may appear in multiple messages at the same time or the message may contain some “URL”. This phenomenon is because the same social information may be included in multiple social network messages, so two nodes may satisfy the connection condition many times. It is further explained that the more connection conditions are satisfied between two nodes, the stronger the semantic correlation of nodes. The weighted adjacency matrix A of the social heterogeneous graph is obtained by calculating the number of conditions that satisfy the connection rules between nodes. It is used as the structural information of the heterogeneous graph. If there is an edge e_ij ∈ E between nodes v_i and v_j, then A_ij ∈ W_ij, otherwise A_ij = 0.

3.4 Image extended semantic feature learning network

This section designs a network for node feature representation learning for social heterogeneous graph based on GCN. The reconstructed feature representation of the heterogeneous graph is input into the GCN network, and the extended learning of semantic features is performed on the image attributes. Based on the operational characteristics of graph convolution, the global structural information of the graph model and the matrix features of nodes can be fully preserved in the convolution operation. We add a classification layer, which can expand the image semantics and enhance the classification performance of nodes.

In order to ensure that the receptive field can effectively achieve the feature convergence of indirectly connected nodes during the neighbor aggregation process, a K-layer GCN structure is designed in the heterogeneous graph node embedding learning. In the process of generating node embedding vectors according to the neighborhood attributes of nodes, the heterogeneous graph node feature matrix is X. To highlight the importance of inter-node interaction, the weights of each edge in the social heterogeneous graph model are considered in embedding learning. That is, a weighted adjacency matrix A is used to represent the topology of the social heterogeneous graph. The feature matrix of graph data node and graph structure adjacency matrix of the social heterogeneous graph are input into GCN, and the generated node embedding feature matrix is shown as: $H_{i + 1} = α_{i} ({\hat{AH}}_{i} W^{(i)}), H_{0} = X_{U}$ (4)

where $\hat{A}$ is the normalized adjacency matrix, $\hat{A} = D^{- \frac{1}{2}} (A + I) D^{- \frac{1}{2}}$ . I is the identity matrix, D is the degree matrix of A + I, and W⁽ⁱ⁾ is the hidden layer parameter matrix. i represents the number of GCN layers, α_i is the activation function of each layer. The activation functions on the hidden layers are selected as ReLU.

The generated semantic feature matrix H_K is classified and predicted line by line in the classification layer. We adopt softmax function: $P = softmax (H_{K} W_{f})$ (5)

where W_f ∈ R^(N-n)*C is the trainable parameter matrix. C is the number of event categories contained in the dataset.

The cross-entropy loss is taken as the loss function, and the model is trained by the gradient descent algorithm. The calculation is shown as: $L = - \sum_{i = 1}^{N} \sum_{j = 1}^{C} l_{i} {logP}_{ij}$ (6)

where, P_ij is the j-th dimension in the probability vector of the output image node i of the softmax function, which represents the probability that the node i belongs to the j-th class. c_i is the real class label of the node. The SHGN network minimizes the loss function on the training set and computes the gradient of descent through backpropagation.

3.5 Implementation of SHGN

The network of the SHGN algorithm adopts a joint training method and fine-tunes the original image feature representation sub-network in advance. In the process of joint training with the GCN classification sub-network, the VGGNet-19 pre-training parameters are unchanged to avoid over-fitting. We only update the parameter W⁽ⁱ⁾ of the GCN classification sub-network, so that the entire network model can achieve an effective train. The implementation steps of the SHGN are as Algorithm 1.

Algorithm 1 SHGN Algorithm

Input:

M, the dimension of the image feature generated by the sub-network of the original image feature representation d_v, GCN classification sub-network layers K, GCN classification sub-network hidden layer feature dimension d, the number of training iterations N_r;

Output:

Image semantic feature representation; Image prediction label;

1: Extract images and multi-attribute information in the datasets M;

2: Construction of Social Heterogeneous Graph G = (V, E, f);

3: Generate the characteristic matrix of hashtag nodes according to Eq. (1);

4: Initialize the parameters of VGGNet-19 W_v, b_v;

5: The sub-network of image original feature representation is pre trained;

6: Reconstructing the node feature matrix X_U according to Eq. (3);

7: Initialize the GCN classification sub-network parameter matrix W⁽ⁱ⁾;

8: Joint training image raw feature representation sub-network and GCN classification sub-network;

9: if N_r < 200 then

10: Calculate the cross-entropy loss;

11: end if

12: Calculate the gradient of descent using Adam algorithm;

13: Update GCN classification sub-network parameters W⁽ⁱ⁾;

14: Output L and image semantic feature representation.

The proposed SHGN is efficient. Before model training, we construct the social heterogeneous graph via social relations with the complexity O (n (N - n) + N²), where n is the number of image nodes, (N - n) is the number of hashtag nodes, N is the total number of the node in the social heterogeneous graph, respectively. The time complexity of generating the hashtag node feature vectors is O (N - n). Image node features are obtained from Vgg-Net19, and the time complexity is $O (n \sum_{l = 1}^{k^{'}} d_{l}^{2} C_{l - 1} C_{l})$ , where k′ is the number of convolution layers, d is the characteristic dimension of image nodes, C_l-1 is the number of input channels, and C_l is the number of output channels. In the worst case, C_l-1, C_l, and d take the maximum value, then the time complexity can be simplified to $O (k^{'} C_{max}^{2} {nd}_{max}^{2})$ . The typical GCN takes O (kNd² + k|E|d) time to learn the node feature, where k is the number of layers, d is the node feature dimension, |E| is the number of edges. We designed the social heterogeneous graph network based on the GCN. The time complexity is $O (n (N - n) + N^{2} + N - n + k^{'} C_{max}^{2} {nd}_{max}^{2} + kN d^{2} + k | E | d)$ , where N ⪢ n, d_max and d are node feature dimensions with the same order of magnitude. k′ and k are the number of layers of the different network structures. Hence, the complexity of SHGN ∼ O (N² + Nd² + |E|d), where d is the consistent feature matrix of node feature dimension. SHGN scales linearly in the number of edges similar to GCN.

4 Experimental results and analysis

In this section, the generated image semantic feature by the SHGN algorithm is used for the image classification task. The image semantic representation quality of the SHGN algorithm is analyzed from the aspect of classification performance. Firstly, the overall SHGN algorithm is evaluated compared with the comparison algorithms. Then the ablation study is carried out to analyze the rationality and effectiveness of each part of SHGN. Finally, we estimate the sensitivity of the SHGN algorithm, including the number of layers of the GCN network, the number of neurons, and different datasets. The experiments sufficient verify the effect of SHGN on semantic extension learning of social network image features.

4.1 Setup

4.1.1 Dataset

We choose the public disaster event datasets CrisisLexT6 [41] and CrisisLexT26 [42] on Twitter. The event “West Texas Explosion” and unrelated event classes are extracted from CrisisLexT26 to form a single event dataset. Each dataset is category annotated by event type and “NONE”. The image data of each message is crawled by the Twitter API based on the Tweet ID and the image link “url”. Due to the lack and mismatch of image information in the message, the crawled image data is filtered and preprocessed to form three image datasets. The multi-attribute features of social networks are extracted for each dataset, including topic hashtags and hyperlink information “URL". The final dataset statistics are shown in Table 1. We randomly divide each dataset into training and test sets in a ratio of 8 : 2.

Table 1
Statistics of dataset

Datasets #Message #Edges #Topic hashtags #Hyperlinks #Category

West Texas Exoplosion 4990 89250 1213(24.3%) 1442(28.9%) 2

CrisisLexT6 12568 312339 4838(38.5%) 5479(43.6%) 7

CrisisLexT26 13096 673270 6665(50.9%) 5657(43.2%) 27

Datasets	#Message	#Edges	#Topic hashtags	#Hyperlinks	#Category
West Texas Exoplosion	4990	89250	1213(24.3%)	1442(28.9%)	2
CrisisLexT6	12568	312339	4838(38.5%)	5479(43.6%)	7
CrisisLexT26	13096	673270	6665(50.9%)	5657(43.2%)	27

4.1.2 Baseline and parameter setting

In order to verify the effectiveness of the SHGN algorithm in the expression of image semantics, the relevant image semantic representation methods are selected for experimental comparison. They are Corr-LDA [43], which is an algorithm for mining image semantics through topic models; VGGNet-19 [22], MaxNet [6], and GCN [34], a method for image feature representation based on deep neural networks. In addition, because the text information of topic hashtags is integrated into the visual feature of images as extended semantics, typical multi-modal fusion algorithms GCH [44] and att-RNN [45] are introduced to further verify the effectiveness of the algorithm in feature fusion. They adopt topic hashtags as text data.

We implement our proposed SHGN model based on TensorFlow and adopt Adam optimizer with a batch size of 64. The initial learning rate is 0.2 and we decrease the learning rate by 0.01 every 20 steps. Following Reference [34], the number of training iterations Nr is set to a maximum of 200. The GCN classification sub-network layers K is tuned from [1 , 4]. the GCN classification sub-network hidden layer feature dimension d is tuned from [16, 32, 64, 128, 256]. Their optimal values are verified by the following experiment. Refer to the algorithm settings of the original literature to make the comparison algorithm reach the best state.

The details of the comparison algorithms are as follows.

Corr-LDA [43]: is the shallow image semantic representation method that builds the relationship between text and image modal data based on the topic model. It uses the topic semantic distribution of image modal to represent image features. This feature is fed into the SVM classifier for evaluation and comparison of image semantic quality.

VGGNet-19 [22]: is the image feature expression method based on deep learning that has excellent performance and is widely used in tasks such as image data classification and retrieval. In this chapter, the generated image features are input into the fully connected layer designed by the SHGN algorithm to achieve image classification.

MaxNet [6]: is a deep convolutional neural network-based model for image retrieval. It extracts the features from various pipelines in the inception module and aggregates the maximizing values. Tthe softmax classification results are adopted.

GCN [34]: is a deep learning method that is widely used in graph data to implement node embedding representation. The image data is generated according to Rule 3 to generate a connection graph containing only the image data, and GCN is used to learn and classify the node features of the graph data. The generation of image node features is the same as the SHGN algorithm.

GCH [44]: is a graph convolutional hash structure based on GCN, which is used to learn a consistent feature space representation of cross-media data in association graphs. Sigmoid activation is adopted to output predicted labels.

att-RNN [45]: is a multi-model fusion method based on attention mechanism and recurrent neural network. It integrates the text, images, topic hashtags, and links for event classification.

4.1.3 Evaluating measure

When evaluating the result of image classification, the result of image classification is judged according to the class label of each image data. Four common classification evaluation indexes are selected, including accuracy, recall, classification precision and F₁ value.

Precision (P) represents the proportion of the actual positive sample data in the sample data marked as correct by the classifier in the classification result: $precision = tp / (tp + fp)$ (7)

Recall (R) is used to measure the proportion of positive samples that are marked as correct by the classifier: $recall = tp / (tp + fn)$ (8)

Accuracy is the ratio of the number of correctly classified samples to the total number of samples in the classification: $Accuracy = (tp + tn) / (tp + tn + fp + fn)$ (9)

F Score (F₁) is a comprehensive index, which is the harmonic mean of precision and recall.

$2 / F_{1} = 1 / P + 1 / R$ (10) $F_{1} = 2 * (P * R) / (P + R) = 2 tp / (2 tp + fp + fn)$ (11)

where tp and tn represent the number of positive samples and negative samples that are correctly classified in the classification results of the same category, respectively. fp and fn are the number of wrongly classified positive samples and negative samples quantity.

4.2 Performance comparison

To more comprehensively evaluate the ability of the SHGN algorithm to learn the image semantics, the proposed SHGN and image feature expression methods are compared in image classification performance. The typical image feature representation, and semantic learning algorithms are selected, and experiments are carried out on three datasets respectively. We report the four evaluation indicators to reveal the performance of image classification. The performance comparison is shown in Fig. 3.

Fig. 3

Performance comparison of SHGN and baselines in classification task.

From the comparison results in Fig. 3, it can be seen that the SHGN algorithm has achieved a high improvement in the four evaluation indicators of P, R, F₁, and Accuracy. VGGNet-19, MaxNet and GCN algorithms use deep neural networks to perform global extraction and representation learning of data features in a deeper level, so they are significantly better than the shallow topic model algorithm Corr-LDA. Compared with deep learning-based deep feature representation methods VGGNet-19 and MaxNet, the performance of SHGN algorithm is significantly improved. Among them, the classification effect is better than that of VGGNet-19 and MaxNet. The results indicat that the SHGN algorithm is based on VGGNet-19, and the social heterogeneous graph convolution model effectively supplements and mines image semantic features. MaxNet does not consider the context of image in social networks. Thereby SHGN improves the performance of image semantic learning. Compared with the GCN algorithm, SHGN achieves better classification effect. The reason is that the SHGN introduces the multi-attribute feature of the social network, and generates a compact graph model with rich semantics in the construction of the social heterogeneous graph. The embedding representation of nodes in learning is more adequate and complete. Under the comprehensive comparison of the four indicators, it fully shows that the SHGN algorithm has a good ability to representation image semantics.

4.3 Ablation study

Topic hashtags and hyperlink information play a decisive role in constructing the social heterogeneity graph. In order to verify the influence of the above two types of social network multi-attribute features on the SHGN algorithm, the related connection conditions in the graph model establishment rules are deleted, and a new graph model is established in the SHGN algorithm. Among them, the method of deleting the “co-h” connection relationship in the graph model building process is expressed as SHGN^-h. The approach for removing connection rules related to “co-URL" is denoted as SHGN^-u. To make the comparison effect more significant, experiments are carried out on three datasets. Table 2 reports the experimental results on the two evaluation indicators of F₁ and Accuracy.

Table 2
Influence of multi-attribute features of social networks on SHGN

Algorithm West Texas Explosion CrisisLexT6 CrisisLexT6

F ₁ Accuracy F ₁ Accuracy F ₁ Accuracy

Corr-LDA 0.598 0.589 0.609 0.594 0.612 0.602

VGGNet-19 0.620 0.632 0.631 0.644 0.643 0.653

MaxNet 0.641 0.673 0.644 0.663 0.651 0.674

GCH 0.643 0.674 0.646 0.669 0.653 0.678

att-RNN 0.648 0.677 0.650 0.679 0.659 0.680

GCN 0.706 0.721 0.713 0.730 0.726 0.739

SHGN^-h 0.567 0.504 0.569 0.511 0.590 0.521

SHGN^-u 0.602 0.588 0.610 0.624 0.623 0.645

SHGN 0.799 0.785 0.784 0.791 0.789 0.799

Algorithm	West Texas Explosion	CrisisLexT6	CrisisLexT6
Corr-LDA	0.598	0.589	0.609	0.594	0.612	0.602
VGGNet-19	0.620	0.632	0.631	0.644	0.643	0.653
MaxNet	0.641	0.673	0.644	0.663	0.651	0.674
GCH	0.643	0.674	0.646	0.669	0.653	0.678
att-RNN	0.648	0.677	0.650	0.679	0.659	0.680
GCN	0.706	0.721	0.713	0.730	0.726	0.739
SHGN^-h	0.567	0.504	0.569	0.511	0.590	0.521
SHGN^-u	0.602	0.588	0.610	0.624	0.623	0.645
SHGN	0.799	0.785	0.784	0.791	0.789	0.799

In Table 2, we observe that SHGN is better than the comparison algorithms under the two indexes. Notely, deleting the two social network attribute feature can significantly reduce the performance of SHGN. The experimental results of SHGN^-h are more obvious than SHGN^-u, which shows that hashtags play a more important role than “URL” in the construction of a social heterogeneous graph. Hashtags guide the formation of explicit relationships in the graph model, while external link “URL” identifies implicit relationships between nodes with weaker semantic strength. The results demonstrate the importance of two social multi-attribute features in the proposed SHGN algorithm. Because the multi-attribute features of social networks determine the closeness of the relationship in the construction of the social heterogeneous graph, which affects the topology structure between nodes and the process of association aggregation. By reducing the connection rules, the generated new social heterogeneous graph model becomes looser, and the semantic association strength decreases. This result is in the lack of richer semantics in the semantic learning of images and affects the final image classification effect. Besides, compared with multi-modal fusion algorithms GCH and att-RNN, the proposed SHGN also has certain advantages. Because we can not only fuse text semantics but also mine image context based on the heterogeneous graph, which fully enriches image representation. The results verify the effectiveness of the proposed algorithm in data fusion.

4.4 Sensitivity analysis

4.4.1 Influence of GCN layers on SHGN

The number of GCN network layers included in the SHGN algorithm can affect the classification effect of the entire algorithm to a certain extent. It determines the scope of the receptive field in the neighbor aggregation. By setting different layers K, SHGN algorithm can achieve feature convergence of connected nodes in different neighborhoods, so as to achieve different degrees of semantic expansion of image features. Based on different K values, we evaluate the classification performance of SHGN algorithm, and analyze the influence of GCN layer number on the image semantic representation effect of SHGN algorithm. The value of the number of layers K are set to {1, 2, 3, 4}. Under the classification precision, the image classification experiments are carried out on three image datasets. The experimental results are shown in Table 3.

Table 3
Influence of different GCN layers on classification performance of SHGN

Dataset K

1 2 3 4

West Texas Exoplosion 0.778 0.793 0.759 0.735

CrisisLexT6 0.764 0.781 0.763 0.738

CrisisLexT26 0.775 0.790 0.766 0.747

Dataset	K
West Texas Exoplosion	0.778	0.793	0.759	0.735
CrisisLexT6	0.764	0.781	0.763	0.738
CrisisLexT26	0.775	0.790	0.766	0.747

By analyzing the results in Table 3, the following conclusions can be drawn. When the number of GCN layers is K = 2, the precision on the dataset can achieve the best results. From the distribution of P, it can be found that when K > 2, with the increase of K value, the P shows a downward trend. This is because when K = 1, only the neighbors directly connected to the node are aggregated in the convolution operation of GCN. While when K = 2, the semantic relationship between the indirectly connected nodes can be mined, and the nodes can be effectively expanded and replenish. When K > 2, the multi-layer GCN may incorporate unnecessary and weakly semantically related node information, so the P can decrease. So, we set the number of GCN layers of SHGN as 2 to achieve the best node embedding learning performance in all experiments.

4.4.2 Influence of the number of hidden layer units on SHGN

In the proposed SHGN algorithm, the number of hidden layer units of GCN in the social heterogeneous graph convolutional network involves the mapping of node features and the number of reserved bits. Hence, it affects the effect of image semantic feature learning. In this section, the number of units that make the SHGN algorithm achieve the best performance is obtained by cross-validation. We select M as {16, 32, 64, 128, 256} to evaluate the change of classification accuracy of the SHGN algorithm. Table 4 records the classification results within the three datasets.

Table 4
Sensitivity analysis of SHGN to the number of hidden layer units

Dataset 16 32 64 128 256

West Texas 0.669 0.678 0.793 0.699 0.681

Exoplosion

CrisisLexT6 0.683 0.699 0.781 0.711 0.695

CrisisLexT26 0.696 0.713 0.790 0.717 0.701

Dataset	16	32	64	128	256
West Texas	0.669	0.678	0.793	0.699	0.681
Exoplosion
CrisisLexT6	0.683	0.699	0.781	0.711	0.695
CrisisLexT26	0.696	0.713	0.790	0.717	0.701

In Table 4, we can observe that when M = 64, SHGN achieves the best classification accuracy on all three datasets. The reason is that when the dimensionality is low, the network model cannot preserve the integrity of node information. At the same time, when the number of hidden layer units is large, the computational complexity will increase significantly. Therefore, when the number of hidden layer units in the SHGN algorithm is 64, the overall model can achieve the best results. In the following experiments, the value of M is set to 64 bits to keep the SHGN algorithm in a better learning state.

4.4.3 Sensitivity analysis of SHGN to datasets

To analyze the sensitivity of the SHGN algorithm to different datasets, this section verifies the impact of different numbers of event categories in the dataset and different proportions of social multi-attributes on the SHGN algorithm through experiments. The performance of the SHGN algorithm for binary classification and multi-classification is tested against three datasets.

Fig. 4 reveals the training process of the SHGN algorithm on various datasets. The recall of SHGN for 200 iterations are obtained on different datasets. From the results of Fig. 4, it can be found that the recall rate of each dataset shows a rising trend with small fluctuations. Combined with the statistical information of the dataset in Table 1, it can be seen that the dataset “CrisisLexT26” contains the largest number of event categories, but its recall rate is the highest. The multi-classification performance on the multi-event dataset is better than the binary classification effect on the single-event dataset. It shows that the SHGN algorithm has strong stability in both binary classification and multi-classification problems. Therefore, the number of categories of the dataset has less influence on the SHGN algorithm.

Fig. 4

Training process of SHGN under different datasets.

In addition, the influence of the number of multi-attribute features of social networks on the SHGN algorithm are analyzed by combining the results in Fig. 4 and the information in Table 1. We can see that the higher the proportion of topic hashtags and “URL” contained in the dataset, the better the SHGN algorithm can obtain the better classification recall. Therefore, the Recall is positively related to the proportion of social multi-attribute features in the dataset. This is because the content of multi-attributes in social networks can semantically associate independent and scattered image data, which determines the degree of compactness of connected edges in social heterogeneous graphs. It affects the compactness of semantic relationships in social heterogeneous graph models. In social heterogeneous graphs with different semantic closeness, SHGN algorithm will generate node feature representations with different aggregation degrees through GCN embedding learning. The number of social attributes involve the semantic expansion degree of image nodes, so they affect the classification effect of SHGN algorithm.

5 Conclusions

This paper proposes an image semantic feature representation algorithm based on the social heterogeneous graph network. The semantic correlation of image context is discovered and a social heterogeneous graph model is constructed based on social attributes. The key text semantics (i.e. topic hashtag) can be effectively correlated and supplemented with the visual information by the node aggregation operation of the graph convolutional neural network. The effective extended representation of the image semantics of social network is realized. In classification experiments on the Twitter image dataset, there is a significant performance improvement compared to contrasting algorithms.

References

Sanjay Kumar , Muskan Saini , Muskan Goel and Panda

B.S.

, Modeling information diffusion in online social networks using a modified forest-fire model, J Intell Inf Syst 56(2) (2021), 355–377.

Mai

, Li

, Jeong

, Quispe

and Sanner

, Online continual learning in image classification: An empirical survey, Neurocomputing 469 (2022), 28–51.

Seal

, Bhattacharjee

and Yazidi

, Deep learning techniques for infrared image/video understanding, Multim Tools Appl 81(7) (2022), 9245.

Liang

, Yuan

, Zhou

and Su

, Opinion separation in leader-follower coopetitive social networks, Neurocomputing 434(1), 2021.

Weigang Wang , Juchao Ma , Chendong Xu , Yunwei Zhang , Ya Ding , Shujuan Yu , Yun Zhang and Yuanjian Liu , Low rank sparse feature selection for image classification, Expert Syst Appl 189 (2022), 115685.

Saddam Hussain , Muhammad Ahmad Zia and Waqas Arshad , Additive deep feature optimization for semantic image retrieval, Expert Syst Appl 170 (2021), 114545.

JiancanWu , Xiangnan He , XiangWang , QifanWang , Weijian Chen , Jianxun Lian and Xing Xie , Graph convolution machine for context-aware recommender system, Frontiers Comput Sci 16(6) (2022), 166614.

Hao Wang , Liyan Dong and Minghui Sun , Local feature aggregation algorithm based on graph convolutional network, Frontiers Comput Sci 16(3) (2022), 163309.

Zhouxia Wang , Tianshui Chen , Jimmy Ren

S.J.

, Weihao Yu , Hui Cheng , Liang Lin , Deep reasoning with knowledge graph for social relationship understanding, In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 1021–1028. ijcai.org, 2018.

10.

Pouya Ghiasnezhad Omran , Kewen Wang and Zhe Wang , An embedding-based approach to rule learning in knowledge graphs, IEEE Trans Knowl Data Eng 33(4) (2021), 1348–1359.

11.

Rex Ying , Ruining He , Kaifeng Chen , Pong Eksombatchai , William Hamilton

, Jure Leskovec , Graph convolutional neural networks for web-scale recommender systems, In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pages 974–983. ACM, 2018.

12.

Fei Jie , Fei Xie , Lei Li and Xindong Wu , Implicit incident detection in social networks, Journal of Automation 44(4) (2018), 13.

13.

Cunhua Li Zhaoman Zhong , Yan Guan and Zongtian Liu , Detection of top-k emergencies in microblog network area, Journal of Computer Science 41(7) (2018), 13.

14.

Chang Su , Min Chen and Xianzhong Xie , Graph convolutional matrix completion via relation reconstruction, In 10th International Conference on Software and Computer Applications, Kuala Lumpur, Malaysia, February 23-26, 2021, pages 51–56. ACM, 2021.

15.

Chuxu Zhang , Dongjin Song , Chao Huang , Anuanthram Swami , Nitesh Chawla

, Heterogeneous graph neural network, In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery&Data Mining, pages 793–803. ACM, 2019.

16.

Chuan Shi , Jiayu Ding , Xiaohuan Cao , Linmei Hu , Bin Wu and Xiaoli Li , Entity set expansion in knowledge graph: a heterogeneous information network perspective, Frontiers Comput Sci 15(1) (2021), 151307.

17.

Jianran Liu , Shiping Wang and Wenyuan Yang , Sparse autoencoder for social image understanding, Neurocomputing 369 (2019), 122–133.

18.

Zechao Li , Jinhui Tang and Tao Mei , Deep collaborative embedding for social image understanding, IEEE Trans Pattern Anal Mach Intell 41(9) (2019), 2070–2083.

19.

Zhanyu Ma , Andrew Teschendorff

, Arne Leijon , Yuanyuan Qiao , Honggang Zhang and Jun Guo , Variational bayesian matrix factorization for bounded support data, IEEE Trans Pattern Anal Mach Intell 37(4) (2015), 876–889.

20.

Zhanyu Ma , Jing-Hao Xue , Arne Leijon , Zheng-Hua Tan , Zhen Yang , Jun Guo , Decorrelation of neutral vector variables: Theory and applications, IEEE Trans Neural Networks Learn Syst 29(1) (2018), 129–143.

21.

Jun Yu , Zhenzhong Kuang , Baopeng Zhang , Wei Zhang , Dan Lin and Jianping Fan , Leveraging content sensitiveness and user trustworthiness to recommend fine-grained privacy settings for social image sharing, IEEE Trans Inf Forensics Secur 13(5) (2018), 1317–1332.

22.

Karen Simonyan , Andrew Zisserman , Very deep convolutional networks for large-scale image recognition, In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, pages 1–14, 2015.

23.

Shaowei Liu , User intention oriented social image retrieval, Tsinghua University, 2016.

24.

Huilan Luo Xiao Li and Kang Tong , An image semantic segmentation method based on context and shallow spatial codec network, 2020.

25.

Sarvamangala

D.R.

and Raghavendra Kulkarni

, Convolutional neural networks in medical image understanding: a survey, Evol Intell 15(1) (2022), 1–22.

26.

Rani Saritha

, Varghese Paul and Ganesh Kumar

, Content based image retrieval using deep learning process, Clust Comput 22(Supplement) (2019), 4187–4200.

27.

Xiaofeng Yang , Zhe Wang , Hongxia Deng , Haifang Li , Rong Yao , Peng Gao and Saddam naji abdu Nasher , Recognizing image semantic information through multi-feature fusion and ssae-based deep network, J Medical Syst 44(2) (2020), 46.

28.

Zechao Li and Jinhui Tang , Weakly supervised deep matrix factorization for social image understanding, IEEE Trans Image Process 26(1) (2017), 276–288.

29.

Zechao Li , Jing Liu , Changsheng Xu , Hanqing Lu , Mlrank: Multi-correlation learning to rank for image annotation, Pattern Recognit 46(10) (2013), 2700–2710.

30.

Zechao Li and Jinhui Tang , Weakly-supervised deep non-negative low-rank model for social image tag refinement and assignment, In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4–9, 2017, San Francisco, California, USA, pages 4154–4160. AAAI Press, 2017.

31.

Xiangbo Shu , Liyan Zhang , Yunlian Sun , Jinhui Tang , Host-parasite: Graph lstm-in-lstm for group activity recognition, IEEE Trans Neural Networks Learn Syst 32(2) (2021), 663–674.

32.

Jinhui Tang , Xiangbo Shu , Rui Yan and Liyan Zhang , Coherence constrained graph LSTM for group activity recognition, IEEE Trans Pattern Anal Mach Intell 44(2) (2022), 636–647.

33.

Jinhui Tang , Xiangbo Shu , Zechao Li , Yu-Gang Jiang and Qi Tian , Social anchor-unit graph regularized tensor completion for large-scale image retagging, IEEE Trans Pattern Anal Mach Intell 41(8) (2019), 2027–2034.

34.

Thomas Kipf

and Max Welling , Semi-supervised classification with graph convolutional networks, In 5th International Conference on 960 Learning Representations, ICLR 2017. OpenReview.net, 2017.

35.

Zhao-Min Chen , Xiu-Shen Wei , Peng Wang , Yanwen Guo , Multi-label image recognition with graph convolutional networks, In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pages 5177–5186. Computer Vision Foundation / IEEE, 2019.

36.

Amirreza Salamat , Xiao Luo , Ali Jafari , Heterographrec: A heterogeneous graph-based neural networks for social recommendations, Knowl. Based Syst 217 (2021), 106817.

37.

Xiaoling Long , Chao Huang , Yong Xu , Huance Xu , Peng Dai , Lianghao Xia and Liefeng Bo , Social recommendation with self-supervised metagraph informax network, In Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong, editors, CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1–5, 2021, pages 1160–1169. ACM, 2021.

38.

Yiming Zhang , Lingfei Wu , Qi Shen , Yitong Pang , Zhihua Wei , Fangli Xu , Ethan Chang and Bo Long , Graph learning augmented heterogeneous graph neural network for social recommendation, CoRR, abs/2109.11898, 2021.

39.

Liang Yao , Chengsheng Mao and Yuan Luo , Graph convolutional networks for text classification, In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, pages 7370–7377. AAAI Press, 2019.

40.

Guo-Jun Qi , Charu Aggarwal

, Qi Tian , Heng Ji and Thomas Huang

, Exploring context and content links in social media: A latent space method, IEEE Trans Pattern Anal Mach Intell 34(5) (2012), 850–862.

41.

Alexandra Olteanu , Carlos Castillo , Fernando Diaz and Sarah Vieweg , Crisislex: A lexicon for collecting and filtering microblogged communications in crises, In Proceedings of the Eighth International Conference on Weblogs and Social Media, 2014.

42.

Alexandra Olteanu , SarahVieweg , Carlos Castillo , What to expect when the unexpected happens: Social media communications across crises, In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, pages 994–1009. ACM, 2015.

43.

Tao Chen , Hany SalahEldeen

, Xiangnan He , Min-Yen Kan , Dongyuan Lu , VELDA: relating an image tweet’s text and images, In Blai Bonet and Sven Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25–30, 2015, pages 30–36, 2015.

44.

Ruiqing Xu , Chao Li , Junchi Yan , Cheng Deng , Xianglong Liu , Graph convolutional network hashing for cross-modal retrieval, In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10–16, 2019, pages 982–988. www.ijcai.org, 2019.

45.

Zhiwei Jin , Juan Cao , Han Guo , Yongdong Zhang , Jiebo Luo , Multimodal fusion with recurrent neural networks for rumor detection on microblogs, In Proceedings of the 2017 ACMon Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23–27, 2017, pages 795–816. ACM, 2017.