Infusing external knowledge into user stance detection in social platforms

2.1 Text classification research of introducing external knowledge

Infusing external knowledge can improve the performance of many natural language processing tasks (NLP), including text classification. Chen et al. [21] proposed a natural language model on the basis of neural networks and combined it with external inference knowledge. The model uses a co-attention mechanism to achieve soft-alignment of premises and hypotheses and then merges synonym, antonym, hyponym, and hypernym shared between the words included by premises and hypotheses as relational features of the two words into the co-attentive mechanism. Experimental results demonstrate that utilizing external knowledge can effectively improve the model’s performance and external knowledge by providing additional valid information when the training data is limited. However, excessive external knowledge may increase the amount of noise to the model, eventually decreasing the prediction performance of the model. To reduce the noise, the KES model [22] uses PPR to filter irrelevant external knowledge while introducing external knowledge to obtain external knowledge that is more relevant to the given text object. The model introduces external knowledge using WordNet, an open source knowledge graph, for key information in the premise and hypothetical text, and constructs subgraphs with key information and external knowledge as nodes. However, for most of the classification models that introduce external knowledge, the prediction performance of the model is seriously affected by the excessive noise caused by the external knowledge to the model. The model adopts the PPR filtering method to filter the irrelevant external knowledge in the graph to reduce the noise impact of the external knowledge on the model. Meanwhile, to capture the structural and semantic information of the knowledge graph, the model uses relational graph convolutional neural network (RGCN) to encode subgraphs. The experimental results show that appropriately introducing external knowledge and using PPR to filter irrelevant external knowledge can capture the effective information from the large-scale and noisy knowledge graphs and further improve the detection performance of the model.

Hence, this paper extends the method of introducing external knowledge to stance detection. By using WordNet to introduce external knowledge to the review and applying PPR filtering method to reduce noise to enhance the performance of our model. In addition, this paper uses RGCN to encode the subgraph to obtain the vector representation for the review.

2.2 Stance detection research of incorporating structural information

Stance detection incorporating structural information aims to build the stance detection model that takes into account the network structural relationships between reviews. Several studies have found that in addition to the textual features of reviews, structural information also can improve model performance of stance detection. BranchLSTM [23] is a time-series model based on stance detection model that uses conversation threads to introduce the structural relationships of the reviews. Each conversation thread comprises an original review and some embedded replies, where the embedded replies may be a response to the original review or a re-reply review to another embedded replies. The model divides each conversation thread into multiple linear chain branches. A branch is a linear branch of a leaf node review in a conversation thread and that node going up the parent node to the root node, which contains the original review and multiple embedded replies. The model uses word2vec to combine the word vectors of the review text with additional extracted features to obtain the initial vector representation of the reviews during data preprocessing. It then feeds the reviews contained in each branch into the LSTM model in order of depth in the branch to achieve information propagation of the relevant reviews. The model is further improved by constructing a time series model to combine structural information between reviews and text features. In addition to the use of time series models, incorporating structural information can be achieved using graph neural network methods. Li et al. [17] used news articles as the research object and first used deep learning based on text features, such as Hierarchical LSTM [24], Skip-Thought Embedding [25], Bias Features [26], and several other methods. Then, GCN is used to obtain network information between tweets for graph models constructed with political figures, users, and news articles as objects. The experiments show that the effect of joint text and network features for stance detection is significantly better than that of stance detection using either of the features.

To incorporate structural information and textual features, this paper constructs a graph model which based on deep learning networks, to obtain the network structure information between different reviews.

This part introduces external knowledge by using the subgraphs mapped from the reviews in the knowledge graph as contexts. First, the knowledge graph retrieves external knowledge related to the reviews, and a contextual association subgraph of the reviews is extracted in the knowledge graph. Given that the introduced external knowledge is often overly large and contains significant noise, PPR method filters the contextual association subgraphs for external knowledge. Finally, a relational graph convolutional network is used to encode the contextual association subgraph and obtain a vector representation of the reviews. The representation learning model of reviews is shown in Fig. 2.

OPEN IN VIEWER

Fig. 2

Review representation learning model by introducing external knowledge.

3.1.1 Review on subgraph extraction of external knowledge

To extract a contextual subgraph of the review text from the knowledge graph, key information in the review must be mapped to the knowledge graph to retrieve external knowledge related to the review. Then, the contextual subgraphs associated with the reviews are extracted from the knowledge graph to achieve the introduction of external knowledge. For the target review, stopwords are filtered to obtain keywords based on the review. The set of words constructed by the keywords extracted from the reviews is denoted as: W = ( w 1 , w 2 , … , w n ) ∈ TC(1) where TC is the target review, W is the set of keywords extracted in TC, and w _i is some keyword extracted in TC.

To introduce external knowledge related to the reviews, W must be mapped to the WordNet knowledge graph. For the word w _p in the review TC, the words that are semantically related to the word and the semantic relations between them are extracted in the WordNet dictionary to construct a contextual association subgraph on the basis of the target review. The semantic relations considered include:

Hypernym: They are used to specify the category to which a specific instance belongs. If X is a type of Y, then Y is a hypernym of X. For example, “flower” is the hypernym of “rose”.

Hyponym: A special term used to specify the members of a class. If X is a type of Y, then is the hyponym of Y. For example, “rose” is the hyponym of “flower”.

Synonym: Words that have the same meaning, such as “father” and “dad”.

Entailment words: Words that have an implied relationship between two words. If a verb X implies Y, then X can be completed only when Y has already been completed. For example, “to eat” implies “to swallow”.

For example, the text of review TC is “The young dancing girl is in red dress” and the review must be filtered for stopwords to obtain the keywords on the basis of the review. The set of keywords extracted from the review is young, girl, red, dress, dance.

The extracted keywords are then mapped to the Wordnet knowledge graph to extract the contextual association subgraph G. First, keywords extracted from the review are used as the initial nodes, and then the knowledge graph is used to retrieve the first-order neighbors of each initial node. Figure 3 shows that the word “girl” in TC is mapped to the knowledge graph for contextual subgraph matching to achieve external knowledge introducing. Context matching for a single word yields a contextual subgraph of a single word comprising an initial node, its first-order neighbors in the knowledge graph, and the edges connecting these nodes.

OPEN IN VIEWER

Fig. 3

Matching process of external knowledge contextual association subgraph of keywords.

To introduce the external knowledge of the reviews, the contextual words of all keywords in the reviews and the semantic relationships between these words must be extracted from the knowledge graph. Figure 4 shows the contextual information extracted for all keywords in the reviews to construct a contextual association subgraph, denoted as graph G. The red nodes denote the initial nodes, i.e., the keywords W. The black nodes comprise the first-order neighbor nodes that are retrieved from the knowledge graph.

OPEN IN VIEWER

Fig. 4

Subgraph constructed by key information of review.

3.1.2 Filtering of contextual association subgraphs

The subgraph G obtained after contextual matching of reviews may contain substantial noise from the first-order neighbor nodes. For example, the concept of “girl” is directly mapped to WordNet with over 500 first-order neighbor nodes, and a large number of these nodes are only related to the semantic relationship of the word, not to the current review. In this paper, we employ PPR algorithm to unrelated nodes and reduce noise from excessive external knowledge. The PPR algorithm adds a bias to each node on the basis of the PPR: p i = { 1 | W | , i ∈ W 0 , i ∉ W(2) where W denotes the initial set of nodes in the graph, i.e., the set of keywords extracted from the reviews. The deviation of node i is 1 | W | if the word it represents is a keyword of the review, and it is 0 if it does not belong to the review. The purpose is to select initial nodes of the review in graph G have higher importance. First, the PPR scores of all nodes are initialized randomly. Then, the PPR scores of the nodes in graph G are updated according to Equation 3 until convergence. PP R t = ( 1 - α ) A × PP R t - 1 + α p(3) where p = [p₁, p₂, …, p _n ]  ^T denotes the number of nodes. PPR _ti denotes the PPR score of the node i in the tth iteration, and PPR _t  = [PPR_t1, PPR_t2, … PPR _tn ]  ^T . A ∈ R^n×n is the adjacency matrix of graph G. αϵ (0, 1) is the decay factor, and setting different values of α can converge the PPR scores to different values.

After the iteration is completed, the PPRϵR ⁿ is normalized to obtain PPR ˜ . Then a threshold θ is used to filter the graph G. The nodes with PPR ˜ score less than θ and the edges connecting these nodes are removed, and the obtained filtered contextual association subgraph is denoted as G′. For Fig. 4, the PPR method is used to filter the subgraph, and the obtained contextual association subgraph is shown in Fig. 5.

OPEN IN VIEWER

Fig. 5

Association subgraph of review context after filtering nodes.

This section considers the external structural relationships between reviews by using the GGNN model to capture the structural features between reviews. Given that reviews are structured as conversation threads on a topic, conversation threads are used to construct the graph model.

This section considers the external structural relationships between reviews by using the GGNN model to capture the structural features between reviews. Given that reviews are structured as conversation threads on a topic, conversation threads are used to construct the graph model.

First, the definition of a conversation thread is given. For a particular topic T, S is an original review posted on that topic. Moreover, the review S comprises a series of replies R _i .

In Fig. 6, the original review S is replied by users with ids 6l5b2no, dl5xleh, and dl5hnsx. Meanwhile, user dl5xleh is replied by user dl6bm8l. Based on the above interactions between users, the original reviews and all replies between users are constructed as a conversation thread. In this conversation thread, each review holds a stance on topic T. The conversation thread contains the original review and multiple replies, i.e. C< –Source: [Reply1, Reply2:[Reply3], Reply4], and a mutual relationship exists between the reviews and the reviews.

OPEN IN VIEWER

Fig. 6

Conversation threads for reviews.

The conversation thread shown in Fig. 6 can be constructed as the graph model, as shown in Fig. 7. Each node represents a review with its corresponding stance label, and the edges of different colors represent various depths of the conversation thread. To investigate the effect of review depth on stance detection in the conversation threads, the depth is constructed for all reviews in this study. This depth is represented as the depth size of different reviews located in the conversation thread.

Given that the reviews of each conversation thread affect one another, extracting the structural features of the reviews can contribute to detecting the stance of reviews. In this paper, a GGNN [28] is used to capture the structural features between reviews. The method uses GRU units [29] to efficiently propagate information to neighboring nodes of the target node to capture relevant features of other nodes. The GGNN method is an extension of GNN [30] that does not need to reach node feature convergence to obtain the representation vectors of nodes and is able to handle heterogeneous graphs. Before passing the nodes’ feature representations FϵR ^d into the GGNN model, the adjacency matrix A for the graph model must be constructed. To make the stance detection model capable of handling directed heterogeneous graphs, the outgoing edges, the incoming edged, and types of edges are taken into account when constructing the adjacency matrix.

The set of nodes of the graph model is assumed to be NϵR ⁿ (denotes the number of nodes), and r denotes the total depth of the conversation thread of graph model, with each depth denoted as a type of edge. The adjacency matrix is denoted as A = [A _in ;A _out ]ϵR^{n,(n ×r ×2)} where A _out ϵR^(n,n ×r) and A _in ϵR^(n,n ×r) represent the adjacency matrix of the outgoing edge and the adjacency matrix of the incoming edge, respectively. In Fig. 7, two types of edges are defined. This includes the edge of a node with Depth 0 pointing to a node with Depth 1, which is denoted as a “type 1” edge, and the edge of a node with depth 1 pointing to a node with Depth 2, which is denoted as a “type 2” edge. Then, the outgoing edge adjacency matrix and the incoming edge adjacency matrix are constructed on the basis of the edge types. The graph model shown in Fig. 7, which constitutes the adjacency matrix of incoming and outgoing edges as shown in Fig. 8.

OPEN IN VIEWER

Fig. 7

Graph model based on conversation threads.

The information propagation process of GGNN is described in the following.

OPEN IN VIEWER

Fig. 8

Adjacency matrix.

H in ( t - 1 ) = W 1 H ( t - 1 ) + b 1 H out ( t - 1 ) = W 2 H ( t - 1 ) + b 2(6)

Where W₁, W₂ and b₁, b₂ denote learnable parameters. Given that we consider incoming and outgoing edges separately, the node hidden states H^(t–1) also consider the hidden representations H out ( t - 1 ) , H in ( t - 1 ) for node information entry and node information output, respectively. Before information propagation, the feature vectors of all nodes in the graph model of the conversation thread are stitched to construct the feature matrix XϵR^(n,d), which will be used as the feature representation of the conversation thread. Where n is the number of nodes of the conversation thread and d is the feature dimension of the review. The feature matrix of this conversation thread is used to initialize the feature matrix of the model H⁽⁰⁾ϵR^(n,d), i.e., H⁽⁰⁾ = X.

For H out ( t - 1 ) and H in ( t - 1 ) , the neighboring matrices A _in and A _out of the corresponding outgoing and incoming edges are multiplied to capture the features of the neighboring nodes, respectively. Here, the matrix representations of the outgoing and the incoming edges, which are computed separately, are stitched together to obtain the input value I^(t) of the model at moment t. I ( t ) = [ A in H in ( t - 1 ) ; A out H out ( t - 1 ) ](7)

The next equation is the update process of the model similar to GRU, where I^(t) is used as the input value and H^(t–1) as the hidden state at the previous moment. Z ( t ) = σ ( W z I ( t ) + U z H ( t - 1 ) ) ,(8) R ( t ) = σ ( W r I ( t ) + U r H ( t - 1 ) ) .(9) where Z^(t) and R^(t) denote the update gate and the reset gate, respectively, and W ^z , U ^z and W ^r , U ^r are the learnable parameters of the update gate and the reset gate, respectively.

After obtaining the data of the reset gate and update gate, the hidden state H^(t–1) outputted at the previous moment must be reset and then updated using the data output from the update gate.

H ( t ) ˜ = tanh ( W I ( t ) + U ( R ( t ) ⊙ H ( t - 1 ) ) ) , H ( t ) = ( 1 - Z ( t ) ) ⊙ H ( t - 1 ) + Z ( t ) ⊙ H ( t ) ˜ .(10)

The reset the gate dot the hidden state to reset the hidden state output at the previous moment, and then the data are concatenated with the input value I^(t) and multiplied by the corresponding learnable parameter matrix to obtain H^(t). Next, the update gate Z^(t) is used to update H^(t–1) and H ( t ) ˜ to obtain the output hidden state H^(t).

Through the above node information propagation, the final hidden representation H^(T) ∈ R^n×d′ of all nodes is obtained after iterating over a certain time step T, where d′ is the dimension of the node output.

By splicing the initial feature matrix XϵR ^d of the conversation thread and the final hidden representation H^(t), the nodes’ own features are preserved, and then inputting yϵR ^{n ×c} to the activation function. y ˆ = softmax ( H ( T ) ; X ) .(11)

The problem addressed in this paper is a multi-classification problem (support, negative, comment and query). Therefore, the cross-entropy loss function is used to train the model in this paper, loss = - ∑ i = 1 M ∑ j = 1 N y i j log ( y i j ˜ )(12) where M denotes the set of all reviews and N denotes the set of stance types. y i j is the stance label of the review and y i j ˆ is the predicted stance. In the process of training stance detection, the goal is to minimize the loss function to update the parameters to reach the optimal solution.

4.1 Dataset

The dataset used in this paper is the data of rumor stance detection published for Task 7 of Semeval 2019 competition. It includes 8,574 comments, of which 1,184 are labeled as supportive, 606 as negative, 608 as querying, and 6,176 as commenting. The reviews in the dataset focus on nine different topics, such as climate change and natural disasters. The stances expressed in the reviews of each conversation thread are specific to that topic. The training set contains 272 conversation threads, the validation set contains 25 conversation threads, and the test set contains 28 conversation threads. The distribution of the dataset is shown in Table 1.

Then, we cleaned the review texts in the datasets. Reviews with blank or contained useless characters were deleted, to prevent noise from affecting model training. The statistics of the data after preprocessing are shown in Table 2

The next step is to remove stopwords from the reviews, and NLTK stopwords corpus is used. To introduce external information, WordNet is used in this paper to match hypernym, hyponyms, synonyms, and entailment words of the review keywords.

This paper employs three evaluation metrics: precision rate P, recall rate R, and F₁ score, which is the arithmetic mean of precision rate and recall rate. P = TP TP + FP(13) R = TP TP + FN(14) MacroF 1 = F S + F C + F D + F Q 4(15)

TP, FP, FN, and TN represent the number of predicted positive samples that are correct, the number of predicted positive samples that are incorrect, the number of predicted negative samples that are incorrect, and the number of predicted negative samples that are correct, respectively.

As shown in Table 2, there were many “comment” categories in the dataset, and very little data in the “deny” and “query” categories. Limited data exists for the “deny” and “query” categories. Due to the serious imbalance in data distribution, the macro-average scores are used to evaluate the performance of the constructed stance detection model for this multi-class problem.

In the word vector acquisition of the review text, the Glove word vector with dimension 200 is utilized for initialization. For the review context extracted using external knowledge from WordNet, the RGCN method is utilized to encode the reviews. The size of the hidden layer is set to 200, the dimension of output review encoding to 300, and set two base vectors. When building the GGNN stance detection model for information propagation, three information propagation steps are set. Meanwhile, through several experimental comparisons, the best performance of the model is achieved when the learning rate is set to 0.001, and the batch size is set to 32.

4.4 Baselines

4.5 Analysis of experimental results

The experiments in this paper are stance detection for the information of review on social platforms, namely, Twitter and Reddit. Table 3 verifies the effectiveness of the model proposed in this paper by comparing the KRGGNN stance detection model with other benchmark models. Table 3 presents that the KRGGNN and TreeGGNN models have outstanding performance compared with other models, and the KRGGNN model has the highest accuracy score among all models, and the recall rate is only second to TreeGGNN. The BranchGRU model performs best in terms of recall, considering not only the structural relationship between reviews, but also exploiting the temporal relationship between reviews published sequentially. The KRGGNN model extracts only the structural features between comments. However, KRGGNN even outperforms the new Transformer model BERT, which ignore external knowledge and structural information of reviews.

As shown in Table 4, KRGGNN achieves 1.5% higher macro-average scores compared with TreeGGNN, which does not introduce external knowledge for the context of the reviews. This suggests that external knowledge improves stance detection performance. CLEARumor model has the worst performance compared with other models. It does not use structure or external knowledge, only text features. This shows that text features alone cannot effectively extract stance. Table 4 also provides the F₁ scores of the model on the basis of the four stance categories. Then, the evaluation score of “deny” is significantly lower than the other categories in all models. Considering two possible reasons: (1) the uneven distribution of the original dataset, with the ratio of “deny” data to other data being 1 : 11, this also leads to a certain bias of the model toward other categories of data and undertraining of that category; (2) for a given “deny” review text, which generally behaves as an implicit expression of negative thoughts. Given that tone of the text and the way of expression play crucial roles in whether the model can accurately extract valid information, identifying the “deny” reviews is also difficult for the model. Meanwhile, compared with other baselines, the model’s performance is more stable in each category compared with other models. Therefore, the overall of KRGGNN is better than the baseline models.

To investigate the effect of the number of GGNN propagation steps on the model performance in the stance detection model, the trend of MacroF₁ score is observed by setting different filtering thresholds θ for external knowledge in the step from [1, 4]. In which, the threshold θ of PPR was also adjusted as a hyperparameter. The values for θ in {0.2, 0.4, 0.6, 0.8} are taken accordingly, and the macro-average scores trend of the graph based on different datasets are shown in Figs. 9–11.

OPEN IN VIEWER

Fig. 9

Trend of macro-average scores of the model with different propagation steps.

OPEN IN VIEWER

Fig. 10

Trend of macro-average scores of the model with different propagation steps for Reddit platform data.

OPEN IN VIEWER

Fig. 11

Trend of macro-average scores of the model for Twitter platform data with different propagation steps.

Figure 9 exhibits that the overall performance of our model is optimal when the threshold value θ is set to 0.8, while the worst performance is achieved when θ= 0.2. Considering that the larger the threshold value is, the more external information is filtered. The extracted external knowledge brings more favorable additional background knowledge to the text content. However, an overly low threshold introduces excessive, useless external knowledge, creating noise that negatively impacts prediction effectiveness. Figures 9–11 demonstrate the trend of MacroF₁ derived from data located on Reddit and Twitter data. The experimental results also show that GGNN propagation steps also affect model prediction performance. When the number of propagation steps increases from 1 to 3, the average macro scores also increase, indicating that the model’s performance can be effectively improved by information propagation between nodes, i.e., by using the network structure relationship between nodes. The prediction effectiveness decreases when the number of propagation steps rises to 4, which may be due to overfitting.

From a macroscopic point of view, all the models perform poorly for the given stance detection data. To investigate the reasons for their existence, this paper analyzes them in terms of the depth of the reviews (Table 5). Given the reviews with depth 0 have the highest percentage of “Support” data, the best performance is predicted for this category. Currently, reviews with depth 0 are the original reviews on a topic, and the number of “Deny” category with this depth is 0. Therefore, the evaluation score of the “deny” category with this depth is abnormal and does not have any reference. The number of “query” reviews is mainly distributed in the data of depth 0, and the model has a considerable bias toward other categories in the cases of Depths 1 and 2, thus the prediction is not good. The number of “comment” category is the highest in the whole data, and it is concentrated in the data of Depths 1 and 2, and the experiments show that the model can detect the stance more accurately on the basis of this category.

We also investigated the impact of using different word vectors on the model. During the preprocessing of the data, word2vec and glove were used to construct the initial vector representation of the vocabulary. The vectors obtained based on these two methods were input into the model for training. A comparison of each evaluation metric is shown in Fig. 12.

Figure 12 exhibits that the initialization of the word vectors using glove yields better scores in all aspects than word2vec. Compared with word2vec, the word vectors obtained by the statistically based model glove are more effective for stance detection. Given that the dataset comprises datasets from Reddit and Twitter platforms, this paper explores the effect of the data from the platforms on the model, as shown in Fig. 12(b).

OPEN IN VIEWER

Fig. 12

Comparison of prediction performance of initialized word vectors for w2v and glove-based methods in different platforms.

The results show that the model significantly outperforms Twitter for the Reddit dataset, possibly because Reddit users express more formal reviews, whereas Twitter reviews are more colloquial, which makes it easier for the model to extract feature information from Reddit reviews compared with Twitter. However, the F_1 of the model on the Reddit platform is 0 for the stance category “Deny.” Given that the number of “Deny” in the given Reddit data is extremely small, the model cannot learn the basic features of the “deny” category reviews during training. Therefore, it is poor at predicting this category.

The KRGGNN model incorporates external knowledge into a stance detection framework. Existing stance detection models only consider the internal semantic features of reviews, ignoring external knowledge. Inspired by the text entailment of introducing external knowledge, this work infuses external knowledge into stance detection model in this paper. To verify the effectiveness of external knowledge, we compare the performance of five benchmark models with the KRGGNN stance detection model in this paper. Tables 2 and 3 show that KRGGNN is significantly outperforms the baselines models in macro-average and precision metrics, indicating the importance of external knowledge.

In addition, the KRGGNN also incorporates structural information between reviews, i.e., captures the interactions between reviews. As shown in Table 3, the TreeGGNN and KRGGNN models that incorporate structural information outperform the other benchmark models in terms of macro-average scores, which indicates that incorporating structural information can effectively capture stance information. However, the KRGGNN improves the macro-average score by 1.5% compared to the TreeGGNN model without external knowledge, which also demonstrates that the introduction of external knowledge on top of the incorporation of structural information can be effective for stance detection.

Furthermore, we investigate the effect of different propagation steps on the model performance. Figures 9–11 show performance improves as propagation increases from 1 to 3 steps, indicating the model adequately captures stance. However, the performance of the model decreases when the propagation step is 4. This also indicates that the model is able to capture the stance information adequately when the propagation step is 3 and overfitting occurs when the propagation step is 4. Finally, we investigate the performance of the model at different depths of reviews. In general, the model shows a decreasing trend of effectiveness as the depth continues to increase. Given the different expressions of the reviews at different depths, the textual replies to the reviews when the depth is larger may simply be a reply to the previous reviews. We can only capture the current review’s stance on a specific topic by studying the feature information of the previous reply-review, and in a sense the greater the depth the more implicit the review-expression is, which also makes it more difficult to detect the stance.