Enhancing question answering in educational knowledge bases using question-aware graph convolutional network

In this paper, the question description is separated from the entity to be queried for question modeling. Specifically, the question description can capture the more essential intention unrelated to the entity, while the entity to be queried can be modeled according to the structural information of the original knowledge base. The two adopt the “late fusion” strategy in information fusion to achieve complete semantic supplement when predicting the answer. For the question, this paper first converts the entity into the special symbol “ENT”, and then the Chinese word segmentation tool “jieba” for word segmentation is used to get the serialized representation of the question q = [w₁,  w₂, ⋯ , w_n-1, w _n ], where n represents the length of the sentence and w _i represents the i word. Then the Transformer with multiple heads’ attention is used to model and calculate the question expression e _q in formula (1) [25]. e q = Transformer ( [ w 1 , w 2 , ⋯ , w n - 1 , w n ] )(1)

Here are the detailed instructions for modeling Transformers with multiple heads of attention:

Step 1: Input encoding: The input sequence is first encoded using an embedding layer, which maps each token to a vector representation. This is followed by position encoding, which adds positional information to the embeddings.

Step 2: Multi-head attention: The next step is to perform multi-head attention. In this step, the input sequence is split into multiple segments or heads, and attention is calculated independently for each head. This allows the model to attend to different parts of the input sequence in parallel.

Step 3: Attention calculation: For each head, attention is calculated by computing a query, key, and value matrix. The query matrix is derived from the previous layer’s output, while the key and value matrices are derived from the input sequence. The attention weights are computed as the dot product of the query and key matrices, normalized using a softmax function. The output of the attention calculation is then the weighted sum of the value matrix, where the attention weights serve as the weights.

Step 4: Residual connections and layer normalization: The output of the attention calculation is added to the input sequence to create a residual connection. The residual connection is then normalized using layer normalization to improve training stability.

Step 5: Feedforward layer: The output of the attention layer is then passed through a feedforward neural network, which contains two linear layers with a ReLU activation in between. This allows the model to learn more complex relationships between the input sequence and the output.

Step 6: Residual connections and layer normalization: The output of the feedforward layer is added to the output of the attention layer to create another residual connection. Like before, this is normalized using layer normalization.

Step 7: Repeat steps 2–6 for multiple layers: The above steps are repeated for multiple layers to create a deep, hierarchical model. Each layer takes as input the output of the previous layer and passes its output as input to the next layer.

The multi-head attention mechanism allows the model to attend to different parts of the input sequence in parallel, while the residual connections and layer normalization help improve training stability and allow for faster convergence. The feedforward layer allows the model to learn more complex relationships between the input sequence and the output.

In order to effectively obtain the structure information of the entity to be queried in the original knowledge base, the model uses the knowledge base embedding technology to initialize the vector representation to obtain the entity representation matrix E (where E _i represents the embedding vector ofentity i.

•Entity Representation Matrix E: The entity representation matrix E is a matrix that represents all the entities in a knowledge graph. Each row of the matrix corresponds to an entity and contains its embedding vector. The embedding vector is a dense vector that represents the entity’s semantic meaning in the knowledge graph. The embedding vectors are learned through a training process that aims to capture the relationships between entities in the knowledge graph.

•Embedding Vector of Entity E _i : The embedding vector of an entity E _i is a dense vector that represents the semantic meaning of that entity in the knowledge graph. This vector is a row vector in the entity representation matrix E, corresponding to the entity E _i . The embedding vector is learned through a training process that takes into account the relationships between entities in the knowledge graph, as well as the co-occurrence patterns of entities in text.

In open-domain QA systems, the entity representation matrix E and the embedding vectors of entities E _i are used to represent the semantic meaning of entities in the knowledge graph. These representations are then used to match the question to the relevant entities in the knowledge graph and generate the answer. The matching process typically involves measuring the similarity between the question and the entity embedding vectors, and selecting the entities with the highest similarity scores as the answer candidates.

Distmult method is adopted by default for embedding in this paper [26]. Based on this, the representation e _H of the query entity set ∈ _H can be extracted by embedding the entity into the vector mean in formula (2): e H = 1 | ɛ ɛ H | ∑ i = 1 ɛ ɛ H E i(2)

The DistMult method is a type of knowledge graph embedding model that represents entities and relations as low-dimensional vectors in a continuous space. It is a bilinear model that computes the score of a triple (h, r, t) as the dot product of the vector representations of the head entity h, relation r, and tail entity t.

In the context of query answering, if there are multiple query entities in the question, one approach is to represent each query entity as a separate vector and then combine them to form a query vector. The query vector can be obtained by taking the average of the individual entity vectors or by using other aggregation methods that take into account the relative importance of each entity to the question.

However, simply averaging the embeddings of multiple query entities may not be sufficient if the different query entities have different importance to the question. In this case, it may be more effective to use a weighted average or some other form of aggregation that takes into account the importance of each entity. For example, one approach is to use attention mechanisms to dynamically weigh the contribution of each entity to the query vector based on its relevance to the question. The choice of aggregation method depends on the specific task and the nature of the query entities, and it may require experimentation to determine the most effective approach.

Graph convolution neural network has gradually attracted widespread attention due to its powerful graph structure modeling ability [27]. It generally follows the pattern of iterative message passing to capture the structured information of the neighborhood of nodes. The representation of layer k + 1 is usually based on the representation of layer k. The graph convolution operation (message passing mechanism) is obtained in formula (3): h v k + 1 = ψ c k + 1 ( h v k , ψ a k + 1 ( { ( h u k , h r k , h v k ) : ( u , r ) ∈ N ( v ) } ) )(3)

Where, h v k ∈ R k , h v k + 1 ∈ R k + 1 respectively represent the embedded vector representation of node v at layer k and layer k + 1, ψ c k + 1 , ψ a k + 1 represent the aggregation function and combination function used to define the graph convolution network; N(v) represents the neighbor “relationship node” pair of node v. In the knowledge base question answering, this model uses graph convolution network to represent the structural features of nodes. For specific problems, considering the different impact of neighborhood information on the target node, this paper defines the following attention graph convolution network for specific questions to realize the modeling of the entity structure features of question perception in formula (4): h v , q k + 1 = ∑ ( u , r ) ∈ N ( v ) a u , q k · W 1 k m u , r , v k + W 2 k h v k(4)

Wherein, W 1 k , W 2 k respectively represent the transformation matrix of neighbor information and target node; m u , r , v k represents the information transfer amount of neighbor triplet (u,r,v) to target node v, and its calculation method is defined in formula (5): m u , r , v k = FF 1 ( h u k ; h r k ; h v k )(5)

a u , q k indicates attention score. For the related description information and query entity information contained in the question, this paper defines the corresponding attention calculation methods, and then combines the two to get a u , q k . Specifically, the question description feature or query entity feature is connected with the representation of the target node and input into the feedforward neural network to calculate the attention value. a u , q k the calculation process is formally expressed in formula (6)-(8): a u , q k , 1 = FF 2 ( e q ; h u k )(6) a u , q k , 2 = FF 3 ( e H ; h u k )(7) a u , q k = softmax ( a u , q k , 1 + a u , q k , 2 )(8)

FF represents the feedforward neural network layer, and the symbol “;” represents the splicing operation between vectors. Through the calculation of equations (6) and (7), the graph volume will comprehensively consider the question description information and the corresponding query entity when transmitting information, so the information related to the question will be enhanced and the irrelevant information will be weakened, so the model can better model the entity representation and predict the answer. At the last layer of the graph convolution network, record the node representation [ h 1 T , h 2 T , ⋯ , h N T ] ( N = | ɛ ɛ q | ) in the subgraph G _q for final answer prediction.

Feedforward neural network (FNN), abbreviated as feedforward network, is a type of artificial neural network. The Feedforward neural network adopts a unidirectional multi-layer structure. Each layer contains several neurons. In this type of neural network, each neuron can receive signals from the previous layer of neurons and generate outputs to the next layer. The 0th layer is called the input layer, the last layer is called the output layer, and the other intermediate layers are called hidden layers (or hidden layers, hidden layers). The hidden layer can be one layer. It can also be multiple layers. There is no feedback in the whole network, and the signal propagates unidirectionally from the input layer to the output layer, which can be represented by a Directed acyclic graph.

Computing neighbor triplet (u,r,v): In graph-based models, neighbor triplets (u,r,v) can be computed by iterating over each node u in the graph, and for each node u, iterating over its neighbors v and the edges r that connect them. For example, in a graph neural network, the neighbor triplet (u,r,v) can be computed by aggregating the embeddings of u, r, and v using a set of learnable parameters. In a knowledge graph embedding model, the neighbor triplet (u,r,v) can be computed by projecting the embeddings of u and v onto a shared space, and then computing a score based on the similarity between the projected embeddings.

Computing attention score: In attention-based models, attention scores can be computed by computing a similarity function between a query vector and a set of key vectors. The query vector can be a hidden state in a recurrent neural network or the output of a previous layer in a feedforward neural network. The key vectors can be a set of embeddings representing a sequence of inputs, such as words in a sentence or items in a list. The similarity function can be a dot product or a learned function such as a multi-layer perceptron. The resulting scores can be normalized using a softmax function and used to compute a weighted sum of the value vectors, which can be used as the output of the attention mechanism.

Based on the modeling of question and query subgraph in the previous paper, this paper splices the question representation, query entity representation and candidate node representation, and uses feedforward neural network and sigmoid function to calculate the score in formula (9). s q , i = sigmoid ( FF 4 ( e q ; e H ; h i T ) )(9)

The sigmoid function is a mathematical function that maps any input value to an output value between 0 and 1. It is commonly used in machine learning and neural networks as an activation function, which introduces non-linearity into the output of a neural network node.

When predicting the answer, select the entity with the highest score in the subgraph as the answer according to the above calculation. Finally, the overall loss function of the model is defined by binary cross entropy in formula (10). Loss = - ∑ q , i y q , i ln s q , i + ( 1 - y q , i ) ln ( 1 - s q , i )(10)

Question representation, query entity representation and candidate node representation are spliced, and Feedforward neural network and sigmoid function are used. s_q,i is their calculated score. 0 < y_q,i < 1 is weighted. Loss is the sum of each weighted entropy. Weighted entropy is a Loss function, which is often used in multi classification problems. Compared with the Cross entropy Loss function, weighted entropy considers the relationship between categories, and can better deal with the situation of category imbalance. Specifically, weighted entropy assigns higher weights to a few categories, making the model more focused on these categories, thereby improving the model’s classification performance for minority categories.

To obtain the answer subgraph of a question answering model based on question perception graph convolution, these steps are can followed:

•Construct the question perception graph by using the question as the input and applying graph convolutional neural network (GCN) to encode the information in the graph.

•Extract the relevant nodes from the question perception graph that are likely to contain the answer to the question. This can be done by using a graph attention mechanism or other techniques to identify the most salient nodes in the graph.

•Once the relevant nodes have been identified, extract the subgraph that contains these nodes and their edges.

•Apply additional graph convolutional layers to the subgraph to refine the representations of the nodes and edges based on the context of the question.

•Use the refined subgraph representations to predict the answer to the question, either by selecting a node in the subgraph or by generating a new node and its associated edges as the answer.

This approach combines the power of graph convolutional neural networks with attention mechanisms to extract the most relevant information from a knowledge graph or other data source, and then uses this information to generate an answer to the question.

3.5 The components needed for building the graph neural network

There are several components that are typically needed for building a graph neural network (GNN). These include:

•Graph data: GNNs require input in the form of a graph, which is typically represented as a set of nodes and edges with associated features.

•Node features: Each node in the graph may have associated features, such as node labels, attributes, or metadata. These features can be used to inform the node representation and update process.

•Edge features: Each edge in the graph may have associated features, such as edge weights or types. These features can be used to inform the edge representation and update process.

•Message passing: GNNs typically use a message passing mechanism to propagate information between nodes in the graph. This involves passing messages along the edges of the graph and updating the node representations based on the received messages.

•Aggregation: After receiving messages from their neighbors, nodes typically perform an aggregation step to combine the received information into a new node representation.

•Non-linear transformation: GNNs may use non-linear transformations, such as activation functions or multi-layer perceptrons, to update the node representations.

•Pooling: GNNs may use pooling or downsampling mechanisms to reduce the size of the graph and enable processing of larger graphs.

The particular design choices behind these components can vary depending on the specific GNN architecture and application. For example, different message passing mechanisms, aggregation functions, and non-linear transformations may be used depending on the task at hand and the structure of the input graph. Additionally, different pooling strategies may be used to enable processing of larger graphs or to capture different aspects of the graph structure.

4.1 Experimental data set

The experimental dataset uses the large-scale online education knowledge base MOOCCube constructed by Tsinghua University [3], and the MOOC Q&A dataset is used for the question and answer experiment of the knowledge base. The knowledge base contains 706 real online courses, 700 concepts, and 4723 users, as well as 10 types of relationships such as concept domain, course video, and concept sequence, forming a total of 52195 triples. The detailed statistical information of the knowledge base is shown in Table 1. MOOC Q&A contains two kinds of problems: one-hop and multi-hop. The one-hop question involves only a head entity and a relationship in the knowledge base, while the multi-hop question may contain multiple entities, and to answer such questions, we need to reason from multiple facts in the knowledge base. The number of one-hop and multi-hop problems is 5504 and 13637, respectively. In the experiment, they are divided into training set, verification set, and test set according to the proportion of 80%, 10%, and 10%.

In the experiment, this paper uses the “Jieba” word segmentation tool to segment the question, and uses the DistMult method to embed the entities in the knowledge base. The default embedding dimension is 200. For the one-hop and multi-hop problems, the order of the adjacent subgraph Gq extraction time is set to 1 and 2, respectively. Similarly, the number of layers of the graph convolution network is set to 1 and 2, respectively. The word embedding dimension, the transformer hidden layer dimension, and the graph convolution network dimension are all set to 200 by default. Each feed-forward network layer is activated by the ReLU function, and a dropout layer with a drop rate of 0.1 is added to enhance the generalization performance of the model. During training, this paper uses the Adam algorithm with a learning rate of 0.002 to optimize and sets the maximum number of iterations to 30 [28].

4.3 Comparison model and evaluation index

In order to verify the effectiveness of the model proposed in this paper, a comparison experiment is conducted using EmbedKGQA, the current optimal model. In this model, the question is treated as a one-hop or multi-hop reasoning process of the relationships in the knowledge base, thus making the knowledge base question and answer analogous to the link prediction task. The score of the candidate entity is calculated through the score function of the link prediction. In the experiment, TransE [29], DistMult [26], ComplEx [24], and RotatE [30] were used as the score functions to predict the answers. The prediction accuracy (ACC) and mean reciprocal rank (MRR) are selected as performance evaluation indicators. ACC measures whether the entity with the highest prediction score appears in the actual answer list. MRR indicates the average reciprocal ranking of the entities in the answer set in the prediction list. Their calculation methods are given in formulas (11) and (12). ACC = 1 M ∑ q I ( p , ɛ ɛ T )(11) MRR = 1 M ∑ q ( 1 | ɛ ɛ T | ∑ i ∈ ɛ ɛ T 1 r i )(12)

Where, M represents the number of questions; ɛ _T is the actual answer set of question q; p is the entity with the highest score of model prediction; The function I indicates whether the element is in the collection, and its value range is {0,1}; r _i indicates the ranking of entity i in the forecast answer list.

The DistMult method is a knowledge graph embedding model that represents entities and relations as low-dimensional vectors in a continuous space, and it is capable of handling multi-hop questions in a natural way.

When presented with a multi-hop question, the model first identifies the query entities and relations in the question, and represents them as low-dimensional vectors. It then performs a series of graph traversals by computing the scores of all possible entity-pairings using the DistMult scoring function, and selecting the most promising entities that are likely to be connected by the required relation. This process is repeated for each hop in the question until the final answer is obtained.

This approach is effective because the DistMult model is able to capture the complex relationships between entities and relations in a knowledge graph, and use this information to reason about the answer to a multi-hop question. By representing entities and relations as low-dimensional vectors, the model is able to perform efficient graph traversals and identify the most promising paths between entities.

In comparative experiments on multi-hop question answering datasets, the DistMult method has been shown to achieve state-of-the-art performance, demonstrating its effectiveness in handling multi-hop questions.