Advancing document-level relation extraction with a syntax-enhanced multi-hop reasoning network

Abstract

Document-level relation extraction aims to uncover relations between entities by harnessing the intricate information spread throughout a document. Previous research involved constructing discrete syntactic matrices to capture syntactic relationships within documents. However, these methods are significantly influenced by dependency parsing errors, leaving much of the latent syntactic information untapped. Moreover, prior research has mainly focused on modeling two-hop reasoning between entity pairs, which has limited applicability in scenarios requiring multi-hop reasoning. To tackle these challenges, a syntax-enhanced multi-hop reasoning network (SEMHRN) is proposed. Specifically, the approach begins by using a dependency probability matrix that incorporates richer grammatical information instead of a sparse syntactic parsing matrix to build the syntactic graph. This effectively reduces syntactic parsing errors and enhances the model’s robustness. To fully leverage dependency information, dependency-type-aware attention is introduced to refine edge weights based on connecting edge types. Additionally, a part-of-speech prediction task is included to regularize word embeddings. Unrelated entity pairs can disrupt the model’s focus, reducing its efficiency. To concentrate the model’s attention on related entity pairs, these related pairs are extracted, and a multi-hop reasoning graph attention network is employed to capture the multi-hop dependencies among them. Experimental results on three public document-level relation extraction datasets validate that SEMHRN achieves a competitive F1 score compared to the current state-of-the-art methods.

Keywords

Attention mechanism document-level relation extraction syntactic information multi-hop reasoning

1 Introduction

Relation extraction is a technique that automatically analyzes and processes textual information to extract structured entity relationship information from unstructured natural language text. By automatically analyzing large-scale text, it extracts relationship information, providing more intelligent and accurate information processing and decision support for various application domains such as artificial intelligence applications such as information retrieval, intelligent question answering, and conversational agents [1, 2]. Additionally, in scientific research, relation extraction holds significant value for integrating and analyzing extensive literature, driving scientific discoveries. This contributes to advancing artificial intelligence technology and promoting a deeper understanding of natural language. By addressing this challenge, it not only propels the development of the field of natural language processing but also lays the groundwork for constructing more intelligent technologies that better meet human needs. Early efforts [3, 4] in relation extraction primarily focused on the sentence-level. However, in practice, relations between entities may require expressions spanning multiple sentences. Therefore, researchers [5–7] have delved into in-depth studies of document-level relation extraction.

Identifying relationships between entities throughout the entire document requires the model to possess strong reasoning abilities. Figure 1 illustrates an example taken from the DocRED [8] dataset, which focuses on document-level relation extraction. Determining the relationship between the entities “Washington County” and “Oregon state” is challenging. It requires multi-hop reasoning. Specifically, the relation between the entities “Washington County” and the “United States” needs to be deduced first, along with the relation between the entities “Oregon state” and the “United States”. Only then can the relation between the entities “Washington County” and “Oregon state” be inferred. In this case, the entities “Portland Golf Club” and the “United States” serve as intermediary entities, establishing a logical path connecting “Washington County” to “Oregon state”. By scrutinizing the reasoning process depicted in Fig. 1, it is evident that document-level relation extraction involves handling numerous entities, a variety of relationship types, and complex contextual information.

Fig. 1

An example from the DocRED dataset includes 6 entities, 7 statements, and 10 relations. Different colors represent various entities. The first sentence is parsed, and the part-of-speech tagging of each word, along with the dependency type of important words, is presented. The process of reasoning about the relation between the subject “Washington County” and the object “Oregon” is visualized.

To address the aforementioned challenges, earlier models [9–11] of document-level relation extraction utilized neural architectures such as convolutional neural networks (CNN) [12] and recurrent neural networks (RNN) [13] to perceive contextual information. These approaches face limitations related to the encoder’s contextual modeling capability, making it difficult for them to effectively capture global dependencies. Recently, graph neural networks (GNN) [14, 15] have been widely applied in the field of document-level relation extraction due to their capability to capture long-distance contextual dependencies. These methods primarily utilize heuristic rules [16–18], discourse relations [19], syntactic dependency information [20], or structural attention [21] to construct a document graph. Subsequently, GNNs are employed to perform multi-hop reasoning.

Nonetheless, current GNN-based models exhibit three evident limitations: i) Although syntactic dependency information contributes to establishing semantic connections between words within a sentence, previous methods overly relied on the final discrete output of syntactic parsers. This excessive reliance led to the accumulation of parsing noise in document-level relation extraction systems, undermining the performance of the model. ii) In previous approaches, syntactic dependency information, such as dependency edges and part-of-speech information, has not been fully utilized. These additional syntactic dependency details can also enhance the identification of relationships. As depicted in Fig. 1, when judging the relation between entities “United States” and “Portland” as “country", one can promptly infer that the relation between entities “United States” and “Oregon” is also “country”. This inference is based on the appositive dependency relation between the entities “Portland” and “Oregon”. iii) The previous methods did not explicitly model multi-hop reasoning between entity pairs, resulting in inefficient multi-hop reasoning between entity pairs. In their approach, when inferring two-hop relations, each entity pair needs to consider 2 (n - 1) entity pairs that overlap with them. However, when inferring three-hop relations, each entity pair needs to consider 2² (n - 1) ² entity pairs. When inferring n-hop relations, each entity pair needs to identify entity pairs with dependencies from 2^(n-1) (n - 1) ^(n-1) possible entity pairs, and most of these entity pairs do not contribute to identifying the relations of the target entity pair. Therefore, their method primarily operates effectively in two-hop relations and faces challenges in modeling multi-hop relations.

In this paper, a syntax-enhanced multi-hop reasoning network (SEMHRN) is proposed to address the aforementioned limitations. To overcome the first limitation, the probability matrix of all dependency arcs from a dependency parser is employed to construct a syntax graph. This approach is motivated by the fact that the probability matrix, representing relations between words’ dependencies, has more extensive syntactic information compared to the dependency parser’s final discrete output. To address the second limitation, a novel dependency-type-aware attention mechanism is designed to learn the interaction between words of syntactic dependency edges with different labels. Additionally, a part-of-speech prediction task is introduced to regularize word representation. The third limitation is tackled through a novel two-stage design. In the first stage, only related entity pairs are extracted. In the second stage, multi-hop dependency relations between these related entity pairs are captured, thereby mitigating interference from noisy entity pairs.

In summary, the contributions can be summarized as follows:

A syntax-enhanced multi-hop reasoning network (SEMHRN) is proposed. The network constructs a syntactic graph using the probability matrix of all dependency arcs, providing a fine-grained modeling of information in the document and effectively alleviating interference from syntactic parsing errors.

A novel dependency-type-aware attention is proposed to guide attention in learning interactive patterns of different types of dependency edges and a part-of-speech prediction task is introduced to regularize word representations. This fully leverages syntactic information, enhancing the model’s performance.

A two-stage relation modeling method is designed. By extracting related entity pairs in the first stage and modeling multi-hop interactions between these pairs in the second stage, the model’s ability to capture long-range dependencies is improved.

The experimental results demonstrate that SEMHRN outperforms compared methods by a noticeable margin. Additionally, an extensive series of experiments is carried out to analyze and discuss each critical component of SEMHRN. The experimental results show the remarkable ability of SEMHRN to capture multi-hop dependencies.

2 Related work

2.1 Sentence-level relation extraction

The purpose of relation extraction is to identify connections between entities within a given text. In the early stages, many efforts [3, 4] to identify entity relationships within a single sentence relied on statistical machine learning, where performance was significantly constrained by the quality of manually extracted features, inevitably leading to error propagation within the model. With the rise of neural networks, an increasing number of researchers have turned to using neural networks for automatic feature extraction. For example, Wang et al. [22] proposed a multi-level attention CNN that leverages entity-specific attention at the input layer and relationship-specific pooling attention at the pooling layer to identify patterns within heterogeneous contexts.

However, many relations between entities require multiple sentences to be fully expressed. Therefore, in recent years, there has been a growing emphasis on document-level relation extraction [5, 7].

2.2 Document-level relation extraction

Methods for document-level relation extraction mainly fall into sequence-based methods and graph-based methods. To emphasize the distinctions and connections among these methods, the key techniques employed by them are summarized, as shown in Table 1.

Table 1
Key techniques of existing document-level relation extraction methods

Model Encoder Key technique

Sequence-based methods Coref [16] BERT/RoBERTa Co-reference prediction sub-task

ATLOP [16] BERT/RoBERTa Localized Context

MRN [27] LSTM/BERT Separating intra and inter-sentence reasoning

Docunet [28] BERT/RoBERTa Semantic segmentation

AFLKD [29] BERT/RoBERTa Axial attention

DREEAM [30] BERT/RoBERTa Evidence information

HIN [31] LSTM Local to global information aggregation

SSAN [32] BERT/RoBERTa Structure reasoning

E2GRE [33] BERT/RoBERTa Evidence prediction sub-task

Correl [34] BERT/RoBERTa/DeBERTa Co-occurrence prediction sub-task

ERA [35] BERT Easy Relation Augmentation

ERACL [35] BERT Contrastive Learning

CPT [36] BERT Semantic segmentation

Graph-based methods GAIN [16] LSTM/BERT Structural dependency information

HDT-BP [17] LSTM/BERT Hierarchical dependency tree

DISCO [19] LSTM/BERT Discourse information

LSR [21] LSTM/BERT Optimal graph structure

EoG [38] LSTM Structural dependency information

AGGCN [39] LSTM Syntactic dependency tree and attention guide

SIRE [41] LSTM/BERT Separating intra and inter-sentence reasoning

CGM2IR [42] BERT/RoBERTa Context-guided attention

GRACR [43] LSTM Structural dependency information

Ours SEMHRN BERT/RoBERTa Syntactic probability matrixDependency-type-aware attentionPart-of-speech prediction sub-taskTwo-stage multi-hop reasoning

	Model	Encoder	Key technique
Sequence-based methods	Coref [16]	BERT/RoBERTa	Co-reference prediction sub-task
	ATLOP [16]	BERT/RoBERTa	Localized Context
	MRN [27]	LSTM/BERT	Separating intra and inter-sentence reasoning
	Docunet [28]	BERT/RoBERTa	Semantic segmentation
	AFLKD [29]	BERT/RoBERTa	Axial attention
	DREEAM [30]	BERT/RoBERTa	Evidence information
	HIN [31]	LSTM	Local to global information aggregation
	SSAN [32]	BERT/RoBERTa	Structure reasoning
	E2GRE [33]	BERT/RoBERTa	Evidence prediction sub-task
	Correl [34]	BERT/RoBERTa/DeBERTa	Co-occurrence prediction sub-task
	ERA [35]	BERT	Easy Relation Augmentation
	ERACL [35]	BERT	Contrastive Learning
	CPT [36]	BERT	Semantic segmentation
Graph-based methods	GAIN [16]	LSTM/BERT	Structural dependency information
	HDT-BP [17]	LSTM/BERT	Hierarchical dependency tree
	DISCO [19]	LSTM/BERT	Discourse information
	LSR [21]	LSTM/BERT	Optimal graph structure
	EoG [38]	LSTM	Structural dependency information
	AGGCN [39]	LSTM	Syntactic dependency tree and attention guide
	SIRE [41]	LSTM/BERT	Separating intra and inter-sentence reasoning
	CGM2IR [42]	BERT/RoBERTa	Context-guided attention
	GRACR [43]	LSTM	Structural dependency information
Ours	SEMHRN	BERT/RoBERTa	Syntactic probability matrixDependency-type-aware attentionPart-of-speech prediction sub-taskTwo-stage multi-hop reasoning

2.2.1 Sequence-based methods

Sequence-based methods in the realm of relation extraction often rely on neural structures or pretrained models for encoding entire documents. Wang et al. [23] divided document-level relation extraction into two distinct stages: in the first stage, they determine whether a relationship between entity pairs exists, and in the second stage, they infer specific relationships between the entity pairs. To enhance the model’s co-referential reasoning ability, Ye et al. [24] introduced mention reference prediction and masked language modeling tasks, explicitly aiming to learn co-referential information. Eberts et al. [25] presented a joint model that integrates multi-task and multi-instance learning strategies to capture essential information from comprehensive textual data. Zhou et al. [16] suggested a localized context pooling technique for addressing multi-entity issues and an adaptive thresholding technique for handling multi-label problems. The adaptive thresholding employs a specialized class, “TH”, as the adaptive threshold value for each entity pair, while the localized context pooling incorporates multi-head attention to gather critical information for each entity pair. Li et al. [27] posits that local and global reasoning patterns play different roles in relation extraction, proposing a novel mention-based reasoning module to learn local and global document features. Zhang et al. [28] likened the relation extraction task to semantic segmentation, employing a U-shaped segmentation module to capture global information between entity pairs. Tan et al. [29] leveraged axial attention to strengthen interactions between two-hop entity pairs and used knowledge distillation to glean extensive information from a distantly supervised dataset. Ma et al. [30] employed evidence information as a supervisory signal, thereby guiding the attention module to allocate high weights to the evidence. Tang et al. [31] devised a hierarchical reasoning model that integrates entity-level, sentence-level, and document-level information. Xu et al. [32] introduced a Transformer-based SSAN, incorporating unique entity structural dependencies into the self-attention mechanism and employing two transformation modules for attention flow regulation. Huang et al. [33] believed that relation extraction and evidence prediction were highly correlated, so they proposed the E2GRE model to jointly learn both tasks. Han et al. [34] emphasized the guiding role of co-occurrence correlations of relations in aiding the classifier to identify semantically similar entity pairs. Two granularities of co-occurrence prediction tasks were introduced to capture relation-related information. Du et al. [35] increased tail relation entity pairs to tackle the long-tailed distribution issue and utilized a contrastive learning method to improve the model’s performance. Yuan et al. [36] employed a prompt tuning strategy to manage long texts and adopted a semantic segmentation method to capture entity interaction.∥While these sequence-based methods have achieved commendable performance, their inherent limitations become evident when it comes to capturing non-local syntactic information, which is indispensable for a more comprehensive and nuanced comprehension of the text.∥

2.2.2 Graph-based methods

Graph-based methods utilize semantic information within the document to construct document-level graphs and employ graph convolutional networks to capture non-local dependencies. Wang et al. [37] devised a global representation layer to simulate interactions among numerous hierarchical nodes and a local representation layer to aggregate multiple mentions pertaining to the same entity. Christopoulou et al. [38] introduced an edge-oriented graph neural model to capture inference information from various node and edge types. Guo [39] innovatively designed an AGGCN model, leveraging an attention-guided approach to capture valuable syntactic information while minimizing noise. Xu et al. [40] developed a reconstructor that utilizes the graph representation to reconstruct ground-truth path dependencies. Sahu et al. [20] introduced GCNN to learn both local and global dependencies, along with applying multiple-instance learning to aggregate multiple mention-level pairs. Nan et al. [21] employed structured attention to construct a document graph, which was then processed by a multi-layer graph convolutional network to capture underlying structural information. Zeng et al. [16] introduced a mention-level graph to model complex interactions among mentions and an entity-level graph to construct two-hop reasoning paths between entities. Zeng et al. [41] introduced a sentence-level encoder to collect intra-sentence dependencies and a mention-level graph to obtain inter-sentence structural information. Zeng et al. [42] designed a context-guided attention mechanism to dynamically aggregate content mentioning the same entity through weighted summation, as well as an inter-pair reasoning approach to model interactions between entity pairs. Wang et al. [19] utilized discourse information to construct a document-level graph for capturing semantic dependencies between text units. Liu et al. [43] developed a document-level graph aimed at capturing comprehensive global dependency information. Simultaneously, they established an entity-level graph to facilitate reasoning about inter-sentence entities. Wan [17] reconstructed the document by region and introduced bridge entities to construct a dependency structure, aiming to improve the efficiency of relation extraction.∥While the graph-based methods mentioned above have driven the development of document-level relation extraction, most of them have overlooked the modeling of multi-hop reasoning among entity pairs within the document. To tackle these issues, a syntax-enhanced multi-hop reasoning network is designed to capture remote dependencies between entity pairs. As shown in Table 1, this model incorporates four key techniques. Firstly, a syntactic graph is constructed using the probability matrix of dependency arcs outputted by the dependency parser, rather than the dependency tree, as the probability matrix of dependency arcs provides richer syntactic information. Secondly, a novel dependency-type-aware attention mechanism is designed to learn the interaction strength between words for different types of dependency edges. Thirdly, a part-of-speech prediction subtask is introduced to regularize word representations, motivated by the intuition that part-of-speech information aids in predicting relations between entities. Lastly, a two-stage multi-hop reasoning framework is proposed. In the first stage, entity pairs with relations are extracted, and in the second stage, multi-hop reasoning between these related entity pairs is modeled to identify specific relation types.

3 Method

3.1 Task formulation

Document-level relation extraction is a multi-classification task aimed at identifying relations between entities within the scope of a document.

Input: A predefined set of relations R and a document D = {x₁, x₂, . . . , x_n} consisting of n words that contains a set of entities E = {e₁, e₂, . . . , e_|E|}, where entities may appear multiple times in the form of mentions M_e = {m₁, m₂, . . . , m_{e|M_e|}}.

Predict: A set of related entity pairs and their specific relations $U = {< e_{s}, r, e_{o} >, e_{s}, e_{o} \in E, r \in ℝ}$ .

3.2 Model overview

In this section, the proposed model SEMHRN, incorporating syntax-enhanced and multi-hop reasoning to enhance document-level relation extraction, is described. As illustrated in Fig. 2, SEMHRN mainly consists of five parts, namely (i) the context encoder; (ii) the syntax-enhanced graph convolutional network; (iii) the multi-hop reasoning graph convolutional network; (iv) the part of speech prediction; and (v) the final classifer. The context encoder and the final classifier in these five components largely follow the settings of Zhou et al. [16], while the remaining parts introduce innovative approaches.

Fig. 2

The architecture of SEMHRN incorporates several key components to enhance document-level relation extraction.

3.2.1 Context encoder

The pre-training model performs well in various downstream tasks. For a document D, it can be encoded as:

$\begin{matrix} [h_{1}, h_{2}, . . ., h_{n}] = Encoder ([x_{1}, x_{2}, . . ., x_{n}]) \end{matrix}$ (1) where x_i is the i-th word of the text and h_i is the hidden representation of x_i.

Following the approach of Zhou et al. [16], the hidden representation of the first word within a mention is employed to represent the entire mention.

3.2.2 Syntax-enhanced Graph Convolutional Network

Previous methods employed syntax trees to construct document dependency graphs and fed them into GCN [14] to capture syntax information through multiple iterations. Typically, syntax parsers prioritize the dependency with the highest probability, often neglecting others. However, when the probabilities of two dependencies are similar, ignoring one can result in issues. This rough estimation contributes to syntax parsing errors, which, in turn, hinder model performance.

To address these challenges, the dependency probability matrix generated by an external parser is utilized to construct the document graph. Unlike the sparse dependency matrix, the dependency probability matrix defines the probability of dependency between words within a sentence, providing richer syntactic information and mitigating parsing errors. Furthermore, entity type information proves valuable in identifying relations between entities. Six entity type nodes, namely ORG-node, LOC-node, TIME-node, PER-node, MISC-node, and NUM-node, are introduced into the graph to establish semantic relations with entities. To acquire entity type knowledge, these six entity type nodes are set to be globally shared across all sentences in the dataset. In essence, entity type nodes begin with random initialization and refine their representations through interactions with various entity words.

Six unique types of edges are defined for constructing the document-level graph A. The detailed descriptions of these six types of edges are provided below:

Syntactic dependency edge: The sentence’s syntactic structure offers valuable hints for intrasentential relation extraction. Therefore, labeled syntactic dependency edges connecting the words within each sentence are utilized, considering each syntactic dependency label as a distinct edge type.

Mention-token edge: To capture semantic information related to entities, mention-token edges are established, connecting all tokens belonging to the same mention.

Document-mention edge: To learn global dependencies, document-mention edges are utilized to connect each mention node to the document node.

Mention-Mention edge: Two mentions of different entities are connected if they co-occur in a single sentence to capture co-occurrence dependencies.

Mention-type edge: Entity type information contributes to inferring relations between entities. Mention-type edges connect mention nodes with their corresponding type nodes to capture this information.

Self-node edge: GCN learns a node representation primarily based on its neighboring nodes and their edge types. To incorporate the node’s own information into the representation, self-node type edges are created for all the graph nodes.

The document-level graph construction process is illustrated in Fig. 3.

Fig. 3

Construction of the document-level graph. The document-level graph comprises four types of nodes, namely token nodes, mention nodes, entity type nodes, and document nodes. The document-level graph encompasses six types of edges, including Syntactic dependency edges, Mention-token edges, Document-mention edges, Mention-Mention edges, Mention-type edges, and Self-node edges (not shown in the graph).

Following that, GCN is employed on the document-level graph to capture global structural information. For a given node i, its representation in the l + 1-th layer is computed based on the representations of its neighboring nodes in the l-th layer. The graph convolutional operation can be expressed as follows:

$h_{i}^{(l + 1)} = ReLU (\sum_{j = 1}^{n} A_{ij} W^{(l)} h_{j}^{(l)} + b^{(l)})$ (2) where $W^{(l)} \in ℝ^{d \times d}$ and $b^{(l)} \in ℝ^{d}$ are trainable parameters.

To incorporate more dependency knowledge, dependency-type-aware attention is introduced in the multi-layer GCN. Consequently, Equation (2) undergoes a rewriting as follows: $h_{i}^{(l + 1)} = ReLU (\sum_{j = 1}^{n} α_{ij} A_{ij} W^{(l)} h_{j}^{(l)} + b^{(l)})$ (3)

The normalized attention coefficient between node i and node j is expressed as: $α_{ij} = softmax (W_{ij})$ (4)

Each layer of the GCN represents varying degrees of semantic features. To effectively utilize features from all levels, the hidden states from each layer are concatenated. Subsequently, this concatenated representation is fed through a linear layer to produce the ultimate hidden state of node i:

$h_{i} = ReLU (W_{i} \cdot [h_{i}^{(0)}; h_{i}^{(1)}; . . .; h_{i}^{(N)}])$ (5) where $W_{i} \in ℝ^{d \times Nd}$ is a trainable parameter matrix. N is the number of document-level GCN layers. $h_{i}^{(0)}$ is the initial node representation of node i, and its computation depends on whether node i is a token node, a mention node, a document node, or a entity type node. For a node corresponding to the j-th token in the i-th sentence of the document, its representation is $h_{i}^{(0)} = h_{i}^{j}$ . Mention nodes utilize the hidden representation of the first word within a mention to represent the entire mention. Document nodes use the embedding of the special token [CLS] as their representation. Entity type nodes have their embedding randomly initialized.

It’s important to note that a certain entity may be mentioned multiple times within the document. Effective relation extraction relies heavily on the ability to aggregate pertinent information from these multiple mentions. While the maximum pooling method is a viable option for retaining the most essential mention characteristics, it poses a challenge for optimization due to its non-differentiability. To address this challenge, the logsumexp function, a common operation encountered in machine learning, particularly in the implementation and derivation of cross-entropy, is utilized. Not only does it serve as a smooth approximation of maximum pooling, but it is also differentiable. The logsumexp function, with the introduction of a temperature coefficient T close to 0, is employed to obtain the entity embedding h_i. For an entity e_i, its entity embedding can be expressed as: $e_{i} = T \cdot \log \sum_{j = 1}^{| M_{e} |} \exp (\frac{m_{j}}{T})$ (6)

Finally, document-aware representations for tokens and entities are obtained. After GCN encoding, the node representations of tokens can learn contextual semantics and syntactic structure. The special entity type nodes will also learn general representations of entity types from the training examples in the dataset.

3.2.3 Related entity pair extraction

Previous methods often use the embedding of subject and object directly to predict the relation of entity pairs. This is problematic since different entity pairs should be concerned with different contextual information. Therefore, the average pooling is introduced to convert token-level dependencies learned by the pre-training model into entity-level dependencies, and this transfer process is expressed as: $A_{e_{i}} = \frac{\sum_{j = 1}^{N_{e_{i}}} A_{m_{j} [*]}}{N_{e_{i}}}$ (7) where $A_{m_{j} [*]} \in ℝ^{H \times l}$ is the token-level attention at the start position of mention m_j, H is the number of attention heads, and l is the document length.

For an entity pair (e_s, e_o), context representation c_(s,o) is computed based on well-learned dependencies from the pre-training language model: $c_{(s, o)} = H^{T} \frac{A_{s} \cdot A_{o}}{1^{T} (A_{s} \cdot A_{o})}$ (8) where A_s is attention from the entity e_s to all tokens in the document. Similar for A_o.

The embeddings of entities e_s and e_o are combined with their contextual embeddings:

$z_{h}^{(h, t)} = tahn (W_{h} \cdot e_{h} + W_{c_{1}} \cdot c_{h, t})$ (9)

$z_{t}^{(h, t)} = tahn (W_{t} \cdot e_{t} + W_{c_{2}} \cdot c_{h, t})$ (10) where $W_{h} \in ℝ^{d \times d}$ , $W_{t} \in ℝ^{d \times d}$ , $W_{c_{1}} \in ℝ^{d \times d}$ and $W_{c_{2}} \in ℝ^{d \times d}$ are the weight matrix.

A group bilinear is employed to reduce the number of parameters in the bilinear classifier. To be specific, z_h and z_t will be split into k equal-sized groups:

$z_{h} = [z_{h}^{1}; . . .; z_{h}^{k}]$ (11)

$z_{t} = [z_{t}^{1}; . . .; z_{t}^{k}]$ (12)

The bilinear function is applied to obtain the representation of the entity pair (e_h, e_t).

$p_{(h, t)} = \sum_{i = 1}^{k} (z_{h}^{i T} W_{r}^{i} z_{t}^{i}) + b_{r}$ (13) where $W_{r}^{i} \in ℝ^{d / k \times d / k}$ is the weight matrix and b_r is the bias term.

Then, the probability of entity pairs having a relation can be calculated, formally as follows:

$P (o | e_{h}, e_{t}) = W_{o} p_{(h, t)} + b_{o}$ (14) where $W_{o} \in ℝ^{d \times 2}$ is the weight matrix and P (o|e_h, e_t) denotes the probability that head entity e_h is related to tail entity e_t.

The cross-entropy is employed to calculate loss L_EPE as follows:

$L_{EPE} = - \sum_{e_{i}, e_{j} \in E} (y_{ij} logP (o | e_{i}, e_{j}))$ (15) where y_ij = 1 if e_i and e_j have a relation, and 0 otherwise. E is the set of entities.

3.2.4 Multi-hop reasoning graph attention network

Connecting related entity pairs extracted in the first stage constructs an entity-level graph. Subsequently, a graph attention network (GAT) [15] is employed to learn entity interaction information.

For an entity node e_i, the calculation of its representation in the l + 1-th layer is defined as follows:

$h_{i}^{l + 1} = σ (\sum_{j \in N_{i}} α_{ij} W_{h} e_{j}^{l})$ (16)

$α_{ij} = \frac{\exp (LeakyReLU (a^{T} [W_{e} e_{i}^{l}; W_{e} e_{j}^{l}])}{\sum_{j^{'} \in N_{i}} \exp (LeakyReLU (a^{T} [W_{e} e_{i}^{l}; W_{e} e_{j^{'}}^{l}])}$ (17) where $W_{l} \in ℝ^{d \times d}$ and $W_{e} \in ℝ^{d \times d}$ are weight matrices, a represents the weight vector, and $N_{i}$ is the set of neighboring nodes of node i. The LeakyReLU is used as an activation function, and α_ij denotes the normalized attention coefficient between node i and node j.

Following this, each entity pair is treated as a node, and a GAT is introduced to capture multi-hop dependencies between entity pairs. Diverging from prior methods that account for two-hop dependencies for each entity pair, we aggregate dependency information exclusively for the extracted related entity pairs. The connection is established between these related entity pairs and other entity pairs that share the same entities with them. This is guided by the intuition that reasoning cues often propagate along entity chains. Based on equations (9) - (13), the calculation of the entity pair representation p_u for node u is feasible. Its representation in the (l + 1)-th layer is defined as follows:

$p_{u}^{l + 1} = σ (\sum_{v \in N_{u}} α_{uv} W_{l} p_{v}^{l})$ (18)

$α_{uv} = \frac{\exp (LeakyReLU (a^{T} [W_{p} p_{u}^{l}; W_{p} p_{v}^{l}])}{\sum_{v^{'} \in N_{u}} \exp (LeakyReLU (a^{T} [W_{p} p_{u}^{l}; W_{p} p_{v^{'}}^{l}])}$ (19) where $W_{l} \in ℝ^{d \times d}$ and $W_{p} \in ℝ^{d \times d}$ are weight matrices, a represents the weight vector. $N_{u}$ is the set of neighborhood nodes of node u. LeakyReLU is an activation function. α_uv is the normalized attention coefficient between node u and node v.

The visualization of the multi-hop reasoning process can be seen in Fig. 4. The target entity pairs learn two-hop reasoning information from their horizontally and vertically related entity pairs. As more layers of graph networks are stacked, the target entity pairs can capture multi-hop reasoning information.

Fig. 4

The visualization depicts the multi-hop reasoning process, showcasing that with an increase in the number of GAT layers, the SEMHRN model can effectively capture relations between multi-hop entity pairs.

3.2.5 part of speech prediction

In addition to syntactic dependencies, the part of speech of words in the document also contains valuable information that defines the usage and function of words. Part-of-speech tagging plays a crucial role in relation extraction for several reasons:

Disambiguation: Different usages of certain words can represent distinct meanings, and part-of-speech tagging helps disambiguate them.

Strengthening word-based features: While machine learning models can extract information from various aspects of a word, adding part-of-speech information can enhance the accuracy of these features.

Numerous studies have demonstrated that incorporating auxiliary tasks in relation extraction can enhance the performance of the primary task. To enhance the utilization of syntactic knowledge, an auxiliary task for word part-of-speech classification is introduced. Initially, a part-of-speech tag set T is created based on 36 predefined part-of-speech tags provided by the Natural Language Toolkit. Subsequently, the probability of word x_i belonging to a particular part of speech is calculated by passing h_i through a linear layer:

$P_{x_{i}}^{POS} = softmax (W_{POS} \tanh (h_{i}) + b_{POS})$ (20) followed by the multi-class cross-entropy loss: $L_{POS} = - \sum_{x_{i} \in D} \sum_{t \in T} y_{(x_{i}, t)}^{POS} {logP}_{x_{i}}^{POS}$ (21) where $W_{POS} \in ℝ^{| T | \times H}$ and $b_{POS} \in ℝ^{| T |}$ . $y_{(x_{i}, t)}^{POS} = 1$ if the part-of-speech of the word x_i is t, and 0 otherwise.

In this way, the word embeddings can be regularized by part of speech information.

3.2.6 Classification layer

Then, the probability of the entity pair (h, t) having a relation r can be calculated, formally as follows:

$P (r | e_{h}, e_{t}) = W_{o} p_{(h, t)}^{'} + b_{o}$ (22) where $W_{o} \in ℝ^{d \times r}$ is the weight matrix and P (r|e_h, e_t) denotes the probability that head entity e_h and tail entity e_t have relation r.

In a document, there can exist multiple relations between entities. Previous approaches involved setting a global threshold to determine the presence of relations, resulting in numerous decision errors due to variations in confidence levels among different entity pairs. To address this multi-label problem, Zhou et al. [16] introduced a “TH” class to establish a dynamic threshold for each entity pair. Following their approach, the loss function is partitioned into two categories to compute the loss values separately: $L_{1} = - \sum_{r \in ℙ_{T}} \log (\frac{\exp ({logit}_{r})}{\sum_{r^{'} \in ℙ_{T} ⋃ {TH}} \exp ({logit}_{r^{'}})})$ (23)

$L_{2} = - \log (\frac{\exp ({logit}_{TH})}{\sum_{r^{'} \in ℕ_{T} ⋃ {TH}} \exp ({logit}_{r^{'}})})$ (24)

$L_{RE} = L_{1} + L_{2}$ (25)

Finally, the integration of the relation extraction task and part-of-speech prediction task is achieved by minimizing the multi-task learning objective: $L = L_{RE} + λ_{POS} L_{POS} + λ_{EPE} L_{EPE}$ (26) where λ_POS and λ_EPE refer to the hyperparameters that balance the task weight.

4 Experiments and analysis

4.1 Dataset

The performance of SEMHRN model is assessed using three public document-level relation extraction datasets.

DocRED [8] is a widely utilized dataset in document-level relation extraction tasks derived from Wikipedia and Wikidata. The dataset comprises 97 relation types and includes 3,053 training documents. Within this dataset, approximately 26.6% of relation instances demand logical reasoning, 17.6% necessitate coreference reasoning for identification, and 16.6% involve commonsense reasoning.

CDR [44] is a biomedical dataset comprising 500 training documents, with an average of 7.6 entities per document. The task is to infer the interactions between chemical and disease concepts.

GDA [45] is a large-scale dataset in the biomedical domain. The dataset comprises 29,192 training documents. The task is to predict the interactions between disease concepts and genes.

4.2 Implementation details

Context encoder on DocRED is implemented using BERT-base [46] or RoBERTa-large [47], and on CDR and GDA, it is implemented using SciBERT [48]. The dimension of the dependency edge embedding is configured to be 100. The number of GCN layers is set to 2 and The number of multi-hop reasoning GAT layers is set to 3. Training the network involves 30 epochs with early stopping, and the learning rate is set to 3e-5. Other hyper-parameters including batch size is set to 4, dropout rate [49] between layers is set to 0.1, λ_POS is set to 0.3 and λ_EPE is set to 0.1.

4.3 Baselines methods

Twenty models are employed as benchmarks, categorized into sequence-based methods and graph-based methods. Their specific descriptions are as follows:

Sequence-based methods:

BERT-RE: Wang et al. [23] fine-tuned the BERT-base-uncased model specifically for relation extraction.

RoBERTa-RE: Ye et al. [16] simply fine-tuned RoBERTa-large for relation extraction.

Coref: Ye et al. [16] proposed Coref, which introduces two training tasks, Mention Reference Prediction and Masked Language Modeling, to enhance the model’s ability to capture co-referential information.

ATLOP: Zhou et al. [16] proposed ATLOP, which handles the multi-entity problem by aggregating relevant context information and addresses the multi-label problem by replacing the global threshold with an adaptive threshold.

MRN: Li et al. [27] proposed MRN, which employs a mention-based reasoning framework that jointly learns the local and global representations of mentions.

Docunet: Zhang et al. [28] proposed Docunet, which extends ATLOP by incorporating a U-shaped Segmentation Module to enhance the interaction between entity pairs. In comparison to ATLOP, Docunet places more emphasis on utilizing a reasoning process to identify inter-sentence relations.

AFLKD: Tan et al. [29] proposed AFLKD, which utilizes an axial attention mechanism to capture two-hop relations between entity pairs and employs an adaptive focal loss to tackle the class imbalance issue.

Correl: Han et al. [34] proposed Correl, which introduces two granularities of co-occurrence prediction tasks to learn relation-related information.

ERACL: Du et al. [35] proposed ERACL, which utilizes a contrastive learning method to enhance the model’s ability to address long-tail issues.

CPT: Yuan et al. [36] proposed CPT, which employs a prompt tuning strategy to process long texts and adopts a semantic segmentation method to capture entity interaction.

DREEAM: Ma et al. [30] proposed DREEAM, which integrates evidence information into document-level relation extraction systems by directly guiding the attention mechanism.

Graph-based methods:

GCNN: Sahu et al. [20] proposed GCNN, which learns both local and global dependencies, and utilizes multiple-instance learning to aggregate mention-level pairs.

AGGCN: Ma et al. [39] proposed AGGCN, which introduces an attention-guided approach to capture valuable syntactic information while minimizing noise.

GLRE: Wang et al. [37] proposed GLRE, which constitutes a multi-layer reasoning framework encompassing both a global representation layer and a local representation layer. The global representation layer employs a multi-layer stacked R-GCN to encode the semantic information of the entire document, facilitating reasoning within the global heterogeneous graph. In contrast, the local representation layer combines multiple mention representations using multi-head attention.

GAIN: Zeng et al. [16] proposed GAIN, which constructs two graphs, namely the mention-level graph and the entity-level graph to capture context dependencies, and employs path attention to aggregate reasoning information.

LSR: Nan et al. [21] proposed LSR, which introduces structured attention to induce latent dependency structures and devises an iterative refinement process to generate a more informative structure.

SIRE: Zeng et al. [41] proposed SIRE, which utilizes graph convolutional networks for reasoning and presents two distinct reasoning architectures, namely the intra-sentential and the inter-sentential relation representation module, for generating relation embeddings.

GRACR: Liu et al. [43] proposed GRACR, which develops a document-level graph to capture comprehensive global dependency information and establishes an entity-level graph to facilitate reasoning about inter-sentence entities.

DISCO: Wang et al. [19] proposed DISCO, which utilizes discourse information to construct a document-level graph for capturing semantic dependencies between text units.

HDT-BP: Wang et al. [17] proposed DISCO, which reconstructs the document by region and introduces bridge entities to construct a dependency structure.

CGM2IR: Zeng et al. [42] proposed CGM2IR, which introduces a context-guided attention mechanism for dynamically aggregating content that mentions the same entity through weighted summation. Additionally, it employs an inter-pair reasoning approach to simulate interactions between entity pairs.

4.4 Results on DocRED

Following the standard setup for document-level relation extraction tasks, the performance of these methods is evaluated using F1 and IgnF1 metrics. IgnF1 specifically calculates the F1 score after excluding relation instances mentioned in the development and test sets, providing a measure of the model’s ability to recognize relation instances that were previously unseen.

The experimental results on the DocRED dataset are presented in Table 2, indicating that SEMHRN achieves competitive performance compared to existing baseline methods. In particular, SEMHRN-RoBERTa achieves an IgnF1 of 63.01 and an F1 of 64.97 on the test set of DocRED, surpassing strong baseline models. For example, it outperforms the sequence-based ATLOP-RoBERTa by 1.57 F1 and 1.62 IgnF1, as well as the graph-based CGM2IR-RoBERTa by 1.08 F1 and 1.05 IgnF1. Furthermore, SEMHRN-BERT achieves an F1 of 62.51 and an IgnF1 of 60.53, outperforming the sequence-based Docunet-BERT by 0.6 F1 and 0.65 IgnF1, and the graph-based LSR-BERT by 3.46 F1 and 3.56 IgnF1. These results suggest that SEMHRN is effective in capturing syntactic dependency information and conducting efficient multi-hop reasoning.

Table 2
Main results (%) on DocRED. F1 and IgnF1 are employed as evaluation metrics. IgnF1 is calculated by excluding relation instances mentioned in the development and test sets. Bold values indicate the best performance

Dev Test

Model Ign F1 F1 IgnF1 F1

Sequence-based Methods BERT-RE [23] (2019) – 54.16 – 53.20

Coref-BERT [16] (2020) 55.32 57.51 54.54 56.96

ATLOP-BERT [16] (2021) 59.22 61.09 59.31 61.30

MRN-BERT [27] (2021) 59.74 61.61 59.52 61.74

Docunet-BERT [28] (2021) 59.86 61.83 59.93 61.86

AFLKD-BERT [29] (2022) 60.08 62.03 60.04 62.08

Correl-BERT [34] (2023) 59.39 61.34 59.12 61.32

ERACL-BERT [35] (2022) 59.72 61.80 59.08 61.36

CPT-BERT [36] (2023) 60.02 62.13 59.92 61.87

DREEAM-BERT [30] (2023) 60.51 62.55 60.03 62.49

RoBERTa-RE [16] (2020) 57.14 59.22 57.51 59.62

Coref-RoBERTa [16] (2020) 57.35 59.43 57.90 60.25

ATLOP-RoBERTa [16] (2021) 61.32 63.18 61.39 63.40

Docunet-RoBERTa [28] (2021) 62.23 64.12 62.39 64.55

AFLKD-RoBERTa [29] (2022) 62.16 64.19 62.57 64.28

DREEAM-RoBERTa [30] (2023) 62.29 64.20 62.12 64.27

Graph-based Methods GCNN [20] (2019) 46.22 51.52 49.59 51.62

AGGCN [39] (2019) 46.29 52.47 48.89 51.45

GRACR [43] (2023) 57.85 59.73 56.47 58.54

HDT-BP [17] (2023) 55.36 53.27 54.80 52.40

GLRE-BERT [37] (2020) – – 55.40 57.40

DISCO-BERT [19] (2021) 55.91 57.78 55.01 55.70

HDT-BP-BERT [17] (2023) 57.17 59.37 56.28 58.77

LSR-BERT [21] (2020) 52.43 59.00 56.97 59.05

GAIN-BERT [16] (2020) 59.14 61.22 59.00 61.24

SIRE-BERT [41] (2021) 59.82 61.60 60.18 62.05

CGM2IR-BERT [42] (2023) 60.02 62.01 60.24 62.06

CGM2IR-RoBERTa [42] (2023) 62.03 63.95 61.96 63.89

Our Methods SEMHRN-BERT 60.63 62.72 60.53 62.51

SEMHRN-RoBERTa 62.93 64.87 63.01 64.97

		Dev	Test
Sequence-based Methods	BERT-RE [23] (2019)	–	54.16	–	53.20
	Coref-BERT [16] (2020)	55.32	57.51	54.54	56.96
	ATLOP-BERT [16] (2021)	59.22	61.09	59.31	61.30
	MRN-BERT [27] (2021)	59.74	61.61	59.52	61.74
	Docunet-BERT [28] (2021)	59.86	61.83	59.93	61.86
	AFLKD-BERT [29] (2022)	60.08	62.03	60.04	62.08
	Correl-BERT [34] (2023)	59.39	61.34	59.12	61.32
	ERACL-BERT [35] (2022)	59.72	61.80	59.08	61.36
	CPT-BERT [36] (2023)	60.02	62.13	59.92	61.87
	DREEAM-BERT [30] (2023)	60.51	62.55	60.03	62.49
	RoBERTa-RE [16] (2020)	57.14	59.22	57.51	59.62
	Coref-RoBERTa [16] (2020)	57.35	59.43	57.90	60.25
	ATLOP-RoBERTa [16] (2021)	61.32	63.18	61.39	63.40
	Docunet-RoBERTa [28] (2021)	62.23	64.12	62.39	64.55
	AFLKD-RoBERTa [29] (2022)	62.16	64.19	62.57	64.28
	DREEAM-RoBERTa [30] (2023)	62.29	64.20	62.12	64.27
Graph-based Methods	GCNN [20] (2019)	46.22	51.52	49.59	51.62
	AGGCN [39] (2019)	46.29	52.47	48.89	51.45
	GRACR [43] (2023)	57.85	59.73	56.47	58.54
	HDT-BP [17] (2023)	55.36	53.27	54.80	52.40
	GLRE-BERT [37] (2020)	–	–	55.40	57.40
	DISCO-BERT [19] (2021)	55.91	57.78	55.01	55.70
	HDT-BP-BERT [17] (2023)	57.17	59.37	56.28	58.77
	LSR-BERT [21] (2020)	52.43	59.00	56.97	59.05
	GAIN-BERT [16] (2020)	59.14	61.22	59.00	61.24
	SIRE-BERT [41] (2021)	59.82	61.60	60.18	62.05
	CGM2IR-BERT [42] (2023)	60.02	62.01	60.24	62.06
	CGM2IR-RoBERTa [42] (2023)	62.03	63.95	61.96	63.89
Our Methods	SEMHRN-BERT	60.63	62.72	60.53	62.51
	SEMHRN-RoBERTa	62.93	64.87	63.01	64.97

4.5 Results on CDR and GDA

Table 3 presents the results of two biomedical datasets. Following Zhou et al. [16], SciBERT is also employed as the document encoder. SEMHRN-SciBERT outperforms state-of-the-art models on CDR and GDA datasets, surpassing MRN by 7.4 F1 and 3.7 F1, and ATLOP by 3.9 F1 and 2.7 F1, respectively. The strong performance across multiple datasets demonstrates the effectiveness of SEMHRN-SciBERT in identifying relations between entities in a document.

Table 3
Results (%) on the biomedical datasets CDR and GDA

Model CDR GDA

BRAN 62.10 –

LSR 62.10 62.10

DHG 65.90 83.10

GLRE 68.50 –

SciBERT 65.10 82.50

ATLOP-SciBERT 69.40 83.90

MRN-SciBERT 65.90 82.90

SEMHRN-SciBERT 73.30 86.60

Model	CDR	GDA
BRAN	62.10	–
LSR	62.10	62.10
DHG	65.90	83.10
GLRE	68.50	–
SciBERT	65.10	82.50
ATLOP-SciBERT	69.40	83.90
MRN-SciBERT	65.90	82.90
SEMHRN-SciBERT	73.30	86.60

Table 4

Ablation study (%) of SEMHRN on DocRED

Model	Ign F1	F1
SEMHRN-BERT	60.63	62.72
w/o Syntax-enhanced GCN	59.87	61.73
w/o Multi-hop reasoning GAT	59.96	62.00
w/o both module	59.15	61.09

Table 5

Results (%) of different strategies for utilizing syntax information on DocRED

Model	Ign F1	F1
SEMHRN	60.63	62.72
w/o dependency probability matrix	60.01	61.99
w/ dependency matrix	60.33	62.40
w/o dependency-type-aware attention	59.82	62.14

4.6 Analysis

4.6.1 Ablation study

An ablation study is conducted to analyze SEMHRN and demonstrate the effectiveness of various components within the framework. From the results presented in Table 6, it is observable that the removal of the grammar-enhanced GCN leads to a decrease in model performance by 0.99 in terms of F1 score. Likewise, the elimination of edges in the multi-hop reasoning GAT results in a reduction of 0.72 in the F1 score. Notably, when both the grammar-enhanced GCN and the multi-hop reasoning GAT are simultaneously omitted, the model experiences the most significant decrease in performance, with a decline of 1.63 in F1. These findings indicate that the performance of the framework diminishes as components are removed, underscoring the positive contributions of each component within SEMHRN to relation extraction.

Table 6
Training time and memory usage on DocRED

Model Memory Training time

LSR-BERT 11,436 MB 2.92 it/s

E2GRE-BERT 36,182 MB 0.53 it/s

SEMHRN-BERT 13,633 MB 4.62 it/s

Model	Memory	Training time
LSR-BERT	11,436 MB	2.92 it/s
E2GRE-BERT	36,182 MB	0.53 it/s
SEMHRN-BERT	13,633 MB	4.62 it/s

4.6.2 Analysis of syntax-enhanced graph convolutional network

Three sets of experiments are conducted to examine the effectiveness of dependency probability matrices and dependency type-aware attention in the context of Syntactically Enhanced Graph Convolutional Networks, as presented in Table 3. Initially, the removal of dependency probability matrices results in a significant decline in model performance, with a decrease of 0.73 in F1 score. Substituting dependency probability matrices with syntactic tree models leads to a reduction of 0.3 in F1 score. This indicates that syntactic trees provide some assistance in enhancing model performance, whereas dependency probability matrices encompass more extensive syntactic knowledge, mitigating syntactic parsing errors and enabling more efficient modeling of syntactic dependencies within documents. Furthermore, the omission of dependency type-aware attention results in a decrease of 0.58 in F1 score, with IgnF1 showing a pronounced drop of 0.81. The substantial reduction in IgnF1 underscores the contribution of dependency edge type information to the model’s ability to recognize previously unseen instances of relations. Dependency type-aware attention aids the model in selectively focusing on more critical dependency types, thereby enhancing overall model performance.

4.6.3 Analysis of multi-hop reasoning graph convolutional network

Exploring the impact of different multi-hop reasoning graph attention layer depths (l = 1, 2, 3, 4) on model performance, Fig. 5 illustrates that as the number of graph attention layers gradually increases, the model’s performance improves. When there are 3 graph attention layers, the model’s performance reaches its peak. However, further increasing the number of graph attention layers leads to a decline in model performance. This suggests that adding more graph attention layers enables more multi-hop dependency information between entity pairs to propagate to the current node. Still, an excessive number of graph attention layers can result in the excessive integration of distant noisy information.

Fig. 5

The results on different number of multi-hop reasoning graph attention layers.

4.6.4 Reasoning performance analysis

Following Zeng et al. [16], Infer-F1 is used as a metric to measure the model’s reasoning ability, focusing only on relationships involved in the reasoning process. For example, in the case of multi-hop reasoning, if paths like $e_{h} \overset{r_{1}}{⟶} e_{o} \overset{r_{2}}{⟶} e_{t}$ and $e_{h} \overset{r_{3}}{⟶} e_{t}$ exist, the golden relation instances r₁, r₂, r₃ are considered. Figure 6 clearly illustrates that SEMHRN improves Infer-F1 by 2.3 in the multi-hop reasoning scenario. The significant improvement in Infer-F1 indicates the model’s effective capability to capture multi-hop dependencies between entity pairs.

Fig. 6

The Inter-F1 results for relation instances involving multi-hop reasoning on development set of DocRED.

4.6.5 Efficiency analysis

The training time and memory consumption of SEMHRN are compared with other models under the same conditions. Table 6 shows that SEMHRN’s training occupies only 13,633 MB of memory, significantly lower than the 11,436 MB used by E2GRE. Additionally, SEMHRN’s training speed is 5.51 times faster than LSR and 8.72 times faster than E2GRE. Experimental results demonstrate that SEMHRN is a lightweight model.

4.7 Case study

A case study is conducted to further verify the effectiveness of SEMHRN and discuss the shortcomings of the model. As shown in Fig. 7, SEMHRN can identify most of the intra-sentence relations. For example, it can deduce that the relation between the subject “Square Enix” and the object “Square Enix Europe” is “subsidiary”. Additionally, it can recognize long-distance inter-sentence relations, such as the relation between the subject “Dragon Quest” and the object “Japanese”, which is “country of origin”. This demonstrates SEMHRN’s capability to capture long-distance dependencies between entity pairs.

Fig. 7

Case study of an example from the development set of DocRED. This example shows 19 relation instances and 11 sentences. The relation instances that the SEMHRN model did not correctly identify are highlighted in red, with the ground truth relations in parentheses.

However, SEMHRN fails to identify the relation between the subject “Final Fantasy” and the object “Square Enix”. There are two possibilities. The first possibility is that SEMHRN has limited ability to handle multi-label entity pairs. Another possibility is that it is challenging for the model to identify relation categories that occur less frequently in the dataset. Additionally, SEMHRN does not correctly identify the relation between entities “Eidos Interactive” and “Square Enix Europe”. Their relation should be “replaced by“/”replacements”, not “followed by“/”follows”, reflecting the model’s insufficient ability to distinguish relations with similar semantics. Therefore, addressing multi-label entity pairs, identifying minority relation types in the dataset, and distinguishing challenging relation types may be areas of focus for future research.

5 Conclusion

In this paper, SEMHRN is introduced as a model for performing multi-hop reasoning in the document-level relation extraction task. Instead of using the dependency tree directly to construct the adjacency matrix, a denser dependency probability matrix is employed to capture more syntactic information and alleviate dependency parsing errors to some extent. Additionally, dependency-type-aware attention is utilized to refine the weight of all edges based on the types of edges connecting nodes. The part-of-speech prediction task is introduced to regularize word embeddings. Furthermore, to enhance the interaction between entity pairs, a multi-hop reasoning graph attention network is proposed. This network initially models the two-hop reasoning between entity pairs and then expands to multi-hop reasoning. Experimental results demonstrate the effectiveness of each proposed component in improving the model’s performance.

Footnotes

Acknowledgments

This work was supported by National Natural Science Foundation of China (Grant No. 62376018).

References

Tian

, Li

, Ren

, Zhang

, Han

and Shen

, Intelligent question answering method for construction safety hazard knowledge based on deep semantic mining, Automation in Construction 145 (2023), 104670.

Catania

, Spitale

and Garzotto

, Conversational agents in therapeutic interventions for neurodevelopmental disorders: A survey, ACM Comput Surv 55 (2023), 1–34.

Zeng

, Liu

, Chen

and Zhao

, Distant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks, in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (2015), 1753–1762.

Zhou

, Su

, Zhang

and Zhang

, Exploring Various Knowledge in Relation Extraction, in, Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05) (2005), 427–434.

, Chen

, Mou

and Zhao

, Document-Level Relation Extraction with Sentences Importance Estimation and Focusing, ArXiv Preprint ArXiv:2204.12679, (2022).

Huang

, Yang

, Chen

, Zhao

, Liu

, Sun

and Zhao

, Document-Level Relation Extraction via Pair-Aware and Entity-Enhanced Representation Learning, in Proceedings of the 29th International Conference on Computational Linguistics, International Committee on Computational Linguistics (2022), 2418–2428.

Xie

, Shen

, Li

, Mao

and Han

, Eider: Empowering Document-level Relation Extraction with Efficient Evidence Extraction and Inference-stage Fusion, in Findings of the Association for Computational Linguistics: ACL 2022 (2022), 257–268.

Yao

, Ye

, Li

, Han

, Lin

, Liu

, Huang

, Zhou

and Sun

, DocRED: A large-scale document-level relation extraction dataset, ArXiv Preprint ArXiv:1906.06127 (2019).

Zhou

, Deng

, Chen

, Yang

, Jia

and Huang

, Exploiting syntactic and semantics information for chemical–disease relation extraction, Database, 2016.

10.

, Sun

, Qian

and Zhou

, Chemical-induced disease relation extraction via convolutional neural network, Database, 2017.

11.

Song

, Wu

, Song

, Zhang

and Stojanovic

, Bipartite synchronization for cooperative-competitive neural networks with reaction–diffusion terms via dual event-triggered mechanism, Neurocomputing 550 (2023), 126498.

12.

Lecun

, Bottou

, Bengio

and Haffner

, Gradient-based learning applied to document recognition, Proc IEEE 86 (1998), 2278–2324.

13.

Elman

J.L.

, Finding structure in time, Cognitive Science 14 (1990), 179–211.

14.

Kipf

T.N.

and Welling

, Semi-supervised classification with graph convolutional networks, ArXiv Preprint ArXiv:1609.02907 (2016).

15.

Veličković

, Cucurull

, Casanova

, Romero

, Liò

and Bengio

, Graph attention networks, ArXiv Preprint ArXiv:1710.10903 (2018).

16.

Zeng

, Xu

, Chang

and Li

, Double graph based reasoning for document-level relation extraction, ArXiv Preprint ArXiv:2009.13752 (2020).

17.

Wan

, Du

, Liu

, Fang

, Wei

and Liu

, Document-level relation extraction with hierarchical dependency tree and bridge path, Knowledge-Based Systems 278 (2023), 110873.

18.

Sun

, Zhang

, Huang

, Xu

, Li

and Liu

, Document-level relation extraction with two-stage dynamic graph attention networks, Knowledge-Based Systems 267 (2023), 110428.

19.

Wang

, Qin

, Lu

, Yin

, Zakari

R.Y.

and Owusu

J.W.

, Document-level relation extraction using evidence reasoning on RST-GRAPH, Knowledge-Based Systems 228 (2021), 107274.

20.

Sahu

S.K.

, Christopoulou

, Miwa

and Ananiadou

, Inter-sentence relation extraction with document-level graph convolutional neural network, ArXiv Preprint ArXiv:1906.04684 (2019).

21.

Nan

, Guo

, Sekulic

and Lu

, Reasoning with Latent Structure Refinement for Document-Level Relation Extraction, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020), 1546–1557.

22.

Wang

, Cao

, De Melo

and Liu

, Relation classification via multi-level attention cnns, in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Longapers), (2016), 1298–1307.

23.

Wang

, Focke

, Sylvester

, Mishra

and Wang

, Fine-tune Bert for DocRED with Two-step Process, ArXiv Preprint ArXiv:1909.11898 (2019).

24.

, Lin

, Du

, Liu

, Li

, Sun

and Liu

, Coreferential reasoning learning for language representation, ArXiv Preprint ArXiv:2004.06870 (2020).

25.

Eberts

and Ulges

, An end-to-end model for entity-level relation extraction using multi-instance learning, ArXiv Preprint ArXiv:2102.05980 (2021).

26.

Zhou

, Huang

, Ma

and Huang

, Document-level relation extraction with adaptive thresholding and localized context pooling, in Proceedings of the AAAI Conference on Artificial Intelligence (2021), 14612–14620.

27.

, Xu

, Li

, Fei

, Ren

and Ji

, Mrn: A locally and globally mention-based reasoning network for document-level relation extraction, in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 (2021), 1359–1370.

28.

Zhang

, Chen

, Xie

, Deng

, Tan

, Chen

, Huang

, Si

and Chen

, Document-level relation extraction as semantic segmentation, ArXiv Preprint ArXiv:2106.03618 (2021).

29.

Tan

, He

, Bing

and Ng

H.T.

, Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation, in Findings of the Association for Computational Linguistics: ACL 2022, 2022, 1672–1681.

30.

, Wang

and Okazaki

, DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction, in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, 1971–1983.

31.

Tang

, Cao

, Zhang

, Cao

, Fang

, Wang

and Yin

, HIN: Hierarchical inference network for document-level relation extraction, Advances in Knowledge Discovery and Data Mining 12084 (2020), 197–209.

32.

, Wang

, Lyu

, Zhu

and Mao

, Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction, in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 14149–14157.

33.

Huang

, Qi

, Wang

, Ma

and Huang

, Entity and Evidence Guided Document-Level Relation Extraction, in Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), 2021, 307–315.

34.

Han

, Peng

, Wang

, Liu

, Tiwari

and Wan

, Document-level relation extraction with relation correlations, Neural Networks, 2023.

35.

, Ma

, Wu

, Zhang

, Long

and Ji

, Improving long tailed document-level relation extraction via easy relation augmentation and contrastive learning, ArXiv Preprint ArXiv:2205.10511 (2022).

36.

Yuan

, Cao

and Huang

, Collective prompt tuning with relation inference for document-level relation extraction, Information Processing & Management 60 (2023), 103451.

37.

Wang

, Hu

, Cao

and Sun

, Global-to-Local Neural Networks for Document-Level Relation Extraction, in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020), 3711–3721.

38.

Christopoulou

, Miwa

and Ananiadou

, Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, 4925–4936.

39.

Guo

, Zhang

and Lu

, Attention guided graph convolutional networks for relation extraction, ArXiv Preprint ArXiv:1906.07510 (2019).

40.

, Chen

and Zhao

, Document-level relation extraction with reconstruction, in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 14167–14175.

41.

Zeng

, Wu

and Chang

, SIRE: Separate Intra-and Inter-sentential Reasoning for Document-level Relation Extraction, in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 (2021), 524–534.

42.

Zeng

, Zhao

, Jiang

, Zhu

and Dai

, Document-level relation extraction with context guided mention integration and inter-pair reasoning, IEEE/ACM Trans Audio Speech Lang Process, (2023), 1–9.

43.

Liu

, Kang

, Zhang

, Tian

and Hua

, Document-level Relation Extraction with Cross-sentence Reasoning Graph, ArXiv Preprint ArXiv:2303.03912, (2023).

44.

, Sun

, Johnson

R.J.

, Sciaky

, Wei

C.-H.

, Leaman

, Davis

A.P.

, Mattingly

C.J.

, Wiegers

T.C.

and Lu

, BioCreative V CDR task corpus: a resource for chemical disease relation extraction, Database, 2016.

45.

, Luo

, Leung

H.C.

, Ting

H.F.

and Lam

T.W.

, Renet: A deep learning approach for extracting gene-disease associations from literature, in Research in Computational Molecular Biology: 23rd Annual International Conference, 2019, 272–284.

46.

Kenton

J.D.M.-W.C.

and Toutanova

L.K.

, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in Proceedings of NAACL-HLT, 2019, 4171–4186.

47.

Liu

, Ott

, Goyal

, Du

, Joshi

, Chen

, Levy

, Lewis

, Zettlemoyer

and Stoyanov

, Roberta: A robustly optimized bert pretraining approach, ArXiv Preprint ArXiv:1907.11692 (2019).

48.

Beltagy

, Lo

and Cohan

, SciBERT: A Pretrained Language Model for Scientific Text, ArXiv Preprint ArXiv:1903.10676, (2019).

49.

Srivastava

, Hinton

, Krizhevsky

, Sutskever

and Salakhutdinov

, Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research 15 (2014), 1929–1958.

Advancing document-level relation extraction with a syntax-enhanced multi-hop reasoning network

Abstract

Keywords

1 Introduction

2.1 Sentence-level relation extraction

2.2 Document-level relation extraction

2.2.2 Graph-based methods

3 Method

3.1 Task formulation

3.2 Model overview

4.1 Dataset

4.2 Implementation details

4.3 Baselines methods

4.4 Results on DocRED

Table 3 Results (%) on the biomedical datasets CDR and GDA Model CDR GDA BRAN 62.10 – LSR 62.10 62.10 DHG 65.90 83.10 GLRE 68.50 – SciBERT 65.10 82.50 ATLOP-SciBERT 69.40 83.90 MRN-SciBERT 65.90 82.90 SEMHRN-SciBERT 73.30 86.60

4.6.1 Ablation study

Table 6 Training time and memory usage on DocRED Model Memory Training time LSR-BERT 11,436 MB 2.92 it/s E2GRE-BERT 36,182 MB 0.53 it/s SEMHRN-BERT 13,633 MB 4.62 it/s

4.6.3 Analysis of multi-hop reasoning graph convolutional network

4.7 Case study

Footnotes

Acknowledgments

References

Table 3
Results (%) on the biomedical datasets CDR and GDA

Model CDR GDA

BRAN 62.10 –

LSR 62.10 62.10

DHG 65.90 83.10

GLRE 68.50 –

SciBERT 65.10 82.50

ATLOP-SciBERT 69.40 83.90

MRN-SciBERT 65.90 82.90

SEMHRN-SciBERT 73.30 86.60

Table 6
Training time and memory usage on DocRED

Model Memory Training time

LSR-BERT 11,436 MB 2.92 it/s

E2GRE-BERT 36,182 MB 0.53 it/s

SEMHRN-BERT 13,633 MB 4.62 it/s