Neural Network with Multi-Level Attention for Extraction of Clinical Relationships

Abstract

Background

Accurate extraction of the relations between clinical entities is important for tasks like clinical decision support, automated medical coding, and large-scale analysis of clinical text. However, most of the medical records are written as long, unstructured, and sometimes noisy narratives. This makes it difficult for traditional relation extraction systems to properly analyze the document-level context. Improving relation extraction after entity recognition is still a key challenge in clinical natural language processing.

Methods

In this work, we propose a deep learning framework that combines RoBERTa, a bidirectional gated recurrent units (Bi-GRU), a multi-level attention mechanism, and a conditional random field (CRF) layer for clinical relation extraction. RoBERTa is used to generate strong contextual embeddings for each token. The Bi-GRU layers then model the sequential semantic dependencies in a more lightweight way compared to the transformer-only models. The multi-level attention module operates at both word and sentence level to down-weight the irrelevant or noisy information and highlight the clinically important context. Finally, a CRF layer is added on top to produce the globally consistent relation labels. The model is then trained and evaluated on the MTSamples clinical transcription dataset.

Results

The experimental results shows that the proposed model performs better than the recent baseline systems and achieves higher F1-score. The use of multi-level attention clearly improves the ability of the model to capture long-range context and document-level relation clues.

Conclusion

The RoBERTa-Bi-GRU-Attention-CRF framework offers an effective and scalable way to extract the relationships between entities from unstructured clinical narratives. The improved relation extraction performance can support more accurate downstream applications in clinical information extraction and intelligent healthcare systems.

Keywords

clinical relation extraction natural language processing RoBERTa multi-level attention electronic medical records

1 Introduction

Information extraction is one of the key functions in the Intelligent Communication Systems (ICS). It helps to deliver the intelligent and automated services in many domains. Typical application domain of ICS are: recommendation systems (Ahmadian et al., 2022; Chen et al., 2020, December 8–13), emotion monitoring (Chen et al., 2020, December 8–13), intelligent healthcare services (Li et al., 2022), web data retrieval (Arora et al., 2023; Liang et al., 2023), and wireless sensing technologies (Qiu et al., 2018). Government organizations also use ICS to analyze the public opinion and to support the policy decisions. Within this area, relation extraction (RE) is an important task, which focuses on finding the semantic links between the entities in unstructured text.

The use of the Electronic Medical Records (EMRs) has been increased rapidly since past decade. For example, in the United States, EMR adoption went from around 10% to more than 96% between 2008 and 2017. This change from paper-based to digital records is happened mainly due to the government incentives, advances in technology, and the need for better efficiency and patient care. EMRs store the rich clinical information, such as patient demographics, chief complaints, and medical history, physical examinations findings, diagnostic test results, medication history, treatment plans, progress notes, and discharge summaries. Extracting the structured information from these unstructured texts (EMRs) is an essential task for many of the healthcare researchers.

One common way to extract the structure information is to build the triplets in the form of a subject-predicate- and object model. In the medical domain, a triplet can connect a patient (subject) to a diagnosis (predicate) and a specific disease (object). Such triplets supports several important applications in real life are: (i) analyzing the relationship between the patient attributes and disease outcomes for health risk prediction (Nelson et al., 2022; Ning et al., 2022), (ii) building the clinical knowledge bases to assist the clinical decision-making (Li et al., 2020; Liu et al., 2023), and (iii) developing the diagnostic support tools that can combine the information from clinical records and biomedical literature (Wu et al., 2018).

However, manually extracting the clinical entity relationships from EMRs is a slow process, inconsistent, and error-prone also. Recent progress observed in the natural language processing (NLP) and machine learning has greatly improved the level of automation, accuracy, and scalability for this relation extraction task.

Entity relationship extraction from the clinical abstracts is mainly useful for constructing the clinical knowledge graphs. These Knowledge graphs can automatically identify the relationships among treatments, interventions, outcomes, and patient characteristics. They reduce the effort needed for the evidence synthesis, support compliance with clinical trial protocols, and enables more personalized healthcare. Knowledge graphs built from these relationships helps the clinicians to stay updated on new therapeutic options and the current research trends. In addition to this, analyzing the extracted relationships can also highlight the research gaps in existing studies and guides the researcher to design of future clinical trials (Mohamed et al., 2020).

In this work, we use the MT Samples dataset as a representative benchmark. It contains transcriptions from about 40 medical specialties, with 140,214 total sentences and around 50,000 unique sentences. The proposed entity relationship extraction model is able to process large volumes of clinical text and identify the important clinical entities and their relationships. This model improves the understanding of clinical research data and supports the downstream applications such as intelligent medical question answering, disease prediction, drug recommendation systems, and detection of adverse drug reactions.

The main contributions of these studies are summarized below:

First, based on the insights from UMLS and previous studies, we identify the clinically relevant associations in text and constructs a specialized corpus for clinical entity relationship extraction.

Second, we use pre-trained language models to capture the semantic information and to generate the dynamic word representations. Then, a hierarchical collaborative Bidirectional GRU (Bi-GRU) network is applied to extract the deep latent features from the generated input vectors.

Third, we design a multi-level attention mechanism to effectively capture the important contextual information from each sentence of clinical text.

Finally, we add a conditional random field (CRF) layer on top of the network model to produce more accurate and consistent predictions of relationships among the clinical entities.

2 Related Work

In the NLP, a relation means the meaningful connections or association between two or more entities. The main goal of RE is to automatically find and extract these connections from the text, usually in the form of triplets and their components. Existing RE methods are generally divided into three groups are: rule-based methods, classical Machine Learning (ML) methods, and deep learning (DL) approaches (Shi et al., 2020).

Rule-based methods depends on linguistic experts to design the grammar rules or pattern templates for detecting the relations. These methods can work well in domain-specific environments, but they are usually not generalized and flexible to work on other domains. To improve the pattern-based techniques, Nakashole et al. (2012) suggested using of external knowledge, such as knowledge graphs, to enrich the entity and relation representations. This kind of integration can improve the accuracy of relation extraction and make the results more useful for downstream applications.

Zhang et al. (2017) reviewed the key traditional methods for RE and noted that many of them are based on statistical language models. Classical supervised ML approaches are often divided into two categories are: kernel-based methods (Zelenko et al., 2003) and feature-engineering methods (Xu et al., 2012). In feature-engineering methods, linguistic features are transformed into vector representations. In kernel-based methods, the special kernel functions are used to measure the similarity between these entities.

In recent years, DL based methods have become very popular for RE, especially for those that use Graph Neural Networks (GNNs) and Distant Supervision (DS). Wu et al. (2021) discussed the basic ideas, development, and research trends of GNNs, while Scarselli et al. (2009) introduced the original GNN framework. GNN based models are often effective for RE, because they can capture the structural dependencies among the entities in a sentence or document. DS frameworks assumes that, if two entities are linked by a relation in a knowledge base and appear together in a sentence, then that sentence is likely to express that relation.

The Graph Convolutional Networks (GCNs) are a special type of GNNs, which applies the convolution operations on graph structures, and they have been widely used for the RE tasks. Zhang et al. (Mohamed et al., 2020) proposed a contextualized GCN (C-GCN), which uses a pruned dependency tree to focus on the shortest path between the potential entity pairs. Guo et al. (2019) introduced an attention-based GCN (AGGCN), which allows the model to automatically learn and choose subgraph structures that are very helpful for the RE. Jin et al. (2020) pointed out that many earlier works did not modeled the interdependent relations jointly, which can lead to the loss of relational context. They suggested the use of GCNs can efficiently capture the mutual dependencies among the relations. In another example of DL expanding into new application areas, Wu et al. (2019) presented a deep reinforcement learning approach for the autonomous search by UAVs in complex disaster environments.

In real life applications, the RE process often treated as a text classification problem (Dongmei et al., 2020). This view makes it possible to use the standard text classification architectures in RE. Transformer based pre-trained language models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have shown their strong performance on many NLP tasks, including the text classification (Devlin et al., 2019; Jin et al., 2020). Because they are capable of learning the rich contextual representations from base text, they become standard and very suitable DL models for RE.

Recently, biomedical and clinical RE has seen notable progress using the hybrid models that combines the transformers, graph structures, and Large Language Models (LLMs). Fang et al. (2025) proposed GLiM hybrid model, which dynamically builds a graph-transformed representation of clinical text. On graph, it uses an LLM for reasoning to handle the document-level biomedical RE with incomplete annotations. Yuan et al. (2024) proposed HTGRS model, which uses the hierarchical tree graph and a relation-segmentation module to capture the contextual information and to improve the cross-sentence inference.

Some other research works in this domain combined the syntactic structure with transformers. Kim et al. (Chen et al., 2020, December 8–13) combined the BERT with a Graph Attention Network (GAT), which is applied over the dependency parses. This hybrid framework with a simplified T5 model, proven that using the syntax with transformers can improve the performance. Jain et al. (2023) proposed ReOnto, a neuro-symbolic system that uses the ontological paths and GNNs to bring in domain knowledge from biomedical ontologies. Chaturvedi et al. (2025) focused on temporal relation extraction from the clinical narratives and used a span-based graph transformer to model the event sequences.

From this literature, we find that, in clinical domain, transformer-based RE has given promising results. Yang et al. (2021) systematically evaluated the BERT, RoBERTa, and XLNet models on clinical RE tasks. The results noted that RoBERTa clinical and XLNet clinical were achieved the higher F1-scores on benchmark datasets (MADE1.0, n2c2) than standard BERT.

More recently, the methods that combine domain knowledge with transformers have appeared in this domain. Roy and Pan (2021) enriched the BERT with UMLS knowledge for clinical relation extraction. In this method, the model can use both structured medical knowledge and the contextual language representations.

3 Methodology

In this section, we thoroughly describe the proposed model for extracting the entity relationships from clinical texts. The model is built up on top of the pre-trained RoBERTa language model, which has shown in strong performance on many NLP tasks. First, we explain the overall framework and its basic theory. Then, we define the types of the entity relationships considering and how the extraction task implemented. After that, describes how the data samples are labeled and build a dedicated relation extraction corpus from the clinical trial texts. To improve the generalization and to reduce the overfitting, dropout regularization applied in the network. Finally, the comparative experiments presents that the proposed models are effective and performs better than the baseline methods for clinical entity-relation extraction.

3.1 Overview of the Entity-Relation Extraction (E-RE) Model

The main goal of the Entity-Relation Extraction (E-RE) model is to capture the semantic relationships between entities presented in the clinical text. Using these extracted relations, a knowledge map is constructed for presenting the interconnected domains, which connects the concepts within the clinical domain.

3.1.1 Problem Definition

Let the training dataset for relation extraction be denoted as $D_{s} = {⟨ E_{1}, E_{2}, R_{e} ⟩, S}$ where $E_{1}$ and $E_{2}$ are the two entities, $R_{e}$ is the set of the possible relation types, and $S$ is the corresponding sentence or text segment. The relation extraction task aims to learn a mapping function as: $φ (e_{1}, e_{2}, S) \to r$ from the training data $D_{s}$ , where $r \in R_{e}$ is the relation label. The main objective is to maximize the proportion of the correctly predicted relations on a validation set $V^{'}$ and on a test set $D_{s}^{'}$ , using the same input format and features as in training.

3.1.2 Model Architecture

Figure 1 presents the complete pipeline of our E-RE model, which follows the Hierarchical Attention Networks as backbone was introduced by Yang et al. (Roy & Pan, 2021). The model is organized into five clearly separated levels, namely: Input layer, Representation layer, Word-level encoding and attention, Sentence-level encoding and attention, and CRF decoding layer. The process flow steps of the architecture is defined as follows:

Figure 1.

Entity-relation extraction (E-RE) model (proposed).

Step-1: Input Layer

At the input layer, the document is represented as a sequence of sentences. Each sentence $S_{j}$ consists of a sequence of tokens as: $S_{j} = {w_{1}^{j}, w_{2}^{j}, \dots, w_{i_{j}}^{j}}$ where $w_{i}^{j}$ denotes the i-th word in the j-th sentence, and $i_{j}$ is the length of sentence $S_{j}$ . As shown in architecture, the whole document can be written as: $D = {S_{1}, S_{2}, \dots, S_{n}}$ This input formulation preserves the natural hierarchical structure of clinical text, where words form sentences and sentences form documents.

Step-2: Representation Layer (RoBERTa Encoder)

At this layer, all input tokens $w_{i}^{j}$ are first passed into the RoBERTa representation layer, for converting each word into a contextual embedding vector by considering both left and right context within the sentence. After RoBERTa encoding, each word $w_{i}^{j}$ is mapped to a dense vector: $x_{i}^{j} = RoBERTa (w_{i}^{j})$ these contextual embeddings serve as the input to the word-level Bi-GRU networks. Using this RoBERTa layer the model handle polysemy and contextual variation, which are very common in clinical narratives.

Step-3: Word-Level Encoding with Bi-GRU

For each sentence $S_{j}$ , the corresponding RoBERTa embeddings ${x_{1}^{j}, x_{2}^{j}, \dots, x_{i_{j}}^{j}}$ are fed into a word-level Bi-GRU network and the forward and backward GRU compute hidden states are defined as: $\vec{h_{i}^{j}} = GR U_{f} (x_{i}^{j}), \overset{\leftarrow}{h_{i}^{j}} = GR U_{b} (x_{i}^{j})$ . At the end, the final word representation is obtained by using the concatenation function as: $h_{i}^{j} = [\vec{h_{i}^{j}}; \overset{\leftarrow}{h_{i}^{j}}]$ In this, the hidden states $h_{i}^{j}$ encode both the past and future contextual information within the same sentence.

Step-4: Word-Level Attention Mechanism

In input, not all the words contribute equally to the meaning of a sentence. Therefore, a word-level attention mechanism is applied, as is denoted by $a_{s_{1}}, a_{s_{2}}, \dots, a_{s_{n}}$ . For each word in the hidden state $h_{i}^{j}$ , an attention score is computed as : $α_{i}^{j} = softmax (u_{i}^{j})$ where $u_{i}^{j}$ is obtained through a trainable attention layer and the sentence representation $S_{j}$ is then calculated as a weighted sum of word representations as follows:

S_{j} = \sum_{i = 1}^{i_{j}} α_{i}^{j} h_{i}^{j}

This process allows the model to focus on important clinical words, such as disease names, symptoms, or treatment actions, while reducing the influence of irrelevant tokens.

Step–5: Sentence-Level Encoding with Bi-GRU

After word attention, the sentence vectors ${S_{1}, S_{2}, \dots, S_{n}}$ are next passed into a sentence-level Bi-GRU network. This sentence-level Bi-GRU models the dependencies between sentences, which is important in clinical documents where relevant information may appear across multiple sentences. During the attention process, the forward and backward sentence-level hidden states are defined as: $\vec{v_{j}} = GR U_{f} (S_{j}), \overset{\leftarrow}{v_{j}} = GR U_{b} (S_{j})$ and the final sentence-level hidden representation is calculated as: $v_{j} = [\vec{v_{j}}; \overset{\leftarrow}{v_{j}}]$ .

Step-6: Sentence-Level Attention Mechanism

To further identify the most informative sentences, a sentence-level attention mechanism is applied. In this, each sentence hidden state $v_{j}$ is assigned as an attention weight using softmax as: $β_{j} = softmax (z_{j})$ The document-level representation is then computed as: $V = \sum_{j = 1}^{n} β_{j} v_{j}$ .

This mechanism enables the model to emphasize the sentences that are more relevant for clinical entity prediction, such as diagnostic conclusions or treatment descriptions.

Step-7: CRF Decoding Layer

At the top of the architecture, the final representation is passed to a CRF layer. This layer models the label dependencies between adjacent tokens, for ensuring that the predicted label sequence follows valid constraints. Instead of predicting labels independently, the CRF finds the optimal global label sequence using transition scores. During inference, the Viterbi algorithm is used to decode the best label sequence.

This hierarchical and attention-guided design allows the model to jointly capture the local word-level semantics, sentence-level structure, and global document-level context. This clear and simplified flow making the architecture well suited for complex clinical text analysis.

3.2 Pre-Trained Model

In this study, we uses a pre-trained RoBERTa encoder model to generate the contextual word representations from clinical text. These embeddings are then provided as input to the downstream Bi-GRU and multi-level attention modules of our E-RE framework. RoBERTa belongs to the same Transformer family as BERT, but it is trained with several design changes for stronger and more stable representations. In our work, we select the RoBERTa as backbone because of the two important differences when compared with BERT are (i) the tokenization/segmentation strategy, and (ii) the pre-training procedure (clearly summarized in Figure 2).

Figure 2.

BERT vs RoBERTa comparison for representation learning in our model. (a) Masking policy used during masked language modeling, (b) the next sentence prediction (NSP) modeling, (c) byte-level text encoding/tokenization, and (d) common RoBERTa variants/model scales.

At first, the RoBERTa uses Byte Pair Encoding (BPE) for subword segmentation, while BERT uses the WordPiece algorithm. In BPE, the vocabulary is constructed by starting from a character-level inventory and repeatedly merging the most frequent adjacent symbol pairs until reaching to a fixed vocabulary size. This frequency-driven merging process is simple and efficient than BERT, and it helps to create the consistent subword units across the related word forms. For example, if we consider the words such as “lower” and “lowest” may share the subword stem “low”, which helps the model learn morphological patterns more effectively. In addition, RoBERTa's byte-level encoding technique reduces the number of unknown tokens during the tokenization process. This much useful in clinical narratives where abbreviations, rare terms, and spelling variations occur frequently. In contrast to BPE, the WordPiece chooses subwords based on maximizing the likelihood improvement under the language model objective. This process is more expensive to optimize over very large corpora. Due to this, the BPE strategy is selected and used in RoBERTa to support the robust subword decomposition and improve the coverage for domain-specific terminology.

Second, RoBERTa differs from BERT in the pre-training strategy. BERT is trained with Masked Language Modeling (MLM) together with Next Sentence Prediction (NSP). Its masking pattern is typically selected once during the preprocessing and then kept fixed. In contrary, the RoBERTa removes the NSP objective and adopts dynamic masking. In this, different tokens may be masked in different epochs even for the same sentence. This means the model observes many masking configurations over training. This helps to increase the model's diversity of supervision signals and improves the learning capacity, especially when the training corpora are large. For instance, in a sentence like “The quick brown fox jumps over the lazy dog,” different words may be masked across iterations. So the model learns to recover the missing information under multiple contexts instead of only one fixed mask layout. This dynamic masking is beneficial for building the stronger contextual representations, which is very important for our clinical relation extraction setting where relation cues can be subtle and context-dependent.

3.3 Entity-Relation Extraction (E-RE) Model

In proposed model, the Gated Recurrent Unit (GRU) network is selected, which is a popular variant of the Long Short-Term Memory (LSTM) networks and was originally proposed in (Williams & Zipser, 1989). We employed a Bi-GRU, which considers the past and future context while computing the hidden state at that time t.

The proposed E-RE model integrates the Bi-GRU network as shown in Figure 1. At both word level and sentence level, the Bi-GRU encodes the input into several contextual representations. Bi-GRU keeps the ability of LSTM to model the long-term dependencies but uses a simple internal structure than Bi-LSTM. By combining the forward and backward GRU states, the model learns deep contextual features and captures semantic relationships, which are depended on the entire input sequence.

3.3.1 Bi-GRU at the Word Level

At the word level, the Bi-GRU encodes each token from the sentence. The forward hidden the state $\vec{h_{t}}$ at time $t$ is computed from the current input $I_{t}$ and the previous forward state $\vec{h_{t - 1}}$ . Similarly, the backward hidden state $\overset{\leftarrow}{h_{t}}$ is computed in reverse order. The final hidden state $h_{t}$ obtained by concatenating these two states are:

\begin{aligned} \vec{h_{t}} & = \vec{GRU} (\vec{h_{t - 1}}, I_{t}) \end{aligned}

(1)

\begin{aligned} \overset{\leftarrow}{h_{t}} & = \overset{\leftarrow}{GRU} (\overset{\leftarrow}{h_{t + 1}}, I_{t}) \end{aligned}

(2)

\begin{aligned} h_{t} & = [\vec{h_{t}} ⊙ \overset{\leftarrow}{h_{t}}] \end{aligned}

(3)

Here, $⊙$ denotes the concatenation of operation, and $t$ ranges over the sequence length. The GRU applies a non-linear transformation at each step, which enables the model to capture complex dependencies among the words in the sentence.

3.3.2 Bi-GRU at the Sentence Level

At the sentence level, the model treats each sentence $S_{i}$ as a unit and encodes it with another Bi-GRU (SL Bi-GRU). The forward and backward the sentence-level hidden states are then computed as:

\begin{aligned} \vec{h_{i}} & = \vec{GRU} (\vec{h_{i - 1}}, S_{i}) \end{aligned}

(4)

\begin{aligned} \overset{\leftarrow}{h_{i}} & = \overset{\leftarrow}{GRU} (\overset{\leftarrow}{h_{i + 1}}, S_{i}) \end{aligned}

(5)

\begin{aligned} h_{i} & = [\vec{h_{i}} ⊙ \overset{\leftarrow}{h_{i}}] \end{aligned}

(6)

Using Bi-GRU at both word and sentence levels allows the proposed model to capture the sequential dependencies within sentences and between the sentences. After obtaining these hidden states, a self-attention (SA) is applied to highlight the semantically important parts. At the end of this process, word-level and sentence-level features are combined to form a rich representation of the document, which is used for relation extraction in further.

3.3.3 Attention Mechanism at Word and Sentence Level

In recent years, the attention mechanisms became standard models in the computer vision, speech, and NLP, as they can efficiently capture the long-range dependencies and focuses on important context (Vaswani et al., 2017, December 4–9). The core idea of attention mechanism in RE is to compute a relevance score for each unit (word or sentence) to indicate that how much it contributes to the overall meaning.

Word-Level Attention (WL-Attention): Given a token sequence of length M, the Bi-GRU produces a sequence of contextual word-level features as: $H_{w} = [h_{w 1}, h_{w 2}, \dots, h_{w M}]$ . These features are then passed to the SA module to compute an attention weight vector $α$ , which reflects the relative importance of each hidden state:

\begin{aligned} α = s o f t m a x (t a n h (H_{w}) w^{T}) \end{aligned}

(7)

The attention-weighted representation of the word sequence is then given by:

\begin{aligned} h_{w}^{'} = tanh (α^{T} H_{w}) \end{aligned}

(8)

Sentence-Level Attention (SL-Attention): Before applying sentence-level attention, the Bi-GRU encodes each sentence into a sentence embedding $m_{i}$ . Let $m_{k}$ be the context vector at the sentence level, the final sentence representation is then computed as:

\begin{aligned} h_{s}^{'} = \sum_{i} h_{w}^{'} \cdot \frac{exp (m_{k} m_{i}^{T})}{\sum_{i} exp (m_{k} m_{i}^{T}))} \end{aligned}

(9)

This two-stage attention (word-level and sentence-level) helps the models to focus on the most informative features before making them as final relation predictions.

3.3.4 CRF-Based Relation Label Prediction

To predict the relation label for each entity pair, the CRF layer is placed on top of the Bi-GRU and attention outputs. Unlike the standard sequence tagging with BIO labels, our CRF acts as a structured classifier at the sentence level. It receives the attention-enhanced sentence representation $h_{s}^{'}$ and produces a single relation label. Let $z$ be the combined feature vector passed into the CRF layer. At this stage the CRF models the conditional probability of a label sequence y as:

\begin{aligned} p (y ∣ z) \propto exp (\sum_{t} φ_{t} (y_{t}, z) + \sum_{t} ϕ (y_{t - 1}, y_{t})) \end{aligned}

(10)

Here, $φ_{t}$ are the unary potentials derived from input feature vector, and $ϕ$ are the transition potentials learned between labels. Since the model output only one relation label per entity pair, the CRF behaves like a transition-constrained classifier, where the transition matrix encodes compatibility between the possible relation outcomes.

During the decoding process, the Viterbi algorithm is applied to find out the label sequence with the highest score. This process results a relation prediction that respects the structural constraints learned by the CRF, making it more robust than an unconstrained softmax classifier.

3.3.4.1 Running Example

Consider the sentence: “Aspirin reduces inflammation in patients.” with the entity pair (Aspirin, inflammation).

The Bi-GRU encodes the sentence, and the attention layers to highlight the verb reduces and its relation to both entities. The resulting feature vector is then given to the CRF layer for producing the sample unary scores presented in Table 1:

Table 1.
Unary Scores for Relation Labels.

Relation Label Unary Score

CAUSE 8.3

TREAT 6.9

ASSOCIATED 5.1

NO-RELATION 2.4

Relation Label	Unary Score
CAUSE	8.3
TREAT	6.9
ASSOCIATED	5.1
NO-RELATION	2.4

After adding the transition constraints and applying Viterbi decoding, CAUSE becomes the highest-scoring and most consistent label. Therefore, the model outputs as: $R e l a t i o n (A s p i r i n, i n f l a m m a t i o n) = C A U S E$

3.3.5 Handling Non-Related Entity Pairs

This study assumes that not every entity pair in a sentence from clinical text has a meaningful relation. To avoid forcing the model to assign a relation to every pair, we explicitly include NO-RELATION as a separate label in the CRF output space. During training, all candidate entity pairs are enumerated in each sentence. Pairs that are annotated with a relation are labeled with gold relations (e.g., CAUSE, TREAT), while the remaining pairs are labeled as NO-RELATION. These negative examples are used later to compute the both unary potentials and CRF transition scores. This helps the model learns how to predict the NO-RELATIONs presented in text.

During the evaluation, NO-RELATION predictions are treated as negative cases. To follow the standard biomedical RE evaluation protocols, the metrics such as Precision, recall, and F1-score are calculated only for the positive relations, by excluding the NO-RELATION class.

3.3.5.1 Running Example

Using the similar sentence “Aspirin reduces inflammation in patients.” the entity pair (Aspirin, inflammation) is again considered. The attention mechanism focuses on the word reduces and its connection to both entities. The CRF might output the unary scores as shown in Table 2:

Table 2.
Unary Scores Including NO-RELATION.

Relation Label Unary Score

CAUSE 8.3

TREAT 6.9

ASSOCIATED 5.1

NO-RELATION 2.4

Relation Label	Unary Score
CAUSE	8.3
TREAT	6.9
ASSOCIATED	5.1
NO-RELATION	2.4

After considering transition constraints, the Viterbi decoder still selects CAUSE as the best label, and results as:

R e l a t i o n (A s p i r i n, i n f l a m m a t i o n) = C A U S E

This example also shows that proposed model can distinguish between the real relations and the NO-RELATION case from the clinical text when appropriate.

3.4 Overfitting Mitigation Strategy

Based on the model architecture described in the previous sections (RoBERTa encoder, Bi-GRU hierarchical layers, and self-attention modules), there is a clear risk of overfitting during the model training process. Overfitting happens when the model starts to learn noise or very specific patterns from the training data, instead of the general patterns. In that case, the model shows high accuracy on the training set, but its performance becomes poor on the validation and/or test sets. This problem is considerable in our study because we use a limited amount of annotated biomedical text and the complex clinical language.

In general, overfitting appears mainly for three reasons:

Model complexity: When the model has many parameters or very complex structure, it memorizes the training samples instead of the learned generalized features.

Insufficient training data: If the training corpus is small or the samples are sparse, the model does not see enough variety and cannot properly learn the underlying linguistic and semantic patterns.

Distribution mismatch: When the training data distribution is different from the validation or test data, the model may not generalize well on real-world inputs.

Ideally, increasing the size of the annotated dataset would help to reduce the overfitting. However, in the biomedical domain this is difficult, because the clinical abstracts are limited and the manual annotations by domain experts is very costly and time-consuming. Therefore, instead of relying only on more data, need to use regularization techniques during the training to control overfitting.

This study customizes and apply the Dropout regularization method proposed by Hinton et al. (Luo et al., 2018). Dropout improves the generalization by randomly turning off (dropping) a subset of neurons during each training step. This prevents the strong co-adaptation between units and stops the model from depending too much on a small set of features. In practice, typical dropout rates are between 0.2 and 0.5, and we chosen values in this range for our experiments. As shown in Figure 1 (in the model architecture), dropout the applied at the word-level encoding layers and the sentence-level encoding layers. In this way, the regularization process is enforced at multiple stages of the hierarchical representation learning process to control the model overfitting.

By integrating the dropout into the proposed E-RE framework, our model is able to learn more robust features, reduces the negative effect of limited training data, and achieves better relation extraction performance on unseen biomedical text.

4 Experiments

4.1 Dataset

In clinical text processing, collecting the high-quality medical data for research is still difficult because of the strict privacy rules such as HIPAA. For this reason, limited medical records and medical text datasets are publicly available. In this work, we use the medical transcription dataset from mtsamples.com, which contains transcriptions from many different medical specialties. The dataset covers around 40 categories and includes 140,214 transcribed phrases with about 50,000 unique words. Figures 3 and 4 shows the distribution of samples across medical specialties and the ranking of missing values in descending order.

Figure 3.

Word count of each medical specialty.

Figure 4.

Sentence length distribution.

In total, the dataset has 2,311,419 words and a vocabulary size of 28,581. The maximum sentence length is 3,114 words, while the median and mean sentence lengths are 421.5 and 487.5 words, respectively. These numbers show that the average transcription is quite long, with both the mean and median sentence length is above 400 words. This reflects the narrative and detailed nature of clinical documentation.

4.1.1 Data Splitting Strategy

To support the reliable training and evaluation, we split the dataset into training, validation, and test sets are using a 70:15:15 ratio. That is, 70% of the samples are used for training, 15% for the validation, and 15% for the final testing. Because the number of samples is very different across medical categories, we used the stratified sampling model. This model ensures that the proportion of each medical specialty is similar in the training, validation, and test sets. Stratification also helps to avoid the severe class imbalance and reduces the risk of biased learning. This is important for relation extraction tasks, where the context and style can change frequently between the specialties.

4.1.2 Annotation Process

The annotation process is a key step in preparing the MTSamples clinical transcription dataset for the relation extraction process. Raw medical transcriptions are unstructured, so a systematic procedure is applied to identify and label the entities and their relationships.

First, we identify clinical entities such as: Symptoms, Diagnoses, Procedures, Medications, Anatomical locations, and Laboratory or diagnostic findings. The annotation guidelines are derived from the UMLS semantic types and existing clinical NLP literature. These guidelines helps to ensure that the annotators follow a consistent rule when deciding what counts as an entity and how to classify it. Next, examined the sentences that contain pairs of entities to check whether there is a meaningful clinical relationship between them or not. Annotators assign the relation labels that represent clinically relevant interactions, such as treats, causes, indicates, associated-with, or part-of. To reduce the ambiguity in this annotation process, we design a detailed annotation protocol. This protocol contains with examples, instructions for handling borderline cases, and clear rules for multi-word entity boundaries.

Annotations are done manually by the people with basic domain knowledge of clinical terminology. Each transcript is reviewed independently by at least two annotators. When there is a disagreement in annotation process, the annotators discuss the case, and if needed, a senior reviewer makes the final decision. The inter-annotator agreement scores also computed to check the quality and consistency of the annotations. At the end, the labeled data are converted into a structured format, which is suitable for models training. Sentence-level and document-level records that include entity spans, relation labels, and the surrounding context are generated in annotation.

Categories that have very few transcription records are excluded from the annotation process. Only categories with at least 50 samples are kept. Figure 5 shows the number of labels in the datasets, and there are about 40 target labels in total.

Figure 5.

Sample size of each medical specialty.

Table 3 gives a few examples of medical transcriptions with their corresponding specialties. Each transcription shown in Tables 3 and 4 belongs to a specific medical specialty and contains a rich set of medical terms and abbreviations.

Table 3.

Sample Instances of the Medical Records with Their Medical Specialty.

Medical_specialty	Transcription (Excerpt)
Allergy/Immunology	Subjective: The individual is a 23-year-old Caucasian female who reports allergies. She had allergies while living in Seattle, but she feels they are worse at her current location. She has tried Claritin and Zyrtec in the past, both for a short period …
Bariatrics	Current medical history: I saw ABC today. He is a pleasant 42-year-old man weighing 344 pounds. His height is 5 feet 9 inches, giving a BMI of 51. He has been obese for 10 years, starting at age 33, with a peak weight of 358 pounds.

Table 4.

Data Inspection Sample.

ID	Brief Description	Clinical Specialty	Sample Title	Transcription (Short Excerpt)	Keywords
0	23-year-old woman attending with allergy-related symptoms	Allergy / Immunology	Allergic Rhinitis	SUBJECTIVE: A 23-year-old white female reports ongoing nasal allergy complaints …	Allergy/immunology, rhinitis, allergy case
1	Pre-operative consultation for gastric bypass surgery	Bariatrics	Laparoscopic Gastric Bypass Consult - 2	PAST MEDICAL HISTORY: The patient reports shortness of breath when climbing stairs …	Bariatrics, laparoscopic gastric bypass, weight issues
2	Initial assessment for bariatric surgery	Bariatrics	Laparoscopic Gastric Bypass Consult - 1	HISTORY OF PRESENT ILLNESS: The patient, referred as ABC, is being evaluated for obesity …	Bariatrics, bariatric surgery, cardiac risk
3	Echocardiography report with Doppler findings	Cardiovascular / Pulmonary	2-D Echocardiogram - 1	2-D M-MODE: Findings include left atrial enlargement with additional Doppler observations …	Cardiovascular/pulmonary, 2-D M-mode, Doppler
4	Detailed two-dimensional echocardiogram interpretation	Cardiovascular / Pulmonary	2-D Echocardiogram - 2	The report describes left ventricular cavity dimensions and wall motion on 2-D echocardiogram …	Cardiovascular/pulmonary, 2-D, Doppler, echo

Table 4 shows a portion of the original dataset after basic inspection.

4.2 Relationship Between Entities

In clinical text, the main goal of the relation extraction task is to identify the conceptual connection between each pair of medical entities mentioned in a sentence. When a meaningful connection exists between two entities are, the relation extraction tasks are becoming a classification problem: we must decide which relation label best describes the link.

Before extracting the relations from text, it is important to clearly define that which types of clinical ER types are of interest. Then assign these ER types to entity pairs in the sentence according to their meaning. In general, the clinical health records contain rich and detailed medical information. Meystre et al. (Uzuner et al., 2010) suggest that usage of ‘problem-oriented medical records’ to collect the data that support diagnosis, care planning, and other medical interventions of patient. Uzuner et al. (2010) classified the semantic relations in clinical abstracts. With this, they defined the relation types suitable for problem-centered records, taking into account the special characteristics of clinical text. Inspiring from the literature, this study defines six types of clinical trial data relationships are: Disease-Drug, Symptom-Drug, Intensity-Drug, Disease-Item, Disease-Method and Disease-Intensity. In our implementation, we refine them into the categories as shown in Table 5.

Table 5.
Clinical Entity Relation Types with Example Pairs.

Entity Relation Description Sample Entity Pairs

1. Disease-Drug Relation between the name of a disease and the biologic or drug used in its treatment trials. Obesity is a clinical problem and use of dexfenfluramine hydrochloride for weight reduction.

2. Symptom-Drug Relation between clinical signs or symptoms of disease and the biological agents used to manage them. Statin is used to reduce the amount of cholesterol that the body makes.

3. Intensity-Drug Relation between the severity or extent of patient illness and the use of relevant biologics in therapy. Acetaminophen given IV may cause vomiting and constipation.

4. Disease-Item Relation between the disease name and different test items or physiological indicators used for evaluation. Leukemia is usually identified by analyzing a blood sample through a CBC test.

5. Disease-Method Relation between the disease name and a treatment method or therapeutic approach. Belsomra is a dual orexin receptor antagonist (DORA) used to treat insomnia.

6. Disease-Intensity Relation between the disease and the degree or severity of the patient's condition or infection. When fighting a common cold, the body may respond with fever, coughing, and sneezing.

Entity Relation	Description	Sample Entity Pairs
1. Disease-Drug	Relation between the name of a disease and the biologic or drug used in its treatment trials.	Obesity is a clinical problem and use of dexfenfluramine hydrochloride for weight reduction.
2. Symptom-Drug	Relation between clinical signs or symptoms of disease and the biological agents used to manage them.	Statin is used to reduce the amount of cholesterol that the body makes.
3. Intensity-Drug	Relation between the severity or extent of patient illness and the use of relevant biologics in therapy.	Acetaminophen given IV may cause vomiting and constipation.
4. Disease-Item	Relation between the disease name and different test items or physiological indicators used for evaluation.	Leukemia is usually identified by analyzing a blood sample through a CBC test.
5. Disease-Method	Relation between the disease name and a treatment method or therapeutic approach.	Belsomra is a dual orexin receptor antagonist (DORA) used to treat insomnia.
6. Disease-Intensity	Relation between the disease and the degree or severity of the patient's condition or infection.	When fighting a common cold, the body may respond with fever, coughing, and sneezing.

4.2.1 Pre-Processing of the Text

The pre-processing of text includes several standard NLP steps are: 1. Data cleaning / text normalization, 2. Stop-word removal, 3. Lemmatization and 4. Tokenization. We applied all these four operations on each transcription to make the data ready for main processing. Table 6 (not shown in full here) demonstrates the impact of the pre-processing pipeline on a sample input text data (record no. 400, specialty “Allergy / Immunology”).

Table 6.
Effect of Pre-Processing Procedures on an Example Text Segment.

medical_ specialty Transcription Tokenised Stemmed stemmed_without_stop

0 Allergy / Immunology Subjective this year old white female presents with complaint of allergies she used to have allergies when she lived in seattle but she thinks they are worse here in the past she has tried claritin and zyrtec both worked for short time but then seeme. [subjective, this, year, old, white, female, presents, with, complaint, of, allergies, she, used, to, have, allergies, when, she, lived, in, seattle, but, she, thinks, they, are, worse, here, in, the, past, she, has, tried, claritin, and, zyrtec, both … [subject, this, year, old, white, femal, present, with, complaint, of, allergi, she, use, to, have, allergi, when, she, live, in, seattl, but, she, think, they, are, wors, here, in, the, past, she, has, tri, claritin, and, zyrtec, both, work, for, sho … Subject year old white femal present complaint allergi use allergi live seattl think wors past tri claritin zyrtec work short time seem lose effect use allegra also use last summer began use two week ago doe appear work veri well use counter spray pre..

1 Bariatrics Past medical history he has difficulty climbing stairs difficulty with airline seats tying shoes used to public seating and lifting objects off the floor he exercises three times a week at home and does cardio he has difficulty walking two blocks or f … [past, medical, history, he, has, difficulty, climbing, stairs, difficulty, with, airline, seats, tying, shoes, used, to, public, seating, and, lifting, objects, off, the, floor, he, exercises, three, times, a, week, at, home, and, does, cardio, he, h … [past, medic, histori, he, has, difficulti, climb, stair, difficulti, with, airlin, seat, tie, shoe, use, to, public, seat, and, lift, object, off, the, floor, he, exercis, three, time, a, week, at, home, and, doe, cardio, he, has, difficulti, walk, t … Past medic histori difficulti climb stair difficulti airlin seat tie shoe use public seat lift object floor exercis three time week home doe cardio difficulti walk two block five flight stair difficulti snore muscl joint pain includ knee pain back pai …

	medical_ specialty	Transcription	Tokenised	Stemmed	stemmed_without_stop
0	Allergy / Immunology	Subjective this year old white female presents with complaint of allergies she used to have allergies when she lived in seattle but she thinks they are worse here in the past she has tried claritin and zyrtec both worked for short time but then seeme.	[subjective, this, year, old, white, female, presents, with, complaint, of, allergies, she, used, to, have, allergies, when, she, lived, in, seattle, but, she, thinks, they, are, worse, here, in, the, past, she, has, tried, claritin, and, zyrtec, both …	[subject, this, year, old, white, femal, present, with, complaint, of, allergi, she, use, to, have, allergi, when, she, live, in, seattl, but, she, think, they, are, wors, here, in, the, past, she, has, tri, claritin, and, zyrtec, both, work, for, sho …	Subject year old white femal present complaint allergi use allergi live seattl think wors past tri claritin zyrtec work short time seem lose effect use allegra also use last summer began use two week ago doe appear work veri well use counter spray pre..
1	Bariatrics	Past medical history he has difficulty climbing stairs difficulty with airline seats tying shoes used to public seating and lifting objects off the floor he exercises three times a week at home and does cardio he has difficulty walking two blocks or f …	[past, medical, history, he, has, difficulty, climbing, stairs, difficulty, with, airline, seats, tying, shoes, used, to, public, seating, and, lifting, objects, off, the, floor, he, exercises, three, times, a, week, at, home, and, does, cardio, he, h …	[past, medic, histori, he, has, difficulti, climb, stair, difficulti, with, airlin, seat, tie, shoe, use, to, public, seat, and, lift, object, off, the, floor, he, exercis, three, time, a, week, at, home, and, doe, cardio, he, has, difficulti, walk, t …	Past medic histori difficulti climb stair difficulti airlin seat tie shoe use public seat lift object floor exercis three time week home doe cardio difficulti walk two block five flight stair difficulti snore muscl joint pain includ knee pain back pai …

After pre-processing, the samples are annotated using the BRAT tool, which exports annotation files in. ann format. Once labeling is complete, we align the .ann files with the corresponding raw text files. During this step, we:

Removed all illegal characters and blank lines

Split long texts into smaller segments using sliding windows and punctuation

Merged entity pairs into the sentence patterns that contain both entities

Filtered out overly long sentences to reduce the noise and improve training stability

These steps produces the final input samples in which each sentence (or segment) contains the clear entity boundaries and associated relation labels, that are suitable for training the proposed relation extraction model.

4.3 Assessment Criteria

For evaluating the relation extraction (RE) model, we use common metrics from the literature (Ayush et al., 2016): precision (P), recall (R), and F1-score (F1).

Precision (P) measures the proportion of the predicted positive cases that are actually correct.

Recall (R) measures the proportion of true positive cases are that the successfully identified by the models.

F1-score (F1) is the harmonic mean of the precision and recall and balances both the measures.

In our settings:

Entities = the total number of true entities or relations in the gold standard

Correct = the number of correctly identified entities or relations

Recognized = the total number of entities or relations predicted by the model

The formulas used are summarized in Table 7.

Table 7.
Evaluation Metrics.

Evaluation Metric Description Formula

Precision (P) Correct predictions among all predicted $P = \frac{C o r r e c t}{R e c o g n i z e d} \times 100$

Recall (R) Correct predictions among all true entities $R = \frac{C o r r e c t}{E n t i t i e s} \times 100$

F1-score (F1) Harmonic mean of precision and recall $F 1 = \frac{2 \times P \times R}{P + R} \times 100$

Evaluation Metric	Description	Formula
Precision (P)	Correct predictions among all predicted	$P = \frac{C o r r e c t}{R e c o g n i z e d} \times 100$
Recall (R)	Correct predictions among all true entities	$R = \frac{C o r r e c t}{E n t i t i e s} \times 100$
F1-score (F1)	Harmonic mean of precision and recall	$F 1 = \frac{2 \times P \times R}{P + R} \times 100$

We report F1, precision, and recall for the relation extraction model on the test set.

4.4 Implementation Details

All experiments are conducted on a machine running with the Windows 10 OS, with Python 3.7.0 and PyTorch 1.8.0. The hardware's are including an NVIDIA GeForce RTX 3080 Ti (16 GB) GPU. Under this configuration, the model trains and runs smoothly (Yang et al., 2016).

Table 8 lists the main hyperparameters used during our proposed model training.

Table 8.
Hyperparameter Settings.

Parameter Tuned Range Final Value

Sequence length 256 256

Train batch size 128 128

Learning rate (LR) [1E-5, 3E-5, 3E-4] 3.00E-05

Dropout 0.1 0.1

Optimizer Adam Adam

Parameter	Tuned Range	Final Value
Sequence length	256	256
Train batch size	128	128
Learning rate (LR)	[1E-5, 3E-5, 3E-4]	3.00E-05
Dropout	0.1	0.1
Optimizer	Adam	Adam

4.5 Results and Analysis

To evaluate the effectiveness of the proposed clinical relation extraction (RE) model, we compared it with the four other baseline methods under the same experimental conditions. The results are summarized shows in the Table 9 and presented in Figure 6. The following baseline models are included in the comparison are:

BiGRU-CRF (Standard model): This model uses Word2vec word embeddings on the i2b2 2010 dataset and BiGRU as the main backbone network. Compared with BiLSTM, BiGRU helps to reduce the risk of overfitting, while capturing the contextual information. A CRF layer is added on top to combine the model the attention vector and label sequences, and to output the ER label corresponding to the input text.

BiGRU-Softmax: This variant is used to check the effect of the CRF layers. Here, the CRF decoding layer is removed, and the final predictions are obtained directly using a Softmax classifier on top of the BiGRU outputs. BiGRU + Softmax is also a commonly used baseline for relation extraction.

BiGRU-Att-CRF: In this model, GloVe word embeddings are used to represent the input text. A multi-layer attention mechanism is added on top of the BiGRU, which is inspired by the BiGRU-CRF baseline design. The goal is to study that how the multiple attention layers, at both word and phrase level, can influence the relation classification. By adjusting the attention weights and slightly reducing the number of parameters, this model aims to improve the detection performance and training accuracy.

BERT-BiGRU-CRF: This method uses the BERT pre-trained language model to provide the contextual embeddings with strong parallelism and rich context dependency. A BiGRU layer is then used to further refine the representations, which are produced by the BERT and to highlight the most informative parts of the text, that followed by a CRF layer for final label prediction.

RoBERTa-BiGRU-MAtt-CRF (Ours): This is the proposed fusion model, which combines the RoBERTa embeddings, BiGRU encoding, multi-head (multi-layer) attention mechanism, and a CRF decoder for the relation extraction.

Figure 6.

Comparative visualization of the performance of each model.

Table 9.

Results of Comparison Between Each Experiment.

Method	P	R	F1
BiGRU-CRF	70.32	76.78	72.74
BiGRU-Softmax	68.04	72.42	70.98
BiGRU-Att-CRF	71.63	77.48	74.81
BERT-BiGRU-CRF	74.21	80.12	76.32
RoBERTa-BiGRU-MAtt-CRF (Ours)	80.32	85.73	81.76

#P - Precision, #R - Recall, #F1 - F1-score.

From the Table 9 and Figure 6, we can see that the proposed RoBERTa-BiGRU-MAtt-CRF model achieves the best performance across all the three-evaluation metrics (P, R, and F1). The detailed observations are as follows:

BiGRU-CRF vs. BiGRU-Softmax: The BiGRU-CRF model gives better results than BiGRU-Softmax. Prominently, the F1-score of BiGRU-CRF (72.74%) is higher than that of the BiGRU-Softmax (70.98%). This shows that adding a CRF layer on top of BiGRU can improves the sequence-level label consistency and leads to better relation prediction than using a simple Softmax classifier. It confirms that the CRF module is important for relation classification in this task.

BiGRU-CRF vs. BiGRU-Att-CRF: When we added the attention mechanism (BiGRU-Att-CRF), both precision and recalls are improved when compared to the standard BiGRU-CRF model. The F1-Score increases from 72.74% to 74.81%. This indicates that the attention mechanism helped the model to focus on context-dependent relational cues. By dynamically adjusting the attention weights at different positions, the model can better capture both local and global dependencies from the sentence and thus becomes more expressive.

RoBERTa-BiGRU-MAtt-CRF (Ours): The proposed fusion model improved the performance by combining the RoBERTa embeddings with BiGRU and multi-head attention. RoBERTa provides richer contextual word representations than traditional embeddings or even BERT in this setup. The multi-layer attention mechanism then refines these representations by highlighting the most relevant parts for relation classification, while the CRF layer ensures the coherent label decisions. As a result, our model achieves the highest precision (80.32%), recall (85.73%), and F1-score (81.76%) against all baseline methods.

These results show that integrating a strong pre-trained language model (RoBERTa), BiGRU encoding, and a multi-head attention + CRF architecture leads to a more powerful and robust clinical relation extraction model.

4.6 Discussion

The proposed model shown the better performance than the baselines on all the three-evaluation metrics: precision, recall, and F1-score. This suggests that our clinical E-RE approach is not only effective for the MTSamples dataset but may also be useful for a broader range of practical applications.

In general, a reliable RE model can help to automatically extract the relations between entities from many kinds of text, such as social media posts, news articles, clinical narratives, and biomedical research papers. With improved precision and recall, the model can support the downstream tasks like large-scale text analysis, topic modeling, and even the parts of sentiment or opinion analysis, where the entity-relation structure is important. In this way, our study provides a useful tool for researchers and practitioners in different fields who need the accurate and efficient relation extraction.

At the same time, the model has some clear limitations also. In its current form, our neural network-based RE method mainly focuses on relations between two entities at a time. However, in real clinical text, one sentence often contains multiple entities and several relations that may interact with each other. The present model does not explicitly handle the higher-order or multi-entity relations. In future work, it will be important to extend the framework, so that it can capture the relationships among multiple entities within the same sentence or document, thereby making the model more complete and more useful in real-world clinical settings.

5 Conclusions

In this study, we proposed a deep learning-based model for clinical relation extraction that combines a RoBERTa encoder, a BiGRU network, a multi-head attention mechanism, and a CRF layer for final E-RE prediction. By replacing the traditional BiLSTM with BiGRU, the model efficiently captures the contextual information from both past and future tokens. The integration of word-level and sentence-level attention mechanism allowed the model to focus on important parts of the text and to learn richer feature representations. This, in turn, improves the performance on the relation classification task. Experimental results are proved that the proposed RoBERTa-BiGRU-MAtt-CRF model achieves higher precision, recall, and F1-score than several strong baselines, confirming its effectiveness for extracting the clinical entity relations from medical transcriptions. However, some limitations are remain in this model. At present, the model does not handle more complex linguistic patterns such as multiple overlapping relations in the same sentence or nested entity structures. In addition, our work assumes that gold-standard entity annotations are already available. For real life clinical applications, an end-to-end system that combines both the named entity recognition (NER) and relation extraction will be necessary. For future work, we planned to extend the model to jointly learn the NER and relation extraction in a single framework. Exploring the use of knowledge-graph-enriched representations (i.e., integrating UMLS or other medical ontologies) to further improve accuracy. Evaluating the model on additional clinical and biomedical datasets to test its robustness and generalizability across different types of clinical narratives.

Footnotes

Consent for Publication

All authors agree with the content of this manuscript and give their explicit consent to submit it for publication.

Author Contributions

Challa Vijaya Madhavi Lakshmi: Conceptualization, Methodology, Software, Data curation, Writing - original draft, Visualization.

Tene Ramakrishnudu: Visualization, Investigation, Supervision, Software, Validation, Writing - review and editing.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Competing Interests

The authors declare that they have no competing interests.

Code Availability

The code used in this study is available from the authors upon reasonable request.

ORCID iD

Vijaya Madhavi Lakshmi Challa

References

Ahmadian

Jalili

(2022). A deep learning based trust- and tag-aware recommender system. Neurocomputing, 488, 557–571. ISSN 0925-2312. https://doi.org/10.1016/j.neucom.2021.11.064

Arora

Lewis

Fan

Kahn

Ré

. (2023). Knowledge retrieval over public and private data. Accessed May 8, 2023, from https://knowledge-nlp.github.io/aaai2023/papers/003-PQA-oral.pdf

Ayush

Michael

(2016). Text mining for precision medicine: Automating disease-mutation relationship extraction from biomedical literature. Journal of the American Medical Informatics Association, 23(4), 766–772. https://doi.org/10.1093/jamia/ocw041

Chaturvedi

Baghershahi

Medya

di Eugenio

(2025). Temporal relation extraction in clinical texts: A span-based graph transformer approach. arXiv preprint arXiv:2503.18085. https://arxiv.org/abs/2503.18085

Chen

Hou

Zhang

(2020, December 8-13). End-to-end emotion-cause pair extraction with graph convolutional network. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (pp.198–207).

Devlin

Chang

M.-W.

Lee

Toutanova

(2019) BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, Vol. 1 (Long and Short Papers) (pp. 4171–4186). Association for Computational Linguistics.

Dongmei

Yang

Dongyuan

Danqiong

(2020). Review of entity relation extraction methods. Journal of Computer Research and Development, 57(7), 1424–1448. https://doi.org/10.7544/issn1000-1239.2020.20190358

Fang

Zhang

Feng

Gao

(2025). GLiM: integrating graph transformer and LLM for document-level biomedical relation extraction with incomplete labeling. Proceedings of the ACL Anthology. https://aclanthology.org/2025.findings-acl.727/

Guo

Zhang

(2019). Attention guided graph convolutional networks for relation extraction. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy (pp. 241–251). Association for Computational Linguistics.

10.

Jain

Singh

Mutharaju

(2023). ReOnto: A neuro-symbolic approach for biomedical relation extraction. arXiv preprint arXiv:2309.01370. https://arxiv.org/abs/2309.01370

11.

Jin

Yang

Qiu

Zhang

(2020). Relation of the relations: A new paradigm of the relation extraction problem. arXiv 2020, arXiv:2006.03719.

12.

Liu

Zhao

Zhang

Xing

(2020). Automatic approach for constructing a knowledge graph of knee osteoarthritis in Chinese. Health Information Science and Systems, 8(1), 12. PMID: 32175080; PMCID: PMC7046853. https://doi.org/10.1007/s13755-020-0102-4

13.

Liang

Peng

Zhang

Yang

(2022). Predicting drug-target interactions via dual-stream graph neural network. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 21(4), 948–958. DOI: https://doi.org/10.1109/TCBB.2022.3204188

14.

Liang

Yang

Y.-C.

Xie

K.-C.

Cao

(2023). PDPChain: A consortium blockchain-based privacy protection scheme for personal data. IEEE Transactions on Reliability, 72(2), 586–598. https://doi.org/10.1109/TR.2022.3190932

15.

Liu

Xin

Gao

Zhu

(2023). Automatic knowledge extraction from Chinese electronic medical records and rheumatoid arthritis knowledge graph construction. Quantitative Imaging in Medicine and Surgery, 13(6), 3873–3890. Epub 2023 May 8. PMID: 37284084; PMCID: PMC10240026. https://doi.org/10.21037/qims-22-1158

16.

Liu

Ott

Goyal

Joshi

Chen

Levy

Lewis

Zettlemoyer

Stoyanov

(2019). RoBERTa: A robustly optimized BERT pretraining approach. ArXiv, abs/1907.11692.

17.

Luo

Yang

Zhang

Wang

Lin

Wang

(2018). An attention-based bilstm-crf approach to document-level chemical named entity recognition. Bioinformatics, 34(8), 1381–1388. https://doi.org/10.1093/bioinformatics/btx761

18.

Mohamed

S. K.

Nováček

Nounu

(2020). Discovering protein drug targets using knowledge graph embeddings. Bioinformatics, 36(2), 603–610. PMID: 31368482. https://doi.org/10.1093/bioinformatics/btz600

19.

Nakashole

Weikum

Suchanek

(2012). PATTY: A taxonomy of relational patterns with semantic types. Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Jeju Island, Korea (pp. 1135–1145). Association for Computational Linguistics.

20.

Nelson

C. A.

Bove

Butte

A. J.

Baranzini

S. E.

(2022). Embedding electronic health records onto a knowledge network recognizes prodromal features of multiple sclerosis and predicts diagnosis. Journal of the American Medical Informatics Association, 29(3), 424–434. PMID: 34915552; PMCID: PMC8800523. https://doi.org/10.1093/jamia/ocab270

21.

Ning

Feng

Zhang

(2022). Relation-aware shared representation learning for cancer prognosis analysis with auxiliary clinical variables and incomplete multi-modality data. IEEE Transactions on Medical Imaging, 41(1), 186–198. Epub 2021 Dec 30. PMID: 34460368. https://doi.org/10.1109/TMI.2021.3108802

22.

Qiu

Chai

Liu

Tian

(2018). Automatic non-taxonomic relation extraction from big data in smart city. IEEE Access, 6, 74854–74864. https://doi.org/10.1109/ACCESS.2018.2881422

23.

Roy

Pan

(2021). Incorporating medical knowledge in BERT for clinical relation extraction. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 5620–5633). Association for Computational Linguistics. https://aclanthology.org/2021.emnlp-main.435.pdf

24.

Scarselli

Gori

Tsoi

A. C.

Hagenbuchner

Monfardini

(2009). The graph neural network model. IEEE Transactions on Neural Networks, 20(1), 61–80. https://doi.org/10.1109/TNN.2008.2005605

25.

Shi

Feng

Liao

Huang

(2020). Penalized multiple distribution selection method for imbalanced data classification. Knowledge-Based Systems, 196, 105833. ISSN 0950-7051. https://doi.org/10.1016/j.knosys.2020.105833

26.

Uzuner

Mailoa

Ryan

Sibanda

(2010). Semantic relations for problem-oriented medical records. Artificial Intelligence in Medicine, 50(2), 63–73. ISSN 0933-3657. https://doi.org/10.1016/j.artmed.2010.05.006

27.

Vaswani

Shazeer

Parmar

Uszkoreit

Jones

Gomez

A. N.

Kaiser

Ł.

Polosukhin

(2017, December 4–9). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA (pp. 6000–6010).

28.

Williams

R. J.

Zipser

(1989). A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2), 270–280. https://doi.org/10.1162/neco.1989.1.2.270

29.

Lin

Xiong

Liang

(2019). UAV Autonomous target search based on deep reinforcement learning in complex disaster scene. IEEE Access, 7, 117227–117245. https://doi.org/10.1109/ACCESS.2019.2933002

30.

Liu

Zhang

(2018). Master clinical medical knowledge at certificated-doctor-level with deep learning model. Nature Communications, 9, 4352. https://doi.org/10.1038/s41467-018-06799-6

31.

Pan

Chen

Long

Zhang

P. S.

(2021). A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1), 4–24. https://doi.org/10.1109/TNNLS.2020.2978386

32.

Hong

Tsujii

Chang

(2012). Feature engineering combined with machine learning and rule-based methods for structured information extraction from narrative clinical discharge summaries. Journal of the American Medical Informatics Association, 19, 824–832. https://doi.org/10.1136/amiajnl-2011-000776

33.

Yang

Guo

Bian

(2021). Clinical relation extraction using transformer-based models. arXiv preprint arXiv:2107.08957. https://arxiv.org/abs/2107.08957

34.

Yang

Dyer

Smola

Hovy

(2016). Hierarchical attention networks for document classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California (pp. 1480–1489). Association for Computational Linguistics.

35.

Yuan

Zhang

Qiu

Lin

Zhang

(2024). Document-level biomedical relation extraction via hierarchical tree graph and relation segmentation module (HTGRS). Bioinformatics, 40(7), btae418. https://academic-oup-com-s.web.bisu.edu.cn/bioinformatics/article/40/7/btae418/7698943

36.

Zelenko

Aone

Richardella

(2003). Kernel methods for relation extraction. Journal of Machine Learning Research, 3, 1083–1106.

37.

Zhang

Chen

Liu

(2017). A review on entity relation extraction. 2017 Second International Conference on Mechanical, Control and Computer Engineering (ICMCCE), Harbin, China (pp. 178–183). https://doi.org/10.1109/ICMCCE.2017.14

Neural Network with Multi-Level Attention for Extraction of Clinical Relationships

Abstract

Background

Methods

Results

Conclusion

Keywords

1 Introduction

2 Related Work

3 Methodology

3.1 Overview of the Entity-Relation Extraction (E-RE) Model

3.1.1 Problem Definition

3.1.2 Model Architecture

3.3.1 Bi-GRU at the Word Level

Table 1. Unary Scores for Relation Labels. Relation Label Unary Score CAUSE 8.3 TREAT 6.9 ASSOCIATED 5.1 NO-RELATION 2.4

3.3.5.1 Running Example

Table 2. Unary Scores Including NO-RELATION. Relation Label Unary Score CAUSE 8.3 TREAT 6.9 ASSOCIATED 5.1 NO-RELATION 2.4

4 Experiments

4.1 Dataset

4.1.2 Annotation Process

Table 8. Hyperparameter Settings. Parameter Tuned Range Final Value Sequence length 256 256 Train batch size 128 128 Learning rate (LR) [1E-5, 3E-5, 3E-4] 3.00E-05 Dropout 0.1 0.1 Optimizer Adam Adam

5 Conclusions

Footnotes

Consent for Publication

Author Contributions

Funding

Declaration of Conflicting Interests

Competing Interests

Code Availability

ORCID iD

References

Table 1.
Unary Scores for Relation Labels.

Relation Label Unary Score

CAUSE 8.3

TREAT 6.9

ASSOCIATED 5.1

NO-RELATION 2.4

Table 2.
Unary Scores Including NO-RELATION.

Relation Label Unary Score

CAUSE 8.3

TREAT 6.9

ASSOCIATED 5.1

NO-RELATION 2.4

Table 8.
Hyperparameter Settings.

Parameter Tuned Range Final Value

Sequence length 256 256

Train batch size 128 128

Learning rate (LR) [1E-5, 3E-5, 3E-4] 3.00E-05

Dropout 0.1 0.1

Optimizer Adam Adam