Named entity recognition of local adverse drug reactions in Xinjiang based on transfer learning

Abstract

For deep learning’s insufficient learning ability of a small amount of data in the Chinese named entity recognition based on deep learning, this paper proposes a named entity recognition of local adverse drug reactions based on Adversarial Transfer Learning, and constructs a neural network model ASAIBC consisting of Adversarial Transfer Learning, Self-Attention, independently recurrent neural network (IndRNN), Bi-directional long short-term memory (BiLSTM) and conditional random field (CRF). However, of the task of Chinese named entity recognition (NER), there are only few open labeled data sets. Therefore, this article introduces Adversarial Transfer Learning network to fully utilize the boundary of Chinese word segmentation tasks (CWS) and NER tasks for information sharing. Plus, the specific information in the CWS is also filtered. Combing with Self-Attention mechanism and IndRNN, this feature’s expression ability is enhanced, thus allowing the model to concern the important information of different entities from different levels. Along with better capture of the dependence relations of long sentences, the recognition ability of the model is further strengthened. As all the results gained from WeiBoNER and MSRA data sets by ASAIBC model are better than traditional algorithms, this paper conducts an experiment on the data set of Xinjiang local named entity recognition of adverse drug reactions (XJADRNER) based on manual labeling, with the accuracy, precision, recall and F-Score value being 98.97%, 91.01%, 90.21% and 90.57% respectively. These experimental results have shown that ASAIBC model can significantly improve the NER performance of local adverse drug reactions in Xinjiang.

Keywords

Transfer learning self-Attention mechanism IndRNN named entity recognition of adverse drug reactions deep learning

1 Introduction

Named Entity Recognition (NER) is an important task in the field of natural language processing (NLP) and can be used for many downstream tasks of NLP, such as relationship extraction, event extraction, machine translation, and question-answering systems. In recent years, with increasing in-depth research into deep learning, great breakthroughs have been made in image processing and NLP of deep learning. Deep learning refers to a series of methods for learning from data within a specified range and obtaining specific features of them. The better generalization ability of deep learning can make NER less dependent on manual annotation to a certain degree.

Adverse drug reactions (ADR) are harmful reactions caused by drug quality issues or improper use of drugs, ranging from side effects, toxic effects (toxic reactions), residual effects (post-effects), allergic reactions, idiosyncratic reactions, double infections and dependence caused by anti-infective drugs, and carcinogenic, teratogenic, mutagenic effects, as well as the effects of the drug itself or drug interactions, which are not related to the purpose of the drug and are not conducive to the patient. At present, adverse drug reactions are becoming the focus of medical R&D (research and development) departments, common users and major medical institutions. Therefore the safety of user’s medication is especially highlighted. Considering the bottleneck of researches into new drugs in medical research and development platforms, it’s impossible to research and develop drugs as well as conduct drug testing for a large number of people in short and limited time. Therefore, researches in adverse drug reactions and grasping known and potential drugs’ adverse reactions are of great value to medical research and development. Furthermore, they can also offer theorectical guidance for bettering the clinical drug research and development, together with provide more safe and effective drugs for patients, playing an important role in raising the whole level of local drugs and speeding up the regional development in Xinjiang.

Adverse drug reaction entities refer to drug names, adverse reactions, and indications, etc. The experimental data were obtained from users’ comments on medication on social network media. And the texts of ADR in social networks were obtained from the first-hand information of patients’ medication, which were characterized by adequacy, immediacy and rapid dissemination. There is a wealth of information about ADR that remains to be found accumulated in social networks, but the web text is written arbitrarily, with a lot of spelling mistakes and grammatical errors, which have brought great challenges to the subsequent text recognition work. Take the writing error in the drug name, for example, “zukamu granule” may be written as “zukam granule”. Internet text often contains a lot of “noise,” such as irregular drug aliases, patient information, and medication information. Therefore, in the task of NER of local ADR in Xinjiang, the data is processed by means of sequence labeling is used to process the data. Entities such as drug names, ADR, locations, in users’ medication reviews are labeled by BIO [1]. And a specific mark processing method is also adopted to filter the information outside the entity. Sequence labeling can remove impure information in a wide range and retain key experimental information to the maximum extent.

In the NER task, most of the researches on Chinese are based on the three research ways: word, character and word combined character. And the results of named entity recognition obtained from these three research methods are different. Among them, He and Wang [2], Liu [3], Li [4] et al. carried out an experimental comparison between character-based mode and word-based mode, showing the better performance of character-based method for NER. Chen [5], Lu [6], Dong [7] et al. also adopted the character-based approach in their latter research. And some other researchers like Zhao [8], Peng [9], He [10] et al. also took the character-based approach and began to use word segmentation information to reinforce learning. Although researchers used deep learning algorithm to continuously improve the accuracy of NER task, the results obtained so far have not reached a relatively high level. For example, the LSTM-based joint training proposed by Peng and Dredze [11] only in WeiboNER task, with the F value only achieving 47.92%. Luo and Yang [12] first started to use word segmentation information, and then use it as an additional feature of the sequence tag. This method has improved NER performance, yet the F value in SighanNER task only reached 89.21%. Therefore, the previous research is too singular. The methods proposed by these scholars are not applicable to data sets in small quantity and the accuracy of the model is not high. For these issues, we proposed a named entity recognition of local adverse drug reactions based on adversarial transfer learning. The difference lies in that we have introduced a transfer learning method to deal with data in small account. Furthermore, we use a joint model to solve the problem that the research is too single. This allows our research to obtain more comprehensive information and extract more obvious data features effectively.

What we research is how to further improve the accuracy of the Xinjiang local adverse drug reactions named entity recognition. Therefore, on the basis of these researches, this paper proposes an Adversarial Transfer Learning framework, using the similar processing steps of Chinese word segmentation tasks (CWS) task and entity recognition task of ADR to share multi-task study, and fully make use of the common boundary information. At the same time, it also prevents the special characteristics of CWS from affecting the recognition of Chinese named entities. But the application of traditional convolutional neural network in text lacks the acquisition of context information and the dependence between words, failing to achieve good effect of feature extraction, therefore, self-Attention mechanism and IndRNN are introduced. The experimental results show that the Adversarial Transfer Learning based self-Attention, IndRNN, Bi-directional long short-Term Memory (BiLSTM) and conditional random field (CRF) (ASAIBC) model has achieved good results on entity identification task of adverse drug reactions and public data set WeiBoNER and MSRA data set in Xinjiang.

To sum up, the main contributions of this paper are as follows:

A kind of Adversarial Transfer Learning framework is proposed, using the common boundary information in the named entity recognition task of local ADR and the CWS task to conduct joint training, so as to realize task sharing and the transfer based on examples, as well as reduce the cost of manual labeling, and improve the accuracy of the experiment.

Introducing the self-Attention into the ASAIBC model can capture the overall dependence of the whole sentence and learn the internal structural features of the sentence.

In order to learn the long-term dependence of sentences better, IndRNN is introduced. According to current researches, this is the first time to apply IndRNN into Xinjiang NER task.

The proposed ASAIBC model was tested on the Xinjiang local named entity recognition of ADR (XJADRNER) data set and obtained the experimental results F-Score as 90.75%, which was verified on the Chinese public data set WeiBoNER and MSRA. The experimental results showed that the results obtained by the ASAIBC model were all superior to traditional algorithms.

The remaining of this paper is structured as follows: the next section surveys related work on named entity recognition of adverse drug reactions. Section 3 describes the techniques employed. Sections 4 describes the experimental results. Lastly, some conclusive remarks are given in Section 5.

2 Related work

2.1 Named entity recognition of adverse drug reactions

In the named entity recognition task, two processing methods are generally adopted: linear statistical model and deep learning method. Linear statistical models include support vector machines [13], hidden Markov models [14] and conditional random fields. For deep learning methods, Hammerton et al. [15] were the first to adopt neural networks in NER tasks, while Collobert [16] was earlier combined with deep learning algorithms applied to NER. Turian [17] combines pre-trained word embedding in feature processing and traditional CRF methods to be applied to NER tasks. Lample [18] uses two Bi-LSTM to extract features, one of which extracts the features of each word in the corpus, while the other Bi-LSTM extracts the word-level features, and finally uses the CRF layer to carry out the model of sequence prediction. Ma [19] proposed a model similar to Lample’s, but different in that convolutional neural network is used for character-level feature extraction in model selection. In 2011, Collobert et al. [20] came up with a CNN-CRF structure, and got more precise results. In 2015, Santos et al. [21] proposed using character CNN to enhance the CNN-CRF model. In 2017, Strubell et al. [22] proposed applying Improved Dilated Convolutional Neural Network (IDCNN-CRF) into NER task. This new neural network can extract sequence information efficiently and accurately. Peters [23] pre-trained a character language model [24] to generate word context representation to enhance word representation, and in 2018 Devlin et al. [25] used Bert, a pre-trained model with strong semantic expressive ability which proposed in Google, to train the language model.

At present, the exponential growth of biomedical literature along with the rapid development of social media, has generated unlimited potential research resources. Foreign researchers mainly focus on biomedical texts [26] and Twitter messages [27]. At the same time, this study is based on Chinese social media, such as Baidu Tieba, Blog, user medication websites to crawl users’ medication comments, and the obtained user medication comments are manually marked by BIO for experimental data.

Entity recognition of ADR uses deep learning methods to extract adverse reaction information from users’ medication reviews. For example, in the case that “Because of having a cold and the following runny nose, the doctor prescribed the Zukamu granules. Yet, why the child has no spirit after taking all of the medicine?” The drug adverse reaction in the BIO sequence is “no spirit”. When the BIO sequence is used for annotation, the result of corpus tagging is shown in Table 1.

Table 1
Corpus tagging for entity recognition of adverse drug reactions

In Table 1, we can see that the corpus contains a large number of invalid information, which are labeled with the mark “O”. For ADR entity, we respectively tag them with mark “B-ADR” and “I-ADR”. Then, ASAIBC model is adopted to learn a large number of training sets. Finally, the accuracy of the trained model is verified through the test set.

2.2 Transfer learning

Transfer learning refers to a learning process in which the models learned in the old field are applied to the new field based on the similarity between data, tasks, or models. Since its inception, transfer learning has been widely used in fields like computer vision [28], NLP [29], text classification [30]. According to the classification of learning method forms, the earliest definition was given in authoritative reviews concerning transfer learning [31]. It divides transfer learning methods into the following four categories: Instance based Transfer Learning, Feature based Transfer Learning, Model based Transfer Learning, and Relation based Transfer Learning. According to the characteristics of CWS task and Xinjiang local named entity recognition of ADR (XJADRNER), this study adopted the method of the instance based transfer learning, and added the deep adversarial network on the basis of transfer learning. In summary, an Adversarial Transfer Learning framework was proposed to conduct the NER task of Xinjiang local adverse drug reactions.

In the framework of Adversarial Transfer Learning, the similar processing method of labeling data of CWS task and XJADRNER task is used to make full use of the common boundary information, and at the same time to prevent the influence of the characteristic characteristics of CWS task on XJADRNER, in which the CWS task and XJADRNER task are used to deal with, for example, “why is the child spiritless after taking medicine?”, the comparison of its annotation methods is shown in Fig. 1.

Fig. 1

Comparison of annotation methods between XJADRNER and CWS.

In Fig. 1, when the CWS and XJADRNER tasks deal with the corpus “What reason is the child no spirit after taking medicine”, the entity “child” and “no spirit” are annotated. For non-entity words, they are marked with “O” and “S” respectively, therefore it can make use of rich knowledge of CWS on joint training mission in XJADRNER task to solve the cost of manual annotation, and to improve the efficiency of Xinjiang local entity recognition of ADR. In the defined framework of Adversarial Transfer Learning, both the source domain and the target domain are composed of two parts: the public part and the private part. The public part can learn the public characteristics of the XJADRNER and CWS tasks, and the private part is used to keep the separate features of each domain. The Adversarial Transfer Learning loss function is defined as:

$Loss = L_{task} + α L_{difference} + β L_{same} + γ L_{refactor}$ (1)

Where in L_task, L_difference, L_same, L_refactor respectively represent the conventional training loss of the network, the difference loss of the public part and the private part, the similarity loss of the public part of the source domain and the target domain, and the reconstruction loss. They are introduced to ensure that the private part can still play a role in the learning goal.

In this study, the constructed schematic diagram of Adversarial Transfer Learning framework is shown in Fig. 2, in which embedding layer for multitasking learning, encoding the embedding of XJADRNER, CWS two parts at the same time, and then loaded adversarial network to get loss (adversarial_loss), finally through the normalized processing can get the final result of the adversarial transfer learning’s outputs. These obtained outputs are used as the input of the ASAIBC model, iterating continuously through the current loss to train the optimal model.

Fig. 2

Schematic diagram of building adversarial transfer learning framework.

3 Methods

In this paper, the adversarial network is first introduced to realize the sharing of boundary information between CWS and XJADRNER. Due to the limited number of manual annotation, the introduction of the adversarial network can not only reduce the cost of manual annotation, but also expand the amount of experimental data, which is also a discovery of the application of transfer learning in the field of NLP. Through the processing of Adversarial Transfer Learning framework we can get two parts: public and private part. Public part refers to XJADRNER and public characteristics of CWS tasks, and the data of this part can be directly combined with the boundary of public information, which expands the experimental data of the entity identification task of local adverse drug reactions in Xinjiang. While the private part was used to maintain the independent characteristics of each field, because some information could not be shared in the CWS task, the private part of the data is excluded from the confrontation processing. Then, the self-Attention mechanism makes full use of the semantic information within texts and input them into BiLSTM to extract local features of the text. Besides, IndRNN is combined to mine in-depth contextual information to effectively complete the entity identification task of local adverse drug reactions in Xinjiang. As shown in Fig. 3, ASAIBC model is mainly composed of six parts:

Fig. 3

ASAIBC neural network model framework.

Input layer of character embedding: For an experimental corpus S with length N, there is {s₁, s₂, s₃... ... s_n}, which contains M labeled entity labels {e₁, e₂, e₃... ... e_m}. The input layer can convert the experimental corpus S into a matrix X, which includes word embedding, position embedding and BIO tag embedding for each word. The martix X is shown in the input layer in Fig. 3.

Layer of Adversarial Transfer Learning: Firstly, by inputting CWS and XJADRNER into the adversarial network for word embedding training, we can get their private parts respectively as well as their public parts. Then we conduct the feature extraction of these public and private parts. In the Adversarial Transfer Learning layer, the biggest innovation is to combine the task of entity identification of local adverse drug reactions in Xinjiang with the similar labeling method of CWS data set, therefore there is mutual reference and integration in data processing, which is also the main reason for the introduction of the Adversarial Transfer Learning framework.

Joint processing layer: Input the word embedding obtained from the Adversarial Transfer Learning layer into the self-Attention and IndRNN, which w_i is the I th word vector. Self-Attention can capture the important information in the input sequence, IndRNN can learn long term text dependencies, and dig text context characteristics deeper, then the features obtained in the self-Attention and IndRNN is input into BiLSTM to mine sequence data information.

CRF layer: CRF is a probabilistic undirect graphical model [32], which calculates the optimal joint probability in a sequence and optimizes the whole sequence instead of splicing the optimal solution at each moment. In this respect, CRF is better than LSTM. Therefore, the output of joint processing layer is decoded by CRF in this paper to obtain the globally optimal annotation sequence.

Fully connected layer: The fully connected layer is usually placed at the end of the network and used to synthesize all information. Features representation gained in the CRF layer is mapped to the sample mark space. In this paper, the full connection layer adopts Softmax as the activation function. The schematic diagram of the fully connected layer is shown in Fig. 4.

Output layer: This paper selects the Softmax classifier to classify and determine the sequence of sequence tags of the predicted and the actual labels, thus the accuracy of the local entity recognition of adverse drug reactions in Xinjiang can be obtained. In the predicted results, ASAIBC model has predicted the labels B-, I- and O corresponding to their entities. At the same time, a loss rate is also generated to stand for the ratio of the predicted labels to the actual labels.

Fig. 4

The schematic diagram of the fully connected layer.

3.1 Overview

The experimental data used in this paper is the XJADRNER obtained by manual BIO sequence labeling method. In order to complete NER of ADR in Xinjiang better, this paper introduced Adversarial Transfer Learning framework. First of all, a basic network was trained on the CWS task, and then the learned features were readjusted and transferred to the trained network of the XJADRNER task for training target data sets and tasks. Then the obtained character embedding vectors were input to the self-Attention mechanism, so as to make the model focused on entity important information from different levels. Besides, IndRNN model was added to study long-term dependence of sentences and mine sequence information. Then the obtained feature vectors were merged with and the other feature vectors trained by BiLSTM and then input to optimize the whole sequence CRF layer, eventually getting predict entity sequence tags. Then it entered the full connection layer for processing, and the accuracy rate of XJADRNER was obtained after normalization by using Softmax as activation function.

3.2 Embedding layer

The embedding layer can be considered as a fully connected layer. The parameter of the full connection layer can be constructed as a matrix, and then the one-hot coding of each word is multiplied by the coefficient matrix of the full connection layer. A new vector is obtained by reducing or increasing the dimension, which is the word vector. Finally, the word is mapped to the mapping matrix of the vector.

Word embedding (distributed word representation) [33] refers to the ability to capture semantic and syntactic information of words from a large unlabeled corpus. Compared with the BOW (bag-of-word), the word vector has the characteristics like low dimension and density. Most of the corpus obtained by XJADRNER task is from user comments, which are authentic but inevitably contain some noisy words. Therefore, considering the characteristics of the data sets of XJADRNER task, word vectors can obtain better experimental results. In recent years, Word2vec [34] and GloVe [35] have been widely used in various fields of NLP, such as NER, text classification, machine translation, etc., which tend to use Word2vec model. Thus, Word2vec is used in this study to obtain the word vector. Firstly, whether a word is a named entity is related to its context, therefore language model is used to train the word vector on large-scale unlabeled corpus. Word2vec has two implementation models: Skip-gram and CBOW [36]. Skip-gramm can predict the words of the context by current words, while the CBOW is precisely the opposite. In the NER task, CBOW is generally used for this implementation. Since in NER, CBOW plays a smaller role in predicting the contextual information by current words, but bigger role in predicting the current words based on contexts, making for entity identification.

Therefore, this paper adopts CBOW model to train the corpus, the first step of our proposed model is to map discrete characters into the distributed representations. For a given Chinese sentence s ={ s₁, s₂, s₃ …… s_n } from XJADRNER dataset, we look up embedding vector from pre-trained embedding matrix for each character, denoted as $w_{c}^{t}$ . Additionally, each character in a word also has an impact on the recognition of named entities. These character-level features also contain rich structural features of entities, such as prefixes and suffixes. In some texts with many interference, users are more likely to use abbreviations or simplified words to describe entities in the their comments on major drug websites. These characteristics are helpful to improve the effect of NER. Then, Word2vec is used to get the character level vector for each word and denoted as $w_{c}^{v}$ . Finally, the word embedding and its corresponding character-level vector were concatenated and splited, denoted as $w_{c} = [w_{c}^{t}, w_{c}^{v}]$ . The matrix vector obtained was used as the input of ASAIBC model to extract and integrate features, and the model was used to continuously learn features. The embedding process pseudo code is shown in Table 2.

Table 2
Load embedding

1. file = open(’. /cws.utf8’,’r’,encoding=’utf-8’)

2. file.readline()

3. while True:

4. nowread = file.readline()

5. if nowread==”:

6. break

7. else:

8. nowread = nowread.strip().split()

9. embeddingload[nowread [0]] = len(embeddingload)

10. id[len(id)] = nowread [0]

11. nowread = nowread [1:]

12. nowread = [float(i) for i in nowread]

13. vector.append(nowread)

14. file.close()

3.3 Adversarial transfer learning framework

The adversarial network first achieved great success in computer vision [37, 38]. In NLP, adversarial network training learning has been applied to domain adaptability [39, 40] and multi-tasking [41, 42] processing. Different from these studies, the XJADRNER task and CWS task are jointly trained by the adversarial network in this paper, aiming to extract the shared boundary information from the CWS task and filter out the specific information in the CWS. This paper adopts instance-based transfer learning and adversarial network. The core idea is to set XJADRNER as the target domain and CWS as the source domain. The purpose is to calculate the similarity between XJADRNER sample and CWS sample, Then adjusting the weight of CWS can update the CWS sample. At the same time, a threshold value is set. If the sample similarity is lower than the threshold, the sample is discarded and a new data set CWS1 is generated after updating. The final experimental training set Train = {XJADRNER, CWS1}. The ultimate goal is to make the training data set meet the training requirements of the ASAIBC model in terms of size and quality. The samples here refer to the named entities in XJADRNER and CWS tasks. The sample similarity refers to the similarity betweent the two named entities in the composition and structure, which is mainly used to evaluate the contribution of CWS samples to XJADRNER samples.

In this paper, the sample similarity is mainly processed by two methods: entity similarity and part-of-speech similarity, which are calculated in units of words. Entity similarity is to calculate the sample similarity at the semantic level, which reflects the semantic information of entities well. Part-of-speech similarity considers considering the application of part-of-speech tagging to the composition of entities as the comprising features of some entities is often accompanied by similar part-of-speech expressions.

For the two experimental corpora C1 and C2, C1 = {t1, t2, t3 . . . Tm}, C2 = {z1, z2, z3 . . . zn}. Where t1 and z1 represent the words in C1 and C2 respectively, and m and n denote the length of corpus C1 and C2 respectively. Calculation methods of the two kinds of similarity are as follows:

Entity similarity refers to the ratio of the maximum number of identical words in two different entities to the maximum number of entities under the premise of the same sequence. The calculation formula is: ${Similar}_{entity} = s / \max (m, n)$ (2)

Where s represents the number of identical entities in C1 and C2 under the premise of the same sequence.

Part-of-speech similarity refers to the ratio of the maximum number of parts of speech in two different entities to the maximum number of entities under the premise of the same sequence. The calculation formula is: ${Similar}_{part} = p / \max (m, n)$ (3)

Where p represents the number of parts of speech of the same entity in C1 and C2 under the premise of the same sequence.

Meanwhile, the sample similarity obtained can be used to calculate the similarity between source domain CWS and target domain XJADRNER: $\begin{matrix} S_{e} = \max \\ {{Similar}_{entity 1}, {Similar}_{entity 2}, \dots \dots {Similar}_{entitym}} \end{matrix}$ (4) $\begin{matrix} S_{p} = \max \\ {{Similar}_{part 1}, {Similar}_{part 2}, \dots \dots {Similar}_{partn}} \end{matrix}$ (5)

For the source domain CWS and the target domain XJADRNER, if the same sample number of CWS and XJADRNER is m_c and m_x, when after the sample similarity of the source domain and the target domain is obtained, according to the set threshold δ, when the sample similarity of a sample in the source domain CWS is lower than the threshold δ, the sample is discarded directly, and the weight w_e and w_p of the entity similarity and part-of-speech similarity of the source domain and the target domain can be obtained at the same time: $w_{e} = m_{c} * m_{x} / {(2 - S_{e})}^{2}$ (6) $w_{p} = m_{c} * m_{x} / {(2 - S_{p})}^{2}$ (7)

Finally, entity similarity and part-of-speech similarity weights of source domain and target domain are integrated, and the final weight of the sample can be obtained as the follow: $w = α * w_{e} + β * w_{p}$ (8)

Among which, α, β is the weight coefficient, and α + β = 1, α, β reflects the degree of different factors’ influence on the results, which can better improve the reliability of the Adversarial Transfer Learning framework.

In summary, the idea of the Adversarial Transfer Learning framework is to obtain the weights of the source domain and the target domain according to the entity similarity and part-of-speech similarity of the source domain. Selective discarding is carried out according to the weight size of specific samples. On the one hand, the contribution of the source domain to the target domain can be maximized; on the other hand, the experimental data can be expanded to make up for the lack of the deep learning model to learn a small amount of data. The adversarial transfer learning process pseudo code is shown in Table 3.

Table 3

Adversarial transfer learning

1. overturn = ADR_model. Overturn ()

2. get_feature = overturn (get_feature)

3. if self.has_train:

4. get_feature = tf.nn.dropout(get_feature,self. prob)

5. adver_one=tf.get_variable(name=’adver_one’,shape=,dtype=tf.float32,initializer=tf.contrib.layers.xavier_initializer())

6. adver_two=tf.get_variable(name=’adver_two,shape=[self.task_num],dtype=tf.float32,initializer=tf.contrib.layers.xavier_initializer())

7. train_log = tf.nn.xw_plus_b(get_feature, adver_one,b_adv)

8. adver=tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(train_log = train_log, labels = self.task_label))

9. return adver

3.4 Joint processing layer

In the joint processing layer, the BiLSTM model is used to mine the input sequence data information to ensure that the over-fitting phenomenon is not easy to occur in the training process. The introduction of self-Attention can capture the important information in the current input sequence, and IndRNN can learn the long-term dependencies in the input information and dig deeper contextual characteristics.

Self- Attention: self-Attention mechanism, which can focus on the important information in the input sequence. Its internal implementation mechanism often adopts the additive Attention function and the dot product Attention function. In this joint processing layer, let the input sequence matrix information obtained by self-Attention from the counter migration layer be M, and there are three weight matrices $W_{j}^{Q}, W_{j}^{K}, W_{j}^{V}$ obtained by training in self-Attention, the matrix M and $W_{j}^{Q}, W_{j}^{K}, W_{j}^{V}$ are mapped for m times, and Q, K and V can be obtained for each mapping. After the mapping is completed, m-outputs can be finally obtained. Then the output is spliced, and the last attention map is performed on the matrix after spliced to get the final self-attention output matrix Z, and then Z is used as the input of IndRNN for further training. In this model, the attention function adopts the method of dot product, and the final calculation formula of the attention score is as follows: $Attention (Q, K, V) = softmax ((Q * K^{T}) / \sqrt{d})$ (9)

Among which, $1 / \sqrt{d}$ represents the stretch factor, which is added to prevent model training speed from slowing down and gradient vanishing.

IndRNN: independently recurrent neural network (IndRNN) is a new network structure proposed by Li et al., which can not only solve the gradient disappearance and gradient explosion problems existing in traditional RNN, but also learn long-term dependence relationships. In addition, with the help of ReLU and other unsaturated activation functions, IndRNN became more robust after being trained, and it was possible to build a deeper network than the existing RNN by stacking multiple layers of IndRNN [43]. The structure diagram of IndRNN model is shown in Fig. 5.

Fig. 5

IndRNN model framework.

Among them, Weight and Recurrent+ReLU represent the processing steps of input at each time step. ReLU is an activation function. In order to speed up training, Batch Normalization is inserted after each layer.

Then a deep IndRNN network can be established, whose formula can be expressed as: $h_{t} = σ (Weight * x_{t} + μ ⊙ h_{t - 1} + b)$ (10)

Where the cyclic weight μ is a vector and ⊙ represents the Hadamard product (the product of the corresponding elements). Each neuron in the same layer is not connected to other neurons. But the neurons can be connected by superimposing two or more layers of IndRNN. And its simple structure makes it easy to be added to different network structures.

BiLSTM: BiLSTM is a new neural network structure proposed by Augenstein et al. to improve RNN and LSTM. The BiLSTM has five steps: the first step is to calculate the forgetting gate. Selecting the information that is to be forgotten. Input the hidden layer state h_t-1 and the input word X_t at the current moment. What is outputted is the forgotten gate value f_t. The specific formula is as follows: $f_{t} = σ (W_{f} \times [h_{t - 1}, X_{t}] + b_{f})$ (11)

The second step is to calculate the memory gate. Selecting the information that needs to be memorized. Input the hidden layer state h_t-1 at the previous moment and the input word X_t at the current moment. The outputs are the value of the memory gate i_t as well as the temporary cell State $\tilde{C_{t}}$ , the specific formula involved are as follows: $i_{t} = σ (W_{i} \times [h_{t - 1}, X_{t}] + b_{i})$ (12) $\tilde{C_{t}} = tan h (W_{c} \times [h_{t - 1}, X_{t}] + b_{c})$ (13)

The third step is to calculate the state of the cell at the current moment. The inputs are the memory gate i_t, the value of the forgetting gate f_t, the temporary cell state $\tilde{C_{t}}$ , the previous cell state C_t-1. The output is cell state C_t currently. The specific formula is as follows: $C_{t} = f_{t} \times C_{t - 1} + i_{t} \times \tilde{C_{t}}$ (14)

The fourth step is to calculate the output gate and the current hidden state. Inputting the hidden layer state h_t-1 at the previous moment, the input word X_t and the cell state C_t at the current moment. The outputs are the value of the output gate o_t and the hidden layer state h_t. Just as follows: $o_{t} = σ (W_{o} \times [h_{t - 1}, X_{t}] + b_{o})$ (15) $h_{t} = o_{t} \times tan h (C_{t})$ (16)

Finally, the sequence {h₀, h₁, h₂, h₃, …… h_n-1} with the same length as the sentence can be obtained. In the coding layer of LSTM, word2vec can usually be used to train the word vector, normally adopting one-hotting coding. The difference between BiLSTM and traditional LSTM is that the latter can only obtain information before the current word, while BiLSTM can acquire the context semantics and grammatical information of the current word. In XJADRNER task, how to find the characteristics of the contextual information corresponding to entity information is particularly important.

The basic structure of the BiLSTM model is shown in Fig. 6. In the figure, BiLSTM is used to learn the sentence sequence {w (1) , w (2) , … w (n)} obtained by Adversarial Transfer Learning. Cir and Cil represent the right and left implicit unit states respectively, and h1l and hnr decibels represent the output of the left and right hidden layers respectively. Learning from the right layer first and we can get Cnr. When it comes to the left one, C1l can be gained. Through the end-to-end connection of the hidden vector, the final output vector C can be obtained which contains the overall information of the text and enhances the expression performance of features.

Fig. 6

BiLSTM layer.

3.5 CRF layer

In this paper, BIO sequence annotation method is adopted, and considering the correlation between adjacent tags, CRF [44] is selected to obtain the global optimal output sequence of a given sequence.

Through the joint processing layer, the output sequence O = {o₁, o₂, o₃ …… o_n} can be obtained, which o_n represents the n-th word vector of the output sequence O. If Y = {y₁, y₂, y₃ …… y_n} represents the corresponding label sequence of the output sequence O, the CRF probability model can be obtained: $\begin{matrix} score (Y | O; Weight, b) = \prod_{i = 1}^{n} ρ_{i} (Y_{i - 1}, Y_{i}, O) / \\ \sum_{Y^{'} \in Y (O)} \prod_{i = 1}^{n} ρ_{i} (Y_{i - 1}^{'}, Y_{i}^{'}, O) \end{matrix}$ (17)

Where Weight, b represents the weight vector and deviation transferred from label Y’ to label Y respectively. In the training process, the maximum likelihood estimation principle is used to optimize the logarithmic likelihood function as follows: $Log (Weight, b) = \sum_{i} log score (Y | O; Weight, b)$ (18)

According to the maximum likelihood estimation principle, the maximized logarithmic likelihood function is the maximized CRF conditional probability model, that is, to optimize objective function as: $Y^{*} = argmax {Y ε Y (O)} score (Y | O; Weight, b)$ (19)

3.6 ASAIBC model

In this paper, BIO sequence annotation method is adopted, and considering the correlation between adjacent tags, CRF is selected to obtain the global optimal output sequence of a given sequence.

According to Fig. 3, the XJADRNER word vector and CWS word vector are inputted in the input layer. Inputting the word vectors of the two tasks into the Adversarial Transfer Learning framework, and using the trained adversary network to learn the CWS word vectors, and the private part of the CWS task is discarded. In addition, the common part of CWS task is extracted, and finally the common part and XJADRNER word vector fusion are input into the joint processing layer, which is denoted as B, and the attention weight is calculated firstly, and the formula is as follows: $C_{i} = tanf (w_{f} * f_{i} + b_{f})$ (20) $A_{i} = \exp (C_{i}) / \sum_{o = 1}^{L} \exp (C_{o})$ (21)

In the above formula, f_i represents the i-th feature (f_i ∈ B); w_f means the weight; b_frepresents the offset, and A_i represents f_i’s attention weight, which can lead to the output att of the attention mechanism: $att = \sum_{k = 1}^{L} A_{k} * f_{k}$ (22)

As the input of IndRNN, the matrix output of the obtained attention mechanism can be used to learn f_i’s long-term dependency relationship and dig deeper contextual features. After IndRNN, the eigenvector graph can be obtained, where: $Ft = f (fw * att + fb)$ (23)

In the obtained eigenvector graph Ft, fw represents the weight matrix, att represents the output result of the self-Attentional mechanism and the bias of the f function is represented by fb.

At the same time, inputting the word vector B obtained in the adversarial transfer layer into the BiLSTM layer, we can get $\underset{B_{t}}{\to} = shared_cell_fw (\underset{B_{t - 1}}{\to}, f_{i})$ (24)

$\underset{B_{t}}{\leftarrow} = shared_cell_bw (\underset{B_{t + 1}}{\leftarrow}, f_{i})$ (25) $B_{t} = \underset{B_{t}}{\to} \oplus \underset{B_{t}}{\leftarrow}$ (26)

In the above formula, f_i stands for the i-th word vector, where f_i ∈ B, and BiLSTM is used to extract the feature and then directly join the vectors obtained from the two directions.

Taking the output result Ft of Self-Attention+IndRNN layer and the result B_t of BiLSTM layer as the input of CRF, we can get: $M_{crf} = M (Ft, B_{t})$ (27)

$\begin{matrix} p (Y | M_{crf}; Weight, b) = \prod_{i = 1}^{n} ρ_{i} (Y_{i - 1}, Y_{i}, M_{crf}) / \\ \sum_{Y^{'} \in Y (O)} \prod_{i = 1}^{n} ρ_{i} (Y_{i - 1}^{'}, Y_{i}^{'}, M_{crf}) \end{matrix}$ (28)

Where, M_crf represents the input of the CRF layer, Y represents the prediction label corresponding to M_crf, and Weight, b represents the weight and deviation respectively. Finally, the output of CRF layer is inputted to the fully connected layer. And the sigmoid activation function is used to capture the nonlinear characteristics of the input vector and enhance the expression ability of ASAIBC model. Among which: $sigmoid (x) = 1 / ((1 + \exp (- x)))$ (29)

Connecting all the features in the full connection layer, extracting the correlation among these features after nonlinear changes of the previously extracted features, finally mapping to the output space and sending the output value to the softmax classifier. In the classification results, the predicted labels can be obtained: $result = softmax (w_{i} * p + b_{i})$ (30)

Where, w_irepresents the weight matrix in the fully connected layer; b_i stands for the offset. The chain rule in bp backpropagation algorithm is used for updating and iterating the weight until the error is minimized.

4 Experiments

4.1 Dataset

The data in this paper are from the major network social media of Xinjiang local medicine. The comments on medication on social media reflect the real experience of users and can improve the effectiveness of the experiment. The experimental data in this paper were collected from various major medication consultation websites, blogs, microblogs, baidu tieba, news journals and other social media sites. A total of 5392 user comments on medication were selected. Under the guidance of experts, 5392 corpus were marked as XJADRNER data set. Among the 5392 user medication comments, 1079 were positive cases of adverse drug reactions (including ADR entities), and 4313 were negative cases without adverse drug reactions. In the experiment, the length of each corpus was about 15 Chinese characters. In the process of annotation, according to the guidance of experts, BIO sequence annotation method is adopted, in which B represents the beginning of entity, I represents the non-beginning part of entity, and O represents the no-entity, so as to ensure that each corpus annotated can provide sufficient theoretical evidence basis for experimental results. At the same time, the experimental corpus of XJADRNER is continuously expanding and enriching. We set the ratio of training set, validation set, and the test set as 0.6 : 0.2 : 0.2 when the best experimental effect can be achieved. The validation set is used for cross-validation. Finally, we verify the model on the test set to guarantee the model’s generalization ability.

In order to verify the validity of the proposed ASAIBC model based on transfer learning, two public data sets WeiboNER data set and MSRA data set are introduced. The WeiboNER data set contains the following four types of entities: person names, place names, organization names and geopolitical entity (region). And the MSRA data set contains three types of entities: person names, institution names and place names. The MSRA data set is a NER data set published by Microsoft, which includes training set and test set. The training set contains 46.4 k sentences and 2169.9 k words. The test set consisted of consists of 4.4k sentences and 172.6k words. At the same time, WeiboNER data set and MSRA data set are the two most widely used public data sets in Chinese NER, which can provide experimental data basis for the validity of ASAIBC model. The WeiboNER, MSRA data sets and the hand-annotated XJADRNER datasets are detailed in Table 4.

Table 4
Statistics of entities in MSRA and WeiboNER datasets

MSRA WeiboNER

Person Name Place Name Organization Name Total Total

Training set 21635 38965 23562 84142 1350

Test set 2015 2362 958 5335 270

				MSRA	WeiboNER
Training set	21635	38965	23562	84142	1350
Test set	2015	2362	958	5335	270

4.2 Model hyperparameter

This paper adopts the ASAIBC neural network based on transference learning. In the experiment, the original vector matrix and weight matrix are combined to get the attention matrix, which is taken as the input of IndRNN. After that, the obtained semantic features and BiLSTM neural network were input to the CRF layer, and the final experimental results were obtained through the full connection layer. After continuous training and adjustment of the experimental model, when the experimental parameters are set as shown in Table 5, the identification effect is the best. The experimental parameter adjustment method adopted in this paper is as follows: firstly, the parameters are randomly initialized; then, the objective function is minimized by gradient descent.

Table 5
Experimental parameter setting

Parameter P_name Value

en Epochs_num 260

aw Atr_weight 0.06

bs Batch_size 64

ed Embedding_dims 100

fn Task_num 2

ld Lstm_dim 120

ctn Cws_tags_num 4

atn Adr_tags_num 7

Parameter	P_name	Value
en	Epochs_num	260
aw	Atr_weight	0.06
bs	Batch_size	64
ed	Embedding_dims	100
fn	Task_num	2
ld	Lstm_dim	120
ctn	Cws_tags_num	4
atn	Adr_tags_num	7

In order to verify the effectiveness of the experimental parameters, a learning curve has be used in the process of parameter selection. The experimental results show that the experimental parameters can ensure the reliability of the model. The learning curve as shown in Fig. 7.

Fig. 7

The learning curve.

4.3 Evaluation index

In order to evaluate the effect of entity identification of local drug adverse reactions based on transfer learning in Xinjiang, the Recall (R), Precision (P), F-Measure(F) value and Accurary are used in this paper as indicators of model training effects.

The Accuracy is the ratio of the samples correctly classified by the classifier to the total samples in the data set given by the experiment, which can be used to reduce losses. The Precision refers to the ration of the item correctly identified to that totally identified of the ADR, which can further reflect the degree of accuracy in processing. The Recall refers to the ratio between the item of the correctly identified entity of adverse drug reaction and the item of the entity of adverse drug reaction that should be identified, which can be used to judge the completeness of the model, F value is a comprehensive index used to measure precision and recall, which can be expressed as: $F = 2 PR / (P + R)$ (31)

At the meantime, Precision, Accuracy, Recall and F value have high tolerance rate in model evaluation, which are also the mainstream model evaluation indicators at present.

4.4 Experimental results and analysis

In Chinese NER task, most scholars research the validity of the model based on the MSRA data set and WeiboNER data set. In order to verify the reliability of the ASIBC model, it was trained on the WeiboNER data set and MSRA data set, which obtained good experimental results, specifically shown in Table 6.

Table 6
Performance comparison between different models trained on WeiboNER dataset and MSRA dataset

MSRA WeiboNER

Models P (%) R (%) F (%) Models P (%) R (%) F (%)

CRF × × × × 56.59 25.28 34.95

CRF+word × × × × 64.68 25.63 36.85

CRF+character × × × × 57.89 34.02 42.86

CRF+character+position × × × × 57.26 34.53 43.09

BiLSTM+CRF(BC) 88.56 87.56 88.06 BC 58.89 44.89 50.95

BC+Transfer(BCT) 89.96 89.25 89.60 BCT 59.89 46.25 52.19

BCT+Self-Attention(BCTS) 90.02 87.26 88.62 BCTS 57.92 48.25 52.64

ASAIBC 91.54 89.73 90.63 × 58.69 50.12 54.07

MSRA	WeiboNER
CRF	×	×	×	×	56.59	25.28	34.95
CRF+word	×	×	×	×	64.68	25.63	36.85
CRF+character	×	×	×	×	57.89	34.02	42.86
CRF+character+position	×	×	×	×	57.26	34.53	43.09
BiLSTM+CRF(BC)	88.56	87.56	88.06	BC	58.89	44.89	50.95
BC+Transfer(BCT)	89.96	89.25	89.60	BCT	59.89	46.25	52.19
BCT+Self-Attention(BCTS)	90.02	87.26	88.62	BCTS	57.92	48.25	52.64
ASAIBC	91.54	89.73	90.63	×	58.69	50.12	54.07

Table 6 provides the experimental results of our proposed model as well as its simplified models on MSRA dataset and WeiboNER dataset. According to Table 6, the CRF, CRF+word, CRF+character, CRF+character+position models proposed by Peng and Dredze et al. [45]. were tested on WeiboNER data set. It can be observed that the CRF+character+position models obtained the best experimental results. The CRF model with word vector was 1.9% higher than the CRF model, and the CRF model withword embedding was 6.01% higher than the CRF model with word vector. Thus it can be seen that the performance of word embedding is usually better than that of word embedding in NER task. Meanwhile, the F value of CRF+character+position model is 0.23% higher than that of CRF+character model, indicating that position information plays an auxiliary role in entity recognition. The F values of BiLSTM+CRF model on MSRA data set and WeiboNER data set are 88.06% and 50.95% respectively; the F values of BiLSTM+CRF+Transfer model on MSRA data set and WeiboNER data set are 89.60% and 52.19%; the F values of BiLSTM+CRF+Transfer+self-Attention model on MSRA data set and WeiboNER data set are 88.62% and 52.64%. In this paper, the F values of ASAIBC model based on the Adversarial Transfer Learning framework are 90.63% and 54.07% respectively on MSRA dataset and WeiboNER dataset.

In summary, the experimental results of ASAIBC model on MSRA data set and WeiboNER data set are the best. As shown in Table 6, we can find that the recognition efficiency of deep learning model in WeiboNER data set is not high, mainly due to the fact that the quantity of current public WeiboNER data set data is relatively small, only 400 or so. Meanwhile, since the WeiboNER data set come from weibo, the experimental data is not standard, with entity nesting, abbreviation and ambiguity. Therefore the F value of deep learning model in WeiboNER data set is not high on the whole.

At the same time, in order to demonstrate the effectiveness of ASAIBC model on XJADRNER data set, different models were tested on XJADRNER data set. The experimental results are shown in Table 7.

Table 7

Performance comparison of different models on XJADRNER dataset

XJADRNER
Models	P (%)	R (%)	F (%)
CRF	88.65	68.31	77.16
BiLSTM	87.15	71.63	78.63
BiLSTM+CRF	88.52	78.92	83.44
TrBiLSTM+CRF	91.57	72.30	80.80
Self-Attention+BiLSTM+CRF	88.92	85.23	87.04
ATR+Self-Attention+BiLSTM+CRF	89.96	88.05	88.99
ASAIBC	91.01	90.21	90.57

In Table 7, the XJADRNER data set was tested by using the currently popular deep learning algorithm. The experimental results showed that the F value of ASAIBC model in entity recognition of local adverse drug reactions in Xinjiang was 90.57%. From the experimental results of Table 7, we have following observations: by introducing adversarial training, ASAIBC boosts the performance as compared with Self-Attention+BiLSTM+CRF model, showing 3.53% improvement on XJADRNER dataset. It proves that adversarial training can prevent specific features of CWS task from creeping into shared space. ASAIBC boosts the performance as compared with ATR+Self-Attention+BiLSTM+CRF model, showing 1.58% improvement on XJADRNER dataset. It proves that IndRNN model can learn the long-term dependence of sentences better. At the same time, ASAIBC boosts the performance as compared with CRF model, showing 13.41% improvement on XJADRNER dataset. It proves that ASAIBC model complements the advantages and disadvantages of the single model and can enhance the expression performance of features.

In order to show the changes of ASAIBC model more clearly during training, the loss curve of the model is drawn below and shown in Fig. 8. It can be seen from Fig. 8 that the loss function of ASAIBC model with IndRNN has a fast convergence the minimum convergence value. In the experiment, 100 is the step length. When the iterative training step = 5000, the loss of ASAIBC model is close to 0.500. This shows that the IndRNN model is easier to train and has better fitting property, which can better fit the data in the training set.

Fig. 8

Comparison curve of model loss.

In order to verify the effectiveness of ASAIBC model based on transfer learning in drug entity recognition of local adverse drug reactions in Xinjiang province from different perspectives, the following comparative experiments were designed in this paper.

The effect of word vector dimension on experimental performance.

In the experiment, the use of different word vector dimension will have different effects on the experimental performance. The semantic information varies in different word vector dimension. In order to get the optimal word vector dimension, the experimental parameters need to be constantly adjusted and updated. In this paper, 20∼200 are selected as the word vector dimension to conduct experiments.

Figure 9 shows that different vector dimensions have a certain influence on experimental performance. When the word vector dimension is increasing, the semantic and grammatical information also rises up. When the word vector is 100, the experimental results are the best. As the dimension of the word vector increases, the semantic and grammatical information also contains more impurity information unrelated to the experiment, which will have a negative impact on the experiment. When the word vector dimension increased to 160 dimensions, the F value obtained by ASAIBC model decreased by 3.3%, and when it increased to 200 dimensions, the F value obtained by ASAIBC model decreased by 4%. As a result, it is necessary to choose an appropriate word vector dimension to improve the experimental performance.

Fig. 9

Influence of different word vector dimensions on experimental performance.

5 Conclusion

In the Chinese NER method based on deep learning, a NER method for ADR based on the Adversarial Transfer Learning framework is proposed to address deep learning’s insufficient ability to learn a small amount of data. The CWS data set is taken as the source domain, while the experimental data set XJADRNER as the target domain in this paper. The boundary sharing information of CWS and XJADRNER data sets was fully utilized to share information, the specific information in CWS was filtered at the same time. Beyond improving the accuracy of entity recognition, the problem of few manual annotation of the current XJADRNER data set was solved. Moreover, the IndRNN is applied in the NER task for the first time, and self-Attention is used to capture the overall dependence of the whole sentence, learn the internal structure characteristics of the sentence, and make full use of the context information to predict the current entity attributes. The experimental results show that the ASAIBC models introduced with the transfer learning framework are superior to the current models and can improve the effect of Chinese NER.

Footnotes

Acknowledgments

We thank the anonymous reviewers for their insightful comments. This work was supported by the National Natural Science Foundation of China (61962057), Key Program of National Natural Science Foundation of China (U2003208), Major science and technology projects in the autonomous region (2020A03004-4).

References

Wei

, Li

, Xie

, Sheng

, Chen

and Zhang

, Exploiting Entity BIO Tag Embeddings and Multi-task Learning for Relation Extraction with Imbalanced Data, In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy: ACL, 2019, 1351–1360.

and Wang

, Chinese named entity recognition and word segmentation based on character[C], Proceedings of the Sixth SIGHANWorkshop on Chinese Language Processing, 2008.

Liu

, Zhu

and Zhao

, Chinese Named Entity Recognition with a Sequence Labeling Approach: Based on Characters, or Based on Words?[M], Advanced Intelligent Computing Theories and Applications, With Aspects of Artificial Intelligence, Springer Berlin Heidelberg, 2010.

, Hagiwara

, Li

, et al., Comparison of the Impact of Word Segmentation on Name Tagging for Chinese and Japanese[C], LREC 2014, 2532–2536.

Chen

, Zhang

and Isahara

, Chinese named entity recognition with conditional random fields[C], Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing 2006, 118–121.

, Zhang

and Ji

D.-H.

, Multi-prototype Chinese character embedding, In LREC, 2016.

Dong

, Zhang

, Zong

, et al., Character-based lstm-crf with radical-level features for chinese named entity recognition[M], Natural Language Understanding and Intelligent Applications, Springer, Cham, 2016:239–250.

Zhao

and Kit

, Unsupervised segmentation helps supervised learning of character tagging for word segmentation and named entity recognition[C], Proceedings of the Sixth SIGHANWorkshop on Chinese Language Processing, 2008.

Peng

and Dredze

, Named entity recognition for chinese social media with jointly trained embeddings[C], Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing 2015, 548–554.

10.

and Sun

, F-score driven max margin neural network for named entity recognition in chinese social media[J], arXiv preprint arXiv:1611.04234, 2016.

11.

Peng

and Dredze

, Improving named entity recognition for chinese social media with word segmentation representation learning, In, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics 2 (2016), 149–155.

12.

Luo

and Yang

, An empirical study of automatic chinese word segmentation for spoken language understanding and named entity recognition, In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 238–248, 2016.

13.

Luo

, Huang

, Lin

C.Y.

, et al., Joint entity recognition and disambiguation [C], In, Conference on Empirical Methods in Natural Language Processing 2016, 879–888.

14.

Passos

, Kumar

and Mccallum

, Lexicon Infused Phrase Embeddings for Named Entity Resolution [C], In, Proceedings of the Eighteenth Conference on Computational Natural Language Learning 2014, 78–86.

15.

Hammerton

, Named Entity Recognition with Long Short-Term Memory[C], Conference on Natural Language Learning at Hlt-naacl, Association for Computational Linguistics, 2003.

16.

Collobert

and Weston

, A unified architecture for natural language processing: Deep neural networks with multitask learning [C], Proceedings of the 25th International Conference on Machine Learning, 2008:160–167.

17.

Turian

, Ratinov

and Bengio

, Word representations: a simple and general method for semi-supervised learning [C], Proceedings of the 48th annual meeting of the association for computational linguistics, Association for Computational Linguistics 2010, 384–394.

18.

Lample

, Ballesteros

, Subramanian

, et al., Neural Architectures for Named Entity Recognition [J], Computer Science 2016, 260–270.

19.

and Hovy

, End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF [J], In, Association for Computational Linguis-tics (ACL) 2016, 848–865.

20.

Collobert

, Weston

, Bottou

, et al., Natural Language Processing (Almost) from Scratch[J], Journal of Machine Learning Research, 2011.

21.

Santos

C.N.D.

and Guimarães

, Boosting Named Entity Recognition with Neural Character Embeddings[J], Computer Science, 2015.

22.

Strubell

, Verga

, Belanger

, et al., Fast and accurate entity recognition with iterated dilated convolutions[J], arXiv preprint arXiv:1702.02098, 2017.

23.

Peters

M.E.

, Ammar

, Bhagavatula

, et al., Semi-supervised sequence tagging with bidirectional language models[J], arXiv preprint arXiv:1705.00108, 2017.

24.

Peters

M.E.

, Neumann

, Iyyer

, et al., Deep contextualized word representations[J], arXiv preprint arXiv:1802.05365, 2018.

25.

Devlin

, Chang

M.W.

, Lee

, et al., Bert: Pre-training of deep bidirectional transformers for language understanding[J], arXiv preprint arXiv:1810.04805, 2018.

26.

Gurulingappa

, Rajput

A.M.

, Roberts

, Fluck

, Hofmann-Apitius

and Toldo

, Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports, J Biomed Inform 45(5) (2012), 885–892.

27.

Alvaro

, Miyao

and Collier

, Twimed: Twitter and pubmed comparable corpus of drugs, diseases, symptoms, and their relations, Jmir Public Health Surveill 3(2) (2017), 24.

28.

Kandaswamy

, Silva

L.M.

, Alexandre

L.A.

, et al., Improving deep neural network performance by reusing features trained with trans-ductive transference [C], International Conference on Artificial Neural Networks (ICANN) 2014, 265–272.

29.

Huang

J.T.

, Li

, Yu

, et al., Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers [C], IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2013, 7304–7308.

30.

Behbood

, Lu

and Zhang

, Text categorization by fuzzy domain adaptation [C], IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) 2013, 1–7.

31.

Pan

S.J.

and Yang

, A survey on transfer learning, IEEE TKDE 22(10) (2010), 1345–1359.

32.

and Hovy

, End-to-end sequence labeling via bi-di-rectional LSTM-CNNs-CRF [C], Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016) 2016, 1064–1074.

33.

Mikolov

, Yih

and Zweig

, Linguistic Regularities in Continuous Space Word Representations[C], Atlanta: Association for Computational Linguistics, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies 2013, 746–751.

34.

Mikolovt , Chenk , Corradog , et al., Efficient Estimation of Word Representations in Vector Space [EB/OL]. [2018-07-20]. https://arxiv.org/abs/1301.3781.

35.

Pennington

, Socher

and Manning

, Glove: Global Vectors for Word Representation[C], Doha: Association for Computational Linguistics, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) 2014, 1532–1543.

36.

Yanhong

, Hong

, Geng

, et al., Named entity recognition method based on blstm [J], Computer Science 45(2) (2018), 261–268.

37.

Goodfellow

, Pouget-Abadie

, Mirza

, Xu

, Warde-Farley

, Ozair

, Courville

and Bengio

, Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014.

38.

Denton

E.L.

, Chintala

, Fergus

, et al., Deep generative image models using a laplacian pyramid of adversarial networks, In Advances in neural information processing systems, pages 1486–1494, 2015.

39.

Ganin

and Lempitsky

, Unsupervised domain adaptation by backpropagation, arXiv preprint arXiv:1409.7495, 2014.

40.

Gui

, Zhang

, Huang

, Peng

and Huang

, Part-of-speech tagging for twitter with adversarial neural networks, In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2411–2420, 2017.

41.

Chen

, Shi

, Qiu

and Huang

, Adversarial multi-criteria learning for chinese word segmentation, arXiv preprint arXiv:1704.07556, 2017.

42.

Liu

, Qiu

and Huang

, Adversarial multi-task learning for text classification, arXiv preprint arXiv:1704.05742, 2017.

43.

, Li

, Cook

, Zhu

and Gao

, Independently recurrent neural network (indrnn): building A longer and deeper rnn, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Utah, USA: IEEE, 2018. 5457–5466.

44.

Lafferty

J.D.

, McCallum

and Pereira

F.C.N.

, Conditional random fields: Probabilistic models for segmenting and labeling sequence data, in: Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 – July 1, 2001, 2001, pp. 282–289.

45.

Peng

and Dredze

, Named entity recognition for chinese social media with jointly trained embeddings, In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 548–554, 2015.

				MSRA	WeiboNER
	Person Name	Place Name	Organization Name	Total	Total
Training set	21635	38965	23562	84142	1350
Test set	2015	2362	958	5335	270

Named entity recognition of local adverse drug reactions in Xinjiang based on transfer learning

Abstract

Keywords

1 Introduction

2 Related work

2.1 Named entity recognition of adverse drug reactions

Table 1 Corpus tagging for entity recognition of adverse drug reactions

3.2 Embedding layer

4.1 Dataset

Table 4 Statistics of entities in MSRA and WeiboNER datasets MSRA WeiboNER Person Name Place Name Organization Name Total Total Training set 21635 38965 23562 84142 1350 Test set 2015 2362 958 5335 270

Table 5 Experimental parameter setting Parameter P_name Value en Epochs_num 260 aw Atr_weight 0.06 bs Batch_size 64 ed Embedding_dims 100 fn Task_num 2 ld Lstm_dim 120 ctn Cws_tags_num 4 atn Adr_tags_num 7

Footnotes

Acknowledgments

References

Table 1
Corpus tagging for entity recognition of adverse drug reactions

Table 4
Statistics of entities in MSRA and WeiboNER datasets

MSRA WeiboNER

Person Name Place Name Organization Name Total Total

Training set 21635 38965 23562 84142 1350

Test set 2015 2362 958 5335 270

Table 5
Experimental parameter setting

Parameter P_name Value

en Epochs_num 260

aw Atr_weight 0.06

bs Batch_size 64

ed Embedding_dims 100

fn Task_num 2

ld Lstm_dim 120

ctn Cws_tags_num 4

atn Adr_tags_num 7