A named entity recognition model based on ensemble learning

Abstract

Knowledge Graph has gradually become one of core drivers advancing the Internet and AI in recent years, while there is currently no normal knowledge graph in the field of agriculture. Named Entity Recognition (NER), one important step in constructing knowledge graphs, has become a hot topic in both academia and industry. With the help of the Bidirectional Long Short-Term Memory Network (Bi-LSTM) and Conditional Random Field (CRF) model, we introduce a method of ensemble learning, and implement a named entity recognition model ELER. Our model achieves good results for the CoNLL2003 data set, the accuracy and F1 value in the best experimental results are respectively improved by 1.37% and 0.7% when compared with the BiLSTM-CRF model. In addition, our model achieves an F1 score of 91% for the agricultural data set AgriNER2018, which proves the validity of ELER model for small agriculture sample data sets and lays a foundation for the construction of agricultural knowledge graphs.

Keywords

Knowledge graph NER Bi-LSTM CRF ensemble learning

1. Introduction

Knowledge acquisition [1] is a core step of knowledge graph construction which aims to extract useful information from massive text data. Named entity recognition [2] is a classical problem in knowledge acquisition, which is to identify information with special meaning or clear referent from text.

Named entities generally refer to entities that have a specific meaning or referentiality in the text, and usually include names of people, places, organizations, dates, times, and proper nouns. NER systems are to extract the above entities from the unstructured input text, and can identity more categories of entities according to demands, such as product names, models, prices and so on. The concept of entity is unlimited, entities involved in the agricultural dataset in this article are the descriptors of the “agricultural thesaurus” [3].

NER can be roughly divided into two methods: rule-based method and machine-learning method. In the early stage of named entity recognition research, rule-base method dominates [4]. However, most rule-based entity recognition methods are constructed manually by linguistic experts, but this manner has poor system portability and is too expensive.

Approaches of machine learning for NER research mainly include feature-based method and neural network-based method [5]. Currently, a variety of machine learning models have been used for named entity recognition, including language model [6], hidden Markov model [7, 8], maximum entropy model, error-driven learning method, decision tree method, DL-CoTrain and CoBoost, etc.

Although NER methods based on traditional characteristics have achieved high performance, such methods rely on artificial features and existing natural language processing tools, so that the scalability is poor, and a lot of effective features need to be designed or mined manually. In recent years, with the development of deep learning, neural network-based methods have been widely used and achieved good results [9, 10, 11, 12].

BiLSTM-CRF model is proposed by Huang et al. [13], in which they constructed a variety of neural network models to solve the sequence annotation problem in natural language processing. Since then, BiLSTM-CRF model has been widely used in field of named entity recognition. However, most of these models cannot capture the semantic information between sentences. On the basis of the work of Huang et al., we introduced the method of ensemble learning to solve this problem and improved the F1 value of BiLSTM-CRF model for data set CONLL2003. In addition, we used the agricultural data set AgriNER2018 constructed by our team to conduct experiments on ELER model and obtained excellent results, which verified the validity of the model for small sample data set for agriculture, and was of great significance to the construction of agricultural knowledge graphs.

The rest of this paper is structured as follows. In Overview, we introduced the research status of named entity recognition and the theoretical basis of the models involved in ELER. Subsequently, the structure and training details of ELER model are discussed in Proposed Method. Thereafter, we present the results of our experiments in Experiments. Finally, a summary is in Conclusion.

2. Overview

2.1 LSTM

Recurrent Neural Network (RNN) can improve the performance of neural network by understanding the sequence dependence, but due to its training algorithm, the model may have the problem of gradient explosion or gradient disappearance in the training process, Long Short-Term Memory (LSTM) [14] is one of the most popular and effective ways to reduce this problem. The LSTM changes the structure of the hidden units from “Sigmoid” or “tanh” to storage units, where their inputs and outputs are controlled by gates that control the flow of information to the hidden neurons and preserve features extracted from previous time steps. A typical LSTM cell consists of an input node, an input gate, a forgetting gate, and an output gate, as well as a cell activation component, as shown in the Fig. 1, where $g_{c}$ represents input node, $i_{c}$ , $f_{c}$ , and $v_{c}$ respectively represent input gate, forgetting gate and output gate, $s_{c}$ represents activation component and superscript t represents the current time step. These units receive activation signals from different sources and control the activation of the unit through a designed multiplier.

Figure 1.

A typical LSTM cell.

In 2003, Hammerton et al. [15] applied LSTM to named entity recognition, which was the first application of neural network in NER; In 2011, Collobert et al. [16] proposed a window-based deep neural network model (CNN-CRF), whose training effect exceeded all previous traditional algorithms. People who also use convolutional neural network for NER include Yao C et al. [17], who adopted the CNN model for named entity recognition to optimize online medical guidance, Chiu et al. [18], who proposed to combine BiLSTM with CNN, break the limitation of fixed window size by using BiLSTM model, and use CNN to obtain clear character-level features such as prefix and suffix, thus improving the effect of named entity recognition, and Zheng et al. [19], who proposed a hybrid neural network model to extract entities and their relations, this model combined a bidirectional encod-decoder LSTM module for entity extraction and a CNN module for relation classification for joint extraction without any manually constructed features.

2.2 CRF

Conditional Random Field (CRF) is a discriminant probability model, whose characteristic is that the output random variable is assumed to constitute Markov random field. CRF is defined as follows:

Assuming that $X$ and $Y$ represent the observation sequence to be marked and the corresponding marker sequence respectively, then $P$ is the conditional probability of $Y$ given $X$ . If random variable $Y$ satisfies Markov independence hypothesis, that is, $P(Y_{v}|X,Y_{w},w\neq v)=P(Y_{v}X,Y_{w},w\sim v)$ is true for any $v$ , then the conditional $P$ is called Conditional Random Field. Where $w\sim v$ represents all adjacent nodes $w$ of node $v$ , $w=v$ represents all nodes $w$ except node $v$ , and $Y_{v}$ and $Y_{w}$ represent random variables corresponding to node $v$ and $w$ .

2.3 BiLSTM-CRF

BiLSTM-CRF model combines a bidirectional LSTM network with a CRF network. When a sentence is input into BiLSTM-CRF, each unit represents a vector composed of self-embedding or word embedding, and all embeds are tuned to the optimum during training. As shown in Fig. 2, firstly, the word embedding vectors are input into BiLSTM layer. One LSTM processes sequences are from start to finish in the forward time direction, and the other LSTM processes the sequence are from start to finish in the negative time direction. Compared with LSTM-CRF, BiLSTM-CRF model can not only effectively use input features before current time step t, but also learn future input features, which can enhance the accuracy of marking. The CRF model has a transition feature that takes into account the sequentially between output labels. BiLSTM layer connected after CRF as the output layer can effectively use past and future tags to predict current tags. So the problem that the tags is independent between each other when the common softmax layer is used as the output layer which does not consider the correlation among outputs, is solved.

Figure 2.

BiLSTM-CRF model.

BiLSTM-CRF model has been widely used in field of named entity recognition since 2015, Lee et al. [20] combine BiLSTM-CRF and CNN to realize the named entity recognition task by building BiLSTM-CNNs-CRF model; Wu et al. [21] added the example-based migration algorithm, and transferred the knowledge from the source domain to the target domain through the method of weight generation and sample selection by using BiLSTM-CRF model, effectively solving the problem of insufficient learning ability of deep learning to a mall amount of data; Chen et al. [22] proposed a divide-and-conquer method to improve their ability to analyze emotions by classifying sentence types using BiLSTM-CRF and CNN; Luo et al. [23] introduced attention mechanism on the basis of BiLSTM-CRF to conduct document-level chemical NER, this model achieved better performance results with fewer feature engineering; Gridach [24] proposed a novel neural network structure based on BiLSTM and CRF, which can acquire knowledge from word- and character-level representations automatically, eliminating the need for most functional engineering for biomedical named entity recognition; Li et al. [25] with CNN to extract character-level representations, BiLSTM-CRF for biomedical entity recognition, and using BiLSTM-RNN for relation classification, etc. [26, 27].

2.4 Ensemble learning

Ensemble learning, also known as multi-classifier system, completes learning tasks by building and combining multiple learners. The system produces several individual learners, which are usually generated from training data by existing learning algorithms, and then are combined with certain strategies to obtain a strong learner. Previous works suggest that ensemble learning can be used to improve the performance of named entity recognition tools [28]. There are two main problems to be solved in ensemble learning: how to get several individual learners and how to choose combination strategies. In this paper, bagging [29] algorithm is adopted to generate individual learners. In this method there are few dependencies between individual learners, so a parallelization method can be generated. Then, the voting method is adopted as a combination strategy, that is, a voting classifier is obtained based on these individual learning machines, and the class with the most votes is taken as the final prediction category.

3. Proposed method

3.1 ELER

At present, common BiLSTM-CRF models construct the input of the model through sentence sampling, without considering the context information among sentences. In this paper, we propose an entity recognition model ELER based on ensemble learning, in addition to the sentence sampling method, we also adopt the truncation sequence sampling method, which means to intercept the sample training set data with a fixed length, so that the length of the intercepted data is the same without padding, so as to learn the context information between sentences. In ELER model, we use bagging algorithm to generate individual learners, and then use voting method as the combination strategy to integrate the sentence sampling method and truncated sequence sampling method.

The basic idea of ELER model is as follows:

1.
Generate three input for training, the one through the sampling method with sentence as unit, and other two through truncated sampling method with different lengths;
2.
Obtains three different individual learners by using three inputs training BiLSTM-CRF model respectively;
3.
Using trained individual learners to predict results;

Figure 3.
ELER model.

Figure 4.
The process of ELER execution.

Votes on the predicted results through three individual learners, which are the predicted results of ELER model, and then calculate the accuracy rate, recall rate and F1. The structure of this model is shown in Fig. 3.

The basic idea of voting in ELER model is as follows:

1.
Pairwise compare the prediction results of the three learners;
2.
Preserving the entities with the same offset and category in a common collection entity_all;
3.
Discard all entities that appear only once;
4.
Entities in the entity_all are unique.

The specific algorithm process is shown in Fig. 4.

According to the above voting strategy, ELER model can obtain the maximum number of correctly predicted entities while the minimum number of predicted entities can be obtained, which will help improve the accuracy of entity identification, recall rate and F1 value.
3.2 Data preprocessing

We preprocesse Numbers and upper and lower case letters. The specific strategies are as follows:

1.
Change all uppercase letters into lowercase letters (in order to solve the problem of data sparseness);
2.
Represents three hundred dimensions of zero for all numbers;
3.
Directly represents the three-hundred-dimensional vector in the word embedding set when a word is embedded in the word embedding set;
4.
Represents three hundred dimensions of zero when a word is not embedded in the word set.

3.3 Model training process

ELER model is obtained through ensemble learning of three BiLSTM-CRF models. Therefore, training of ELER model is mainly conducted on three BiLSTM-CRF models respectively. The training process of BiLSTM-CRF is as follows:

Algorithm 1 Training process of BILSTM-CRF model
1. Program starts
2. Define epoch (number of iterations)
3. Define batch_size (batch size)
4. Define iteration (the number of cycles of data for an iterative process), iteration $=$ number of training/batch_size
5. Define the structural parameters of the LSTM network
6. For $i$ in range (0, epoch):
7. For $j$ in range (0, iteration):
8. (1) BILSTM-CRF model forward propagation:
9. Forward LSTM propagates forward from beginning to end in forward time direction
10. Backward LSTM propagates forward from beginning to end in negative time direction
11. (2) CRF layer
12. CRF layer propagates forward
13. CRF layer propagates backward
14. (3) BILSTM-CRF model propagates backward
15. Forward LSTM propagates backwards from start to end in negative time direction
16. Backward LSTM propagates backward from start to end in forward time direction
17. End of program

For each batch of training, the first runs forward propagation of BiLSTM-CRF model, including forward propagation of forward LSTM from beginning to end in forward time direction and forward propagation of backward LSTM from beginning to end in negative time direction, and output score matrix $f_{\theta}({[x]_{1}^{T}})$ of all labels will be obtained. Next, mark sequences are calculated through forward and back propagation of CRF layer. Then, errors are backpropagation through BiLSTM-CRF model, including backward propagation of forward LSTM from beginning to end in negative time direction and backward propagation of backward LSTM from beginning to end in forward time direction. The parameters of the transfer matrix and BiLSTM then updated through back propagation.

3.4 Objective function

According to ELER model maximizes the logarithmic likelihood of sentence level. The main idea is to consider the probability of tag-to-tag conversion to explore the best road to optimize. First of all, a tag transformation matrix $A$ is defined, where $A_{i,j}$ represents the score jumping from tag $i$ to tag $j$ in the continuous segment, and $A_{0,i}$ represents the score starting from tag $i$ . $\theta$ is defined as the parameter set of the ELER model, and the new parameter set can be expressed as $\theta=\theta\cup{\{}A_{i,j}\forall i,j{\}}$ . For a given sentence $[x]_{1}^{T}$ of length $T$ , the output of the corresponding ELER model is a scoring matrix $f_{\theta}({[x]_{1}^{T}})$ . $[{f_{\theta}}]_{i,t}$ represents the score of the i’th word and the $t$ mark in sentence $[x]_{1}^{T}$ under a given parameter set $\theta$ . Then, the final score of a marking sequence $[i]_{1}^{T}$ corresponding to sentence $[x]_{1}^{T}$ can be expressed as:

$\displaystyle{S}({[x]_{1}^{T},[i]_{1}^{T},\theta^{\prime}})=\sum_{t=1}^{T}(A_{% [i]_{t-1},[i]_{t}}+[f_{\theta}]_{[i]_{t}},t)$ (1)

Let $[y]_{1}^{T}$ represent the correct tag sequence of sentence $[x]_{1}^{T}$ , then the logarithmic likelihood of the sentence level is obtained by using the score of all possible tag sequences $[j]_{1}^{T}$ after softmax:

$\displaystyle\log P([y]_{1}^{T}|[x]_{1}^{T},\theta^{\prime})=S([x]_{1}^{T},[y]% _{1}^{T},\theta^{\prime})-\log\sum_{\forall[j]_{1}^{T}}e^{S([x]_{1}^{T},[j]_{1% }^{T},\theta^{\prime})}$ (2)

The logarithmic likelihood objective function and its gradient descent can be calculated bu dynamic programming method. For the given network output score $[{f_{\theta}}]_{i,t}$ , this paper uses the Viterbi arithmetic [31] to obtain the maximum score $S({[x]_{1}^{T},[j]_{1}^{T},{\theta}^{\prime}})$ of the labeled sequence $[i]_{1}^{T}$ .

3.5 Evaluation index

In order to evaluate the effect of entity recognition, the Precision, Recall Rate and F1, which are used in most existing recognition models are used in this paper. Among them, Precision and Recall Rates are defined below:

$\displaystyle\text{Precision}=\frac{E_{\textit{correct}}}{E_{\textit{% recognition}}}$ (3) $\displaystyle\text{Recall}=\frac{E_{\textit{correct}}}{E_{\textit{sample}}}$ (4)

Among them, $E_{\textit{correct}}$ represents the correct number of entities identified in the output sequence model, $E_{\textit{recognition}}$ represents the total number of entities identified in the output sequence model, and $E_{\textit{sample}}$ represents the number of entities contained in data set. Precision reflects accuracy, which is how many correct entities are identified, and Recall, or how many correct entities are identified by the algorithm. The value of Precision Rate and Recall Rate is between 0 and 1. the closer the value is to 1, the higher the Precision Rate or Recall Rate is. However, Precision and Recall are two values that affect each other, when Precision is increased, Recall Rate decreases, and when Recall Rate is increased, Precision Rate decrease. Therefore, the weighted and average values of Precision and Recall should be considered comprehensively, i.e. F1 for evaluation. It indicates that the experimental method is more effective when F1 value is high. The calculation formula of F1 is defined as follows:

$\displaystyle F1=\frac{2\times\text{Precision}\times\text{Recall}}{\text{% Precision}+\text{Recall}}$ (5)

4. Experiments

4.1 Experimental data set

In our work, Glove [30], the public word embedding set trained by Stanford, is used to represent the input of English data set (download address: https://nlp.stanford.edu/projects/glove/). Glove is the word embedding set trained by Pennington et al. through 6 million words obtained from Wikipedia and other web pages. The word embedding size in this paper is 300 dimensions.

The experiments of this paper are adopted to standard data set CoNLL2003 [31] and agricultural data set AgriNER2018. The first data set is extracted from news data of Reuters, in which the data of named entity identification consists of eight files covering German and English. The English data set of CoNLL2003 mainly marks four types of entities, including location, organization, person and miscellaneous.

The task of entity recognition is to give each word in a sentence an entity tag. A single entity may consist of multiple words. The IOB labeling method is used in this article, Where letter I represents inside, a word that is not the first word in the entity. Letter O is outside, which represents all word that are not contained in the entity set. Letter B is the beginning, which represents the first word in the entity. If a word is first word of an entity, it is labeled B-label. If a word is part of an entity but not the first word, it is indicated as I-label, and all other non-entities are labeled as O.

All data files in CoNLL2003 contain one word per line, with blank lines representing sentence boundaries. At the end of each line is a label indicating whether the current word is an entity. here is an example sentence from the CoNLL2003 dataset:

EU	NNP	I-NP	I-ORG
Rejects	VBZ	I-VP	O
German	JJ	I-NP	I-MISC
Call	NN	I-NP	O
To	TO	I-VP	O
Boycott	VB	I-VP	O
British	JJ	I-NP	I-MISC
Lamb	NN	I-NP	O
.	.	O	O

As you can see, each row of data contains four parts: word (the first column), POS (part of speech) tag (the second column), trunk tag (the third column), and NER tag (the fourth column). The named entities in the dataset are non-recursive and non-overlapping.

The numbers of sentences, fragments and entities in each file of CoNLL2003 and are shown in the Table 1.

Table 1
Number of sentences, tokens, entities in CoNLL2003

English data	Sentences	Tokens	Entities
Training set	14041	203621	33973
Development set	3250	51362	8595
Test set	3453	46435	8092

Since there is no publicly available agricultural entity identification data set, we constructed the data set AgriNER2018 using agricultural thesaurus and supplemented it with web crawlers. In the experiments, we selected three entity categories of sediment, soil forming process and soil layer for labeling. In order to extend the positive sample of entity labeling data, we consider the subcategories of these three types of entities as the same category as them. Taking sediment as an example, in order to construct a sediment entity recognition data set, we crawl all the text information of the webpage of the sediment and all its sub-entities from Baidu Wikipedia, such as avalanches, morains, residues and so on. The crawled contents include abstracts and all text information under the directory title.

The specific steps to construct dataset AgriNER2018 from Baidu Wikipedia are as follows:

Sort out the URLs of web pages to be crawled. URLs of Baidu Wikipedia have a fixed format: “http://baike.baidu.com/item/entry”. Therefore, we can replace the “entry” in the URL to get a collection of URLs to crawl. For example, “http://baike.baidu.com/item/colluvial”, “http://baike.baidu.com/item/sediment”, and “http://baike.baidu.com/item/morains”, etc.

Edit web crawler and run it.

Data cleaning. Clean up the data crawled in the previous step, remove unwanted data, and save the data as a column.

Annotation data. The annotation method used in AgriNER2018 data set is similar to that used in CoNLL2003. Among them, “SED” is used to represent deposits, “TOPSOIL” is for soil layer and “SOILFORMING” represent soil-forming process, etc.

AgriNER2018 is divided into training set and test set, the numbers of sentences, fragments and entities in each set are shown in the Table 2.

Table 2

Number of sentences, tokens, entities in AgriNER2018

Data	Sentences	Tokens	Entities
Training set	1528	71736	1229
Test set	231	10242	127

4.2 Experimental parameter setting

The network model of ELER uses BPTT algorithm to update the parameters, the optimization algorithm uses Adam, and the range of gradient clipping value clip is 5.0. The sampling length of two different stages is 20 and 30 respectively. The number of LSTM layers number_layers is 300, the input dimension of cells input_size is 300, the learning rate lr is 0.001, epoch is 100, batch_size is 128, and the word vector dimension is 300.

4.3 Results

The entity recognition results of all models by using the CoNLL2003 data set are shown in Tables 3 and 4, and the results for AgriNER2018 are show in Tables 5 and 6. Among them, BiLSTM-CRF-Sen model reproduced the model proposed by Huang [13], and the experimental results are as follows.

Table 3
Prediction numbers of different models on CoNLL2003

	Predicted correctly	Predicted	Sample
BiLSTM-CRF-Batch20	6885	7730	8092
BiLSTM-CRF-Batch30	6988	7918	8092
BiLSTM-CRF-Sen (Huang)	7026	7869	8092
ELER	7031	7755	8092

Table 4

Performance comparison of different models on CoNLL2003

PRF	Precision	Recall	F1
BiLSTM-CRF-Batch20	89.07	85.08	87.03
BiLSTM-CRF-Batch30	88.25	86.36	87.29
BiLSTM-CRF-Sen (Huang)	89.29	86.83	88.04
ELER	90.66	86.89	88.74

Table 5

Prediction numbers of different models on AgriNER2018

	Predicted correctly	Predicted	Sample
BiLSTM-CRF-Batch20	108	118	127
BiLSTM-CRF-Batch30	114	131	127
BiLSTM-CRF-Sen (Huang)	97	103	127
ELER	110	114	127

Table 6

Performance comparison of different models on AgriNER2018

PRF	Precision	Recall	F1
BiLSTM-CRF-Batch20	91.53	85.04	88.16
BiLSTM-CRF-Batch30	87.02	89.76	88.37
BiLSTM-CRF-Sen (Huang)	94.17	76.38	84.35
ELER	96.49	86.61	91.29

In Tables 3 and 5, the second, third and fourth columns respectively represent the total number of correctly predicted entities, the total number of predicted entities, and the number of real entities in each named entity recognition experiment. As we can see, ELER obtained the maximum number of correctly predicted entities after voting under the condition that the total number of predicted entities was small. In Tables 4 and 6, the last row is the best results obtained by the ELER model in NER experiments. According to formula (2) and (3), Precision rate and Recall rate are a set of values with a balance between each other, this phenomenon can also be seen in this table. The experimental results in Tables 3 and 4 show that ELER model achieves better results than any individual model, in which the precision for CONLL2003 is increased by 1.37% and the F1 value is increased by 0.7%. The results in Tables 5 and 6 show that our model can be well applied to the entity recognition of agricultural small sample data sets.

5. Conclusion

In this paper, an entity recognition model ELER is designed based on BiLSTM-CRF model, which combines several weak learners into one strong learner by using the feature of ensemble learning. In the construction of ELER model, truncated sequence sampling method is combined with common sentence sampling method to avoid the defect that sentence sampling method will lose the context information among sentences. Compared with BILSTM-CRF-Sen, BILSTM-CRF-Batch30 and BILSTM-CRF-Batch20 models, experimental results show that ELER model has better prediction results by using standard data set CoNLL2003 and agricultural data set AgriNER2018, including higher Precision and F1 value, which indicates that ELER model can effectively improve the recognition effect of BiSLTM-CRF model and can be well applied to the data set of small agricultural samples, which is of great significance to the construction of agricultural knowledge graphs.

Footnotes

Acknowledgments

Our work has been fully supported by the National Key Research and Development Program (2017YFD0301506), the Natural Science Foundation of Hunan Province, China (Grant No. 2019JJ40133), National Natural Science Foundation of China (Grant No. 61972146), Double first-class Construction Project of Hunan Agricultural University, China (Grant No. SYL2019077), as well as the Key Research and Development Program of Hunan Province, China (Grant No. 2017NK2381).

References

Small

S.G.

and Medsker

, Review of information extraction technologies and applications, Neural Computing and Applications 25 (2014), 533–548.

Nadeau

and Sekine

, A survey of named entity recognition and classification, Lingvisticae Investigationes 30(1) (2007), 3–26.

Fang

L.M.

et al.,

AgriculturalThesaurus(thethirdvolume)

, China Agriculture Press, 1994, 14–15.

Grishman

and Sundheim

, Design of the muc-6 evaluation, in: Proceedings of the 6th Conference on Message Understanding, 1995, p. 111.

Young

et al., Recent trends in deep learning based natural language processing, IEEE Computational Intelligence Magazine 13(3) (2018), 55–75.

Y.Z.

Zhao

and Xu

, Chines named entity recognition combining a statistical model with human knowledge, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 15, 2003, pp. 65–72.

Sun

et al., Chinese named entity identification using class-based language model, in: Proceedings of the 19th International Conference on Computational Linguistics, 1, 2002, pp. 1–7.

Zhang

H.P.

et al., Chinese named entity recognition using role model, Computational Linguistics and Chines Language Processing 8(2) (2003), 29–59.

Chowdhury

et al., A multitask bi-directional RNN model for named entity recognition on Chinese electronic medical records, BMC Bioinformatics 19 (2018), 75–84.

10.

Wang

et al., Cross-type biomedical named entity recognition with deep multi-task learning, Bioinformatics 35(10) (2019), 1745–1752.

11.

Camacho-Collados

and Pilehvar

M.T.

, From word to sense embeddings: A survey on vector representations of meaning, Journal of Artificial Intelligence Research 63(1) (2018), 743–788.

12.

Nasar

Jaffry

S.W.

and Malik

M.K.

, Information extraction from scientific articles: A survey, Scientometrics 117(3) (2018), 1931–1990.

13.

Huang

and Yu

, Bidirectional LSTM-CRF Models for Sequence Tagging arXiv: Computation and language, 2015.

14.

Hochreiter

and Schmidhuber

, Long short-term memory, Neural Computation 9(8) (1997), 1735–1780.

15.

Hammerton

, Named entity recognition with long short-term memory, in: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, 4, 2003, pp. 172–175.

16.

Collobert

et al., Natural language processing (almost) from scratch, Journal of Machine Learning Research 12 (2011), 2493–2537.

17.

Yao

C.L.

et al., A convolutional neural network model for online medical guidance, IEEE Access 4 (2016), 4094–4103.

18.

Chiu

J.P.C.

and Nichols

, Named entity recognition with bidirectional LSTM-CNNS, Transactions of the Association for Computation Linguistics 4 (2016), 357–370.

19.

Zheng

et al., Joint entity and relation extraction based on a hybrid neural network, Neurocomputing 257 (2017), 59–66.

20.

Lee

D.Y.

and Lim

H.S.

, Bi-directional LSTM-CNN-CRF for Korean named entity recognition system with feature augmentation, Journal of the Korea Convergence Society 8 (2017), 55–62.

21.

L.V.

and Bi-Hui

Y.U.

, Chinese named entity recognition based on transfer learning and BiLSTM-CRF, Journal of Chinese Computer System 40 (2019), 1142–1147.

22.

Chen

et al., Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN, Expert Systems with Applications 72 (2017), 221–230.

23.

Luo

et al., An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition, Bioinformatics 34(8) (2017), 1381–1388.

24.

Gridach

, Character-level neural network for biomedical named entity recognition, Journal of Biomedical Informatics 70 (2017), 85–91.

25.

et al., A neural joint model for entity and relation extraction from biomedical text, Bmc Bioinformatics 18(1) (2017), 198.

26.

Feng

J.Z.

et al., Research on generative adversarial network of named entity recognition, Journal of Chinese Computer System 40(6) (2019), 1191–1196.

27.

Speck

and Ngomo

A.N.

, Ensemble learning for named entity recognition, International Semantic Web Conference 8796 (2014), 519–534.

28.

Breiman

, Bagging predictors, Machine Learning 24(2) (1996), 123–140.

29.

Viterbi

A.J.

, Error bounds for convolutional codes and an asymptotically optimum decoding algorithm, IEEE Transactions on Information Theory 13(2) (1967), 260–269.

30.

Goldberg

, A primer on neural network models for natural language processing, Journal of Artificial Intelligence Research 57(1) (2016), 345–420.

31.

Sang

E.F.T.K.

and Meulder

F.D.

, Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition, Computer Science 4 (2003), 142–147.