Knowledge enhancement BERT based on domain dictionary mask

Abstract

Semantic matching is one of the critical technologies for intelligent customer service. Since Bidirectional Encoder Representations from Transformers (BERT) is proposed, fine-tuning on a large-scale pre-training language model becomes a general method to implement text semantic matching. However, in practical application, the accuracy of the BERT model is limited by the quantity of pre-training corpus and proper nouns in the target domain. An enhancement method for knowledge based on domain dictionary to mask input is proposed to solve the problem. Firstly, for modul input, we use keyword matching to recognize and mask the word in domain. Secondly, using self-supervised learning to inject knowledge of the target domain into the BERT model. Thirdly, we fine-tune the BERT model with public datasets LCQMC and BQboost. Finally, we test the model’s performance with a financial company’s user data. The experimental results show that after using our method and BQboost, accuracy increases by 12.12% on average in practical applications.

Keywords

Intelligent customer service dictionary mask BERT data preprocessing

1. Introduction

With the development and application of Internet technologies such as big data and artificial intelligence, the traditional lending model of commercial banks has undergone disruptive changes. Intelligent customer service with artificial intelligence technology can shunt traffic, reduce labor costs, and improve operational efficiency and user experience [7–9]. Text semantic matching is the key technology of intelligent customer service, which estimates semantic similarity between the source and the target text pieces [3,22]. In loan Q&A, the semantic matching task is to calculate the similarity between the user and standard questions in the knowledge base. However, user questions without standard, various products and services’ name of large enterprises is unusual, which resulting semantic matching will be complicated [20].

Text semantic matching is divided into traditional and deep learning methods now. The conventional manner, such as TF-IDF, Bm25, VSM, are based on statistical methods, which use the lexical coincidence degree to calculate text literal similarity between two texts [13–15,18]. Deep learning has recently developed rapidly and is widely used in natural language processing [1,21,23,24], gradually replacing traditional methods. In 2013, Mikulov et al. [12] proposed the Word2vec model, which builds distributed word vectors to replace one-hot coding, solving the problem of sparse coding. However, word2vec model does not solve the problem of polysemy. In 2017, Vaswani et al. [19] proposed a Transformer structure based on attention. The transformer can parallel processing all words and solve the problem of polysemy by location coding. In 2018, Devlin et al. [4] proposed Bidirectional Encoder Representations from Transformers (BERT), a pre-training representation model based on a large-scale corpus, which is stacked by the encoder, part of transformer. Devlin et al. [5] also offered methods for BERT to adapt downstream tasks: Masked LM, Next Sentence Prediction, and fine-tuning. BERT with fine-tuning mode achieves state-of-the-art in many NLP tasks, demonstrating great potential of fine-tuning method.

Since BERT is proposed, fine-tuning on large-scale pre-training language model has become a general method for text semantic matching. However, due to the hidden meaning and size of the pre-training corpus, the BERT model is limited in applicable scenes. To improve the performance of BERT in the target domain, Sun et al. [17] used intra-task, intradomain, and cross-domain corpus to perform incremental pre-training on BERT. The results show that incremental pre-training can improve the model’s performance in the target domain.

The above improvement for pre-training phase mask shows its superiority on multiple open datasets. However, limited by the quality and quantity of corpus in the target domain, the BERT model still lacks domain knowledge, which shows the low accuracy of semantic precision matching. A knowledge enhancement method based on a domain dictionary mask is proposed to solve the above problems. Firstly, in pre-training, the target domain-related corpus is used for random and dictionary masking training in the BERT model. Secondly, the universal datasets LCQMC [10] are used to fine-tune and test the influence of pre-training strategy on the model performance. Finally, after replacing the domain-specific vocabulary, professional domain datasets BQ [2] are used to fine-tune again on the model to optimize the performance in downstream tasks. Experimental results show that the enhanced BERT compared with the original BERT on downstream tasks improves by an average of 12.12%.

The remainder of the paper is organized as follows: Section 1 describes the model structure and an enhancement method for knowledge in detail. Section 2 presents the results and discussion. Finally, Section 3 concludes the paper.

2. Knowledge enhancement of domain dictionary mask BERT

Since BERT is proposed, a new paradigm of pre-train-fine-tune has emerged in natural language processing. The pre-training model based on large-scale corpus can effectively alleviate the problem of insufficient corpus. In the downstream task, only a tiny amount of corpus is needed, which may improve the model’s performance by continuing to fine-tune [24]. Pre-train-fine-tune is shown in equations (1)–(2). $\begin{array}{l} (1) & {Model}_{pretrain} = pretrain (A) \\ (2) & {Model}_{final} = fine_tuning ({Model}_{pretrain}, B) \end{array}$

In equations (1)–(2), pretrain as a function represents the pretrain strategy; fine_tuning is also a function that describes the fine-tuning process. After pre-training on the specific domain datasets A, the model_pretrain is obtained. Fine-tuning on the downstream task datasets B on model_pretrain can get the final model model_final.

Fig. 1.

The model structure of incremental pre-training and fine-tuning on domain corpus.

2.1. Model overview

The model structure of incremental pre-training and fine-tuning based on domain corpus is shown in Fig. 1. The model includes three parts: pre-training, fine-tuning and downstream task testing.

Pre-training: inject the domain corpus into the model and adjust the network parameters to make the model adapt to the tasks of the target domain. In the pre-training stage, the input corpus is copied first, and then the mask is generated in a random or predefined way. Finally, the original and masked corpus are input into the model for self-supervised training. The model predicts the masked content according to the context to learn the target domain’s knowledge.

Fine-tuning: adapt the model to specific downstream tasks through training. Firstly, building the input with form as sentence pair and label. Secondly, the transformer encoder can let the model’s attention to the sentence pair simultaneously and then predict the semantic similarity of sentence pair. Finally, we can adjust the network parameters by loss, calculating by label and predicting results. We can obtain the model adapt text Semantic matching task through the steps above.

Downstream task test: After adjusting the parameters of above two modules, this module is used to test the model effect in practical application of target domain.

2.2. Mask self-supervised training

In target domain, owning to high variety and proportion of proper nouns, we propose a method that combines random and domain vocabulary masks in pretrain. In our method, original and masking corpus are input models for self-supervised learning, which can enhance the model’s modeling ability of vital information in downstream tasks. Combining random and domain dictionary masks can maximize learning the knowledge from pre-train datasets.

Random mask method

In the data preprocessing stage, for a given sentence sequence with length T, $x = [x_{1}, \dots, x_{T}]$ , Randomly select 15% of tokens in sentence x, keep 10% unchanged and replace 80% with ‘[MASK]’, 10% with a random token.The sentences after masking are denoted as $x^{'}$ . To input $x^{'}$ and x into BERT, after multi-layer Transformer coding, can obtain the hidden vector sequence $h (x^{'}) = [h_{1}, \dots, h_{T}]$ . Equation (3) is used to calculate the loss value of mask training to update the parameters.

Domain dictionary mask method

Fig. 2.

Domain dictionary mask method.

The domain dictionary masking method is shown in Fig. 2. Using the method of keyword matching, for the sentence sequence $x = [x_{1}, \dots, x_{T}]$ with input length T, to find the domain specialized vocabulary sequence $x_{i : j}$ and replace $x_{i : j}$ with $j - i + 1$ ‘[MASK]’ to get the masked sentence $x^{'}$ . $x^{'}$ is input into BERT as a label with x. After multi-layer Transformer coding, the hidden vector sequence of the last layer is obtained as $h (x^{'}) = [h_{0}, \dots, h_{T + 1}]$ . The cross-entropy loss function shown in equation (3) is used to calculate the loss value and update parameters for $h (x^{'}) i : j$ in $h (x^{'})$ . $\begin{matrix} (3) & L_{MLM} = - \sum_{t = j}^{j} logp (x_{t} | x^{'}) = - \sum_{t = j}^{j} \log \frac{\exp (h {(x^{'})}_{t}^{T}) e (x_{t})}{\sum_{x_{k} \in V} \exp (h {(x^{'})}_{t}^{T} e (x_{t}))} \end{matrix}$

In equation (3), $x^{'}$ is the masked corpus, $x_{t}$ is the masked token, and t is the index of the masked token. For the domain dictionary masking method, t is $i : j$ , $e (x_{t})$ is the word embedding of token $x_{t}$ , V is the vocabulary of the model, and $h {(x^{'})}_{t}$ is the subsequence of the hidden vector sequence $h (x^{'})$ .

2.3. Fine-tuning corpus preprocessing

In the fine-tuning stage, to solve insufficient corpus in target domain, we propose using public datasets similar to target domain datasets to train the model. In this paper, our target domain is financial company loan Q&A, so we choose the public datasets BQ to be source domain datasets, similar to our target domain. To solve the problem that many words in the target domain do not appear in the source domain, we preprocess the datasets with manual screening, as shown in Table 1. At first, we select the sentences that contain proper nouns in source domain. Then, we replace the proper nouns with conceptual words. This way, we can reduce the proprietary nouns and enhance data generalization in similar domains.

Table 1
Dataset preprocessing

Before replacement After replacement

Hello, why hasn’t my microloan been opened Hello, why hasn’t my loan been opened

I want to open a microloan. I wonder what I should do to prepare the materials I want to open a loan. I wonder what I should do to prepare the materials

how do we show that the approval requirements of Weizhong bank are not how do we show that the approval requirements of the bank are not

Why did it fail to meet the approval of Weizhong bank Why did it fail to meet the approval of the bank

Before replacement	After replacement
Hello, why hasn’t my microloan been opened	Hello, why hasn’t my loan been opened
I want to open a microloan. I wonder what I should do to prepare the materials	I want to open a loan. I wonder what I should do to prepare the materials
how do we show that the approval requirements of Weizhong bank are not	how do we show that the approval requirements of the bank are not
Why did it fail to meet the approval of Weizhong bank	Why did it fail to meet the approval of the bank

3. Experiment and analysis

3.1. Experimental data

There are two types of tasks and six datasets in the experiment. Tasks include text classification and semantic matching. Datasets are shown in Table 2. Lcqmc is a universal domain dataset. BQ and Ours_pre, Ours_dev, Ours_test is a professional domain dataset.

3.2. Experimental setup and evaluation criteria

The experimental environment is shown in Table 3. The random seeds are set as 7. We use Adamw to be the model optimizer and set the gradient attenuation factor as 0.9, the square gradient attenuation factor as 0.99, the initialization learning rate as $2 * 10 - 5$ and the weight-decreasing parameter as 0.1. In pre-training and fine-tuning stages, we set epochs as 100 and 10, respectively. In order to shorten the training time, we set batch size as 64 both pre-training and fine-tuning stages.

Table 2
Dataset introduction

Dataset name Related introduction

LCQMC Question matching large-scale Chinese dataset contains a training set, verification set and test set, with a total of 260,000 question pairs.

BQ Question matching of a bank customer service contains a training set, validation set and test set, a total of 120,000 question pairs.

BQboost Based on BQ, manual cleaning replaces the proper nouns with conceptual words

Ours_pre Product manual and activity instructions from a financial company loan business. There are eighty documents in total.

Ours_dev The customer service question matching dataset from a financial company consists of standard questions and user questions, filtered and manually labeled. There are 2000 question pairs in total.

Ours_test User data and standard problem set from a financial bank loan business, a total of 800 and 1700 problems.

Dataset name	Related introduction
LCQMC	Question matching large-scale Chinese dataset contains a training set, verification set and test set, with a total of 260,000 question pairs.
BQ	Question matching of a bank customer service contains a training set, validation set and test set, a total of 120,000 question pairs.
BQboost	Based on BQ, manual cleaning replaces the proper nouns with conceptual words
Ours_pre	Product manual and activity instructions from a financial company loan business. There are eighty documents in total.
Ours_dev	The customer service question matching dataset from a financial company consists of standard questions and user questions, filtered and manually labeled. There are 2000 question pairs in total.
Ours_test	User data and standard problem set from a financial bank loan business, a total of 800 and 1700 problems.

Table 3

Experimental environment

Experimental environment	Environment configuration
Operating system	Ubuntu 20.04.1 LTS
CPU	AMD Ryzen Threadripper
CPU	3970X 32-Core Processor
GPU	TITAN RTX
CUDA edition	11.0

We use accuracy as an evaluation criterion for text classification and semantic matching tasks. The definition of indicators is shown in equations (4)–(5). Equation (4) is used to calculate the accuracy of the text binary classification task, and equation (5) is used to calculate the accuracy of the semantic similarity task. $\begin{matrix} (4) & A {cc}_{1} = \frac{TP + FN}{ALL} \end{matrix}$

In equation (4), TP, the problem pair label and the actual prediction of the model also is 1; FN, the problem pair label is 0, and the actual prediction of the model is 0; All is the number of all test samples. $\begin{matrix} (5) & A {cc}_{2} = \frac{M}{ALL} \end{matrix}$

In equation (5), All is the number of all test samples; M is the number of matching correct questions. m inputs and n questions in the question base form $m * n$ sentence pairs. Select 1 and 5 sentence pairs with the highest probability of 1 prediction label for m questions, and count the number of sentence pairs with similar actual semantics as m.

3.3. Experimental results and analysis

The experimental dataset is divided into four parts: pre-training set, training set, verification set, and test set. Among them, Ours_pre dataset is used for model pre-training, training set of LCQMC and BQ dataset is used for model fine-tuning, Ours_dev, and Ours_test is used to verify the performance improvement of the model in downstream tasks.

Table 4
Mask experiment

Methods LCQMC_test BQ_dev Ours_dev Ours_test

Top1 Top5

BERT 86.92 58.03 76.26 54.88 66.33

+Random 86.74 59.99 80.09 56.23 70.71

+Dict 85.93 57.73 80.13 56.57 71.72

+Random_Dict 86.42 59.39 81.52 59.26 77.10

Methods	LCQMC_test	BQ_dev	Ours_dev	Ours_test
BERT	86.92	58.03	76.26	54.88	66.33
+Random	86.74	59.99	80.09	56.23	70.71
+Dict	85.93	57.73	80.13	56.57	71.72
+Random_Dict	86.42	59.39	81.52	59.26	77.10

As shown in Table 4, BERT means that LCQMC is directly used to fine-tune the original BERT model. +Random, +Dict, +Random_Dict implies that based on the original BERT, it is used for Ours_pre dataset is incrementally trained by using random mask method, domain dictionary mask method, and a combination of the two approaches, and finally fine-tuned on the LCQMC dataset.

The experimental results show that our data is used in the pre-training stage. The pre dataset will damage the generalization of the BERT model, and the accuracy of LCQMC test set will be reduced by 0.18%–0.99%. But, in the downstream tasks, the accuracy of the BERT model pre-trained by +Dict method is the lowest in BQ verification set, and the accuracy of Ours_dev and Ours_test datasets is better than +Random. This shows that the BERT model based on domain dictionary enhancement reduces the performance of downstream tasks in the domain with large differences in professional terms and improves the modeling ability of crucial information of downstream tasks in the target domain. +Random_Dict combining + Random and +Dict method on Ours_dev and Ours_test dataset performs best, and the accuracy is improved by 5.26%, 4.38%, and 10.77%, respectively, compared with BERT. It shows that combining random and dictionary masks can maximize the domain knowledge in the pre-training corpus.

Table 5

Continuous training of corpus in similar domain

Methods	BQ_dev	Ours_dev	Ours_test

			Top1	Top5
+Random_Dict	–	82.73	62.29	77.10
+BQ	85.15	92.15	52.86	76.09
+BQboost	84.36	92.87	64.31	78.45

As shown in Table 5, +BQ and +BQboost indicate that basis on +Random_Dict uses BQ and BQboost datasets for fine-tuning. Ours_test dataset finetune on BERT model, the accuracy of Top1 and Top5 decreased by 9.43% and 1.01%. It indicates that the deviation between target and similar domains will affect the effect of the downstream task. The accuracy of +BQboost on Ours_dev and Ours_test datasets is better than +Random_Dict and +BQ. It indicates that the model’s performance in downstream tasks of similar domains can be improved by cleaning the professional domain dataset and fine-tuning the model.

4. Conclusion

In the practical application of the BERT model, there are some problems, such as insufficient pre-training corpus, quantity of proper nouns in target domain, and expression with colloquializes. The enhancement method for knowledge based on domain dictionary mask is proposed to solve the problems. Random masking and domain dictionary masking are used to mask target domain corpus. After masking, the corpus is input to the model for self-supervised learning, which enhances the model’s ability for professional domain terms in practical application. After pre-training, the model is fine-tuned by the universal dataset LCQMC and domain dataset BQboost. Finally, the user data of a financial company are used for testing. Experimental results show that the knowledge-enhanced BERT compared with original BERT increases by 12.12% on average in downstream tasks.

Our research is only a single corpus from a similar domain to fine-tune the model. Our future work will use multiple datasets from different source domains and comprehensively consider the semantic information hidden in BERT’s underlying and high-level networks [6] to obtain better results.

Footnotes

Acknowledgements

The study is funded by the Guangdong Science and Technology Plan Project (2021B1212100004, 2019B010139001), Guangzhou Science and Technology Plan Project (201902020016), and Guangdong Natural Science Fund Project (2021A1515011243).

Conflict of interest

None to report.

References

S.S.

Ali Zaidi,

M.M.

Fraz,

Shahzad and

Khan, A multiapproach generalized framework for automated solution suggestion of support tickets, Int. J. Intell. Syst. 37 (2022), 3654–3681. doi:10.1002/int.22701.

Chen,

Liu et al., The BQ corpus: A large-scale domain-specific Chinese corpus for sentence semantic equivalence identification, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 4946–4951. doi:10.18653/v1/D18-1536.

Chen and

X.Y.

Qiu, Research on multi task text semantic matching method combined with self supervised learning, Journal of Peking University: Natural Science Edition 58(1) (2022), 8 (Chinese).

Devlin,

M.W.

Chang,

Lee et al., BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint, 2018. arXiv:1810.04805.

Feng and

Li, Pre training paradigm in natural language processing, Foreign Language Studies 38(01) (2021), 1–14+112 (Chinese).

Jawahar,

Sagot and

Seddah, What does BERT learn about the structure of language? in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.

Jiang, Exploration, practice and development strategy of artificial intelligence application in China’s banking industry, Southwest Finance 2 (2018), 44–49 (Chinese).

Lavrinenko and

Shmatko, Twenty-first century skills in finance: Prospects for a profound job transformation, Foresight and STI Governance 13(2) (2019), 42–51. doi:10.17323/2500-2597.2019.2.42.51.

Li,

Ren,

Sun,

Li and

He, Intelligent customer service model based on multi-featurefusion. Computer System Application 30(07) (2021), 239–245 (Chinese). doi:10.15888/j.cnki.csa.008068.

10.

Liu,

Chen,

Deng et al., LCQMC: A large-scale Chinese question matching corpus, in: Proceedings of the 27th International Conference on Computational Linguistics, 2018, pp. 1952–1962.

11.

Liu,

Ott,

Goyal et al., RoBERTa: A robustly optimized BERT pretraining approach, 2019.

12.

Mikolov,

Sutskever,

Chen et al., Distributed representations of words and phrases and their compositionality, Advances in Neural Information Processing Systems 26 (2013), 3111–3119.

13.

Qiao,

Liu and

Qi, Similarity weighted proximity retrieval method for query words, Pattern Recognition And Artificial Intelligence, 26(02) (2013), 189–194 (Chinese). doi:10.16451/j.cnki.issn1003-6059.2013.02.006.

14.

S.E.

Robertson,

Walker,

Jones et al., Okapi at TREC-3, in: Proc. of the 3rd Text Retrieval Conference, Gaithersburg, Zimbabwe, 1996, pp. 109–126.

15.

Shi,

Xu and

Yang, Overview of TFIDF algorithm, Computer Application 29(B06) (2009), 5 (Chinese).

16.

Song,

Guo,

He,

Liu,

Zhao and

Liu, Sentence matching pre training model based on dynamic word masking, Chinese Journal of information 35(11) (2021), 43–50 (Chinese).

17.

Sun,

Qiu,

Xu et al., How to Fine-Tune BERT for Text Classification? Springer, Cham, 2019.

18.

P.D.

Turney and

M.L.

Littman, Corpus-based learning of analogies and semantic relations, Machine Learning 60 (2005), 251–278.

19.

Vaswani,

Shazeer,

Parmar et al., Attention is all you need, in: Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.

20.

Wang, Research on the application of Telecom Science semantic similarity calculation method in financial intelligent customer service. Northeast Forestry University (Chinese).

21.

Yang,

Ouyang,

Fu and

Peng, A decision-making algorithm for online shopping using deep-learning-based opinion pairs mining and q-rung orthopair fuzzy interaction Heronian mean operators, Int. J. Intell. Syst. 35 (2020), 783–825. doi:10.1002/int.22225.

22.

Yu and

Wang, Research on multimedia intelligent customer service system under the background of mobile Internet. Telecommunication Science 29(S1) (2013), 164–168 (Chinese).

23.

Zhang,

Xue,

Ma,

Zhang,

Liang and

Y.-A.

Tan, Hybrid sequence-based Android malware detection using natural language processing, Int. J. Intell. Syst. 36 (2021), 5770–5784. doi:10.1002/int.22529.

24.

Zhu,

Zhou,

Ye,

Cheng and

Li, Resource allocation in IoT edge computing via concurrent federated reinforcement learning, IEEE Internet of Things Journal 9(2) (2022), 1414–1426. doi:10.1109/JIOT.2021.3086910.

Knowledge enhancement BERT based on domain dictionary mask

Abstract

Keywords

1. Introduction

2. Knowledge enhancement of domain dictionary mask BERT

2.2. Mask self-supervised training

3.1. Experimental data

3.2. Experimental setup and evaluation criteria

Table 4 Mask experiment Methods LCQMC_test BQ_dev Ours_dev Ours_test Top1 Top5 BERT 86.92 58.03 76.26 54.88 66.33 +Random 86.74 59.99 80.09 56.23 70.71 +Dict 85.93 57.73 80.13 56.57 71.72 +Random_Dict 86.42 59.39 81.52 59.26 77.10

Footnotes

Acknowledgements

Conflict of interest

References

Table 4
Mask experiment

Methods LCQMC_test BQ_dev Ours_dev Ours_test

Top1 Top5

BERT 86.92 58.03 76.26 54.88 66.33

+Random 86.74 59.99 80.09 56.23 70.71

+Dict 85.93 57.73 80.13 56.57 71.72

+Random_Dict 86.42 59.39 81.52 59.26 77.10