Aspect-based sentiment analysis via relation-aware collaborative learning

Abstract

Aspect-based sentiment analysis (ABSA) contains three subtasks, namely aspect term extraction, opinion term extraction and aspect-level sentiment classification. In order to make full use of the relationship between the three subtasks, some recent studies have successfully tried to use a unified framework to solve the problem of aspect-based sentiment analysis. However, these studies have not yet integrated domain knowledge into the model. Inspired by the post-training task, we propose a joint model (RACL-BERT-PT). This model combines the pre-training model BERT-PT with domain knowledge and the unified joint training framework RACL. The experimental results show that our model has achieved better results than previous experiments on three public data.

Keywords

Aspect-based sentiment analysis post-training domain knowledge

1 Introduction

The aspect-based sentiment analysis (ABSA) task was first proposed by [1]. It is a fine-grained sentiment analysis whose purpose is to determine the sentiment polarity of one or more objects in a single sentence. A specific example of this task, from a review of a restaurant, "The environment of this restaurant is good, but the food is too unpalatable". It is not appropriate to perform sentiment analysis on this sentence alone because there are two aspects: environment and food. The evaluation of the environment is positive, and the evaluation of food is negative. ABSA usually consists of three subtasks, aspect term extraction(AE), opinion extraction(OE) and aspect level sentiment classification(SC). AE aims to extract a set of aspect terms {"environment ", "food"}, OE aims to extract a set of opinion terms {"good", "unpalatable"}. Meanwhile, it is expected for SC to assign a sentiment polarity "negative" and "positive" to the aspect "environment" and "food", respectively. At present, some researchers have developed a series of unified frameworks to solve the ABSA problem using the interactive relationship of the three tasks. For example, the elegant wording of opinions can be used as evidence of the aspect word "environment" because the aspect term must use the corresponding opinion term. Specifically, they use the relationship of the three tasks to enhance each other’s information.

In order to add domain knowledge to the pre-training model, and in order to solve the problem that the small data set cannot be pre-trained, some researchers have proposed a post-training method based on the BERT model. Specifically, they propose a novel joint post-training technique that takes BERT’s pre-trained weights as the initialization for basic language understanding and adapt BERT with both domain knowledge and task knowledge for the domain set. This technique leverages supervised (yet out-of-domain) MRC data [2], where the former enhances domain-awareness. MRC here refers to machine reading comprehension, RRC refers to review reading comprehension. The RRC data set is constructed from a task they proposed, and the RRC task is designed to match appropriate answers from the questions.

The unified training framework does not consider integrating domain knowledge, and BERT based on the post-training method does not consider the correlation of the three subtasks. Therefore, the two methods are combined to make full use of domain knowledge and information interaction between tasks to improve the accuracy of aspect sentiment classification. As a combined method, our contributions are as follows:

We use the relationship between ABSA sub-tasks and the pre-training model combined with domain knowledge to achieve the best results for AE, OE, and SC tasks;

It proves that domain knowledge is beneficial to the effect of AE, OE, and SC tasks.

2 Related work

Aspect extraction is an important task in sentiment analysis [1] and has many applications [3, 4]. We analyze the latest existing method research and comparison and propose the motivation of our method.

Separate Methods Two important tasks of ABSA are aspect extraction and aspect sentiment classification. The former aims to extract aspects, and the latter aims to identify aspect polarity. Some researchers have developed different methods for the two tasks. For AE tasks [5 –14]. These works achieved relatively good results that year. And for SC tasks [15 –21], these methods have reached the best level of the year. Some of them use a pipeline method to divide the extraction (AE, OE) task and the classification (SC) task into two steps, but the errors of the upstream AE and OE tasks will be propagated to the downstream task SC, which we do not want to see the result. Moreover, the results of the pipeline method used are not very satisfactory.

Unified Methods In order to take advantage of the relationship between extraction (AE, OE) and (SC), some recent research views solve the ABSA task within a unified framework. There are two main types: joint training [22 –25] and folding labels [12 , 26]. The folded label method constructs a folded label, such as {B-senti, I-senti, O}. The sub-tasks need to share all trainable features without distinction, which is likely to confuse the learning process. In the joint training method, each subtask has an independent label and has shared and private features. The purpose of the joint training method is to use the interaction between different subtasks explicitly. Nevertheless, these methods do not incorporate domain knowledge.

Pre-trained model with knowledge In order to obtain a better word vector representation, there have been a lot of attempts to incorporate the knowledge graph into the pre-training model [2, 27]. Specifically, three methods are mainly used to incorporate knowledge into the pre-training model:

They are to use the knowledge graph representation vector as the feature input;

Design a new pre-training task;

Add additional modules.

Among them, the work of [2] is representative. A Post-training method is proposed to incorporate domain knowledge into the pre-training model. The best results have been achieved on the two public data sets of ABSA.

Our work combines the above methods. For this reason, we use a pre-trained model [2] incorporating knowledge on a unified framework [28] to improve the results of the ABSA task.

3 Methodology

3.1 Task definition

We give a sentence S = w₁, . . . , w_i, . . . , w_n in order to solve three subtasks AE, OE and SC. Specifically, it is formulated into three sequence labeling problems.

AE task AE aims to extract aspects from sequence tags S, and predict the aspect $Y^{A} = y_{1}^{A},$ $. . ., y_{i}^{A}, . . ., y_{n}^{A}$ , where $y_{i}^{A} \in$ {B, I, O} denotes an aspect term about the beginning, inside, and outside.

OE task OE aims to extract aspects from sequence tags S, and predict the aspect $Y^{O} = y_{1}^{O},$ $. . ., y_{i}^{O}, . . ., y_{n}^{O}$ , where $y_{i}^{O} \in$ {B, I, O} denotes an opinion term about the beginning, inside, and outside.

SC task SC aims to classify the sentiment of the target sequence S and predict the category $Y^{S} = y_{1}^{S}, . . ., y_{i}^{S}, . . ., y_{n}^{S}$ , where $y_{i}^{S} \in$ {pos, neu, neg} denotes the positive, neutral, and negative sentiment polarities towards each word.

3.2 Model architecture

As shown in Fig. 1, our model mainly consists of two parts: RACL and BERT-PT. This chapter will introduce our two core modules RACL and BERT-PT, and how to combine them.

Fig. 1

Interactive relations among subtasks in ABSA.

3.2.1 BERT-PT model

Review Reading Comprehension In order to solve a newly proposed reviewing reading comprehension problem(RRC), [2] explored a novel post-training method in the BERT pre-training model and achieved good results on RRC, AE, and SC tasks. Inspired by them, we added this novel pre-training model to the model framework and used it to encode sentences. Next, we will briefly introduce the two core parts of the pre-training model, the RRC task and the post-training method, to explain the help it brings to the ABSA task. As shown in Fig. 2, we are given a question q = {q₁, . . . , q_m} asking for an answer from a review d = {d₁, . . . , q_n}. Training the RRC model involves minimizing the loss that is designed as the averaged cross entropy on the two pointers:

Fig. 2

BERT-PT model, where [CLS] is a dummy token not used for RRC and [SEP] is intended to separate q and d.

$\begin{matrix} Min (L_{RRC}) = - \frac{\sum {logl}_{1} (s) + \sum {logl}_{2} (e)}{2} \end{matrix}$ (1) where l₁ (s) and l₂ (e) are one-hot vectors representing the ground truths of pointers.

The training process is as follows, we first obtain the hidden representation as h = BERT (x) ∈ R^r_h*|x|, where r_h is the size of the hidden dimension, and |x| is the length of the input sequence x = ([CLS] , q₁, . . . , q_m, [SEP] , d₁, . . . , d_m, [SEP]), where [CLS] is a dummy token not used for RRC and [SEP] is intended to separate q and d. Then the hidden representation is passed to two separate dense layers followed by softmax function: l₁ = Softmax (W₁ · h + b₁) and l₂ = Softmax (W₂ · h + b₂). Finally, the output is a span across the position in d (after the [SEP] token of the input), indicated by two pointers (indexes) s and e computed from l₁ and l₂. The specific mathematical expressions of l₁ and l₂ are:

$\begin{matrix} s = \underset{idx [SEP] < s < | x |}{Argmax} (l_{1}) \end{matrix}$ (2) $\begin{matrix} e = \underset{idx [SEP] < e < | x |}{Argmax} (l_{2}) \end{matrix}$ (3) where idx [SEP] is the position of token [SEP].

Post-training Because the RRC task may require much effort to cover a wide range of data sets, and this will pay a high price, which is an undesirable result. Therefore, this novel method (post-training) proposed by [2] solves this problem. Reducing the bias introduced by the edge of non-review knowledge (because BERT is pre-trained unsupervised on Wikipedia) and fusing domain knowledge and task knowledge into the model. They are inspired by machine reading comprehension (MRC is a routine task that can solve almost all document content questions) because large-scale machine reading supervision corpus may benefit AE and SC tasks.

Specifically, using two pre-training goals: masked language model (MLM) and next sentence prediction(NSP). The training goals are as follows:

$\begin{matrix} Min (L_{DK}) = L_{MLM} + L_{NSP} \end{matrix}$ (4) where L_MLM is the loss function of MLM and L_NSP is the loss function of NSP. The former predicts randomly masked words, and the latter detects whether two sides of the input are from the same document or not. The input format for training is: S = ([CLS] , x_1:m, [SEP] , x_m+1:n, [SEP]), where x_1:n is a document(with randomly masked words) split into two sides x_1:m and x_m+1:n. In order to perform post-training on BERT, the SQuAD dataset is used. We will define L_MRC as the loss on this data set, which is similar to the loss(L_RRC) of RRC. Therefore, the final training target representation for the post-training task is Min (L) = L_DK + L_MRC. By minimizing these losses, BERT has a lot of domain knowledge about RRC tasks, which is beneficial to AE, OE, and SC tasks.

Proof of post-training As there are no existing datasets for RRC and to be consistent with existing research on sentiment analysis, [2] adopt the laptop and restaurant reviews of SemEval2016 Task5 as the source to create datasets for RRC. To prove the effectiveness of the post-training method, the following baseline methods are used: DrQA is a baseline from the document reader of DrQA [29].

DrQA+MRC is derived from the above baseline with official pre-trained weights on SQuAD.

BERT leverages the vanilla BERT pre-trained weights and fine-tunes on the task.

BERT-MRC post-trains BERT’s weights on SQu-AD and then fine-tunes on the task.

BERT-PT post-trains BERT’s weights using the joint post-training algorithm and then fine-tunes on the task.

The evaluation index is a perfect match of EM and F1 scores. EM requires a string that exactly matches the answer with the marked answer range. The F1 score is the average F1 score of a single answer. Figure 1 show that BERT-PT is more effective than the baseline method on reading comprehension tasks, which shows the benefits of having two kinds of knowledge. In the experimental part, we will continue to verify the advantages of BERT-PT on AE, OE, and SC tasks in combination with this method.

3.2.2 RACL model

As shown in Fig. 3, a single RACL contains three modules, each of which is designed for corresponding subtasks. As shown in Fig. 4, the input of each module accepts BERT-PT or BERT shared representation from the underlying sentence coding module. The three modules encode their respective task-oriented features, and then they carry out relational, collaborative learning through the four propagation relations R1,...,R4. Finally, the three modules predict the corresponding label sequences Y^A, Y^O and Y^S. However, the AE, OE, and SC modules in the single-layer RACL model can only extract lower language features, so we, like [28], stack RACL to multiple layers to obtain higher-level semantic features.

Fig. 3

RACL model.

Fig. 4

BERT-PT for RACL input

Shared-Private Scheme After the RACL module receives the sequence(S_e) of the generated word vector E = e₁, . . . , e_i, . . . , e_n from the underlying coding module BERT-PT or the BERT module (S→ [] E), we believe that different subtasks should focus on different features in the training samples. Therefore, we extracted two features: shared features and private features. Specifically, to extract shared features, we feed each e vector in the sequence E directly into a fully connected layer to generate a sequence of shared vectors H = h₁, . . . , h_i, . . . , h_n ∈ R^d_h×n (E→ [] H).

$\begin{matrix} F^{A} : H \to [] CNN X^{A}, X^{A} \in R^{d_{c} \times n} \end{matrix}$ (5)

$\begin{matrix} F^{O} : H \to [] CNN X^{O}, X^{O} \in R^{d_{e} \times n} \end{matrix}$ (6)

In order to extract the private features of the subtasks, we use the CNN proposed by [30] as the encoder function F here. For the AE and OE tasks, we directly extract the local features X^A and X^O (as shown in formula 5, 6, where d_c and d_e represents the word dimension, n represents the number of words), while for the SC task, we also need to consider the semantic information related to the extraction of its context. Therefore, it is necessary to use the semantic relationship between the attention query mechanism and the context feature. The formula is as follows:

$\begin{matrix} F^{ctx} : H \to [] CNN X^{ctx}, X^{ctx} \in R^{d_{h} \times n} \end{matrix}$ (7)

$\begin{matrix} d_{s_{i, j}} = ((h_{i})^{T} \times X_{j}^{ctx}) \cdot [\log (2 + | i - j |)]^{- 1} \end{matrix}$ (8)

$\begin{matrix} M_{i, j}^{ctx} = \frac{\exp (d_{s_{i, j}})}{\sum_{k = 1}^{n} \exp (d_{s_{i, k}})} \end{matrix}$ (9) $\begin{matrix} X_{i}^{S} = \sum_{j = 1}^{n} (M_{i, j}^{ctx} \cdot X_{j}^{ctx}) \end{matrix}$ (10)

where d_{s
_i,j} represents the strength of dependence between the i-th query and the j-th context word, and $M_{i, j}^{ctx}$ the normalized attention weight of d_{s
_i,j}. We believe that the two adjacent words should make a more significant contribution to the sentiment polarity, so we add the absolute distance coefficient log (2 + |i - j|) ^-1. Finally, the SC-oriented global feature $X_{i}^{S}$ is obtained through the weighted sum of context features.

In addition to the use of shared and private solutions, the use of all relationships between subtasks can also enhance the effectiveness of tasks. Specifically, how to use the relationship between R1, R2, R3, and R4.

R1 represents the relationship between AE and OE, and AE and OE can help each other. For example, to describe food, We use words "delicious" to modify, rather than words such as "elegance" and vice versa. To model R1, use the semantic relationship between AE and OE to exchange useful information. For AE and OE, the semantic relationship between words is defined as follows:

$\begin{matrix} {ao}_{i, j} = (X_{i}^{A})^{T} \cdot X_{j}^{O} \end{matrix}$ (11) $\begin{matrix} {oa}_{i, j} = (X_{i}^{O})^{T} \cdot X_{j}^{A} \end{matrix}$ (12) $\begin{matrix} M_{i, j}^{O 2 A} = \frac{\exp ({ao}_{i, j})}{\sum_{k = 1}^{n} \exp ({ao}_{i, k})} \end{matrix}$ (13) $\begin{matrix} M_{i, j}^{A 2 O} = \frac{\exp ({oa}_{i, j})}{\sum_{k = 1}^{n} \exp ({oa}_{i, k})} \end{matrix}$ (14)

where ao_i,j represent the interaction between AE and OE, and $M_{i, j}^{A 2 O}$ is the normalized attention weight of ao_i,j.

For the words w_i in OE, we can perform weighted summation through the semantic relations of all words in AE and obtain useful clues X^A2O from AE. On the contrary, for the word w_i in AE, we use the same method to obtain the clue X^O2A. Then, we connect the AE-oriented feature X^A and the OE-oriented feature X^O with the corresponding useful clues X^O2A and X^A2O respectively to form the final task feature representation and input it to the fully connected layer to predict the label. The formula is as follows:

$\begin{matrix} X_{i}^{O 2 A} = \sum_{j = 1}^{n} (M_{i, j}^{O 2 A} \cdot X_{j}^{O}) \end{matrix}$ (15) $\begin{matrix} X_{i}^{A 2 O} = \sum_{j = 1}^{n} (M_{i, j}^{A 2 O} \cdot X_{j}^{A}) \end{matrix}$ (16) $\begin{matrix} Y^{A} = Softmax (W^{A} \cdot (X^{A} \oplus X^{O 2 A})) \end{matrix}$ (17) $\begin{matrix} Y^{O} = Softmax (W^{O} \cdot (X^{O} \oplus X^{A 2 O})) \end{matrix}$ (18)

where W^A (W^O) ∈ R^3×2d_c is a transformation matrix, Y^A (Y^O) is the predicted tag sequence of AE(OE).

R2 is the triadic relation between SC and R1. The key to the SC task is to determine the dependencies of aspects and contexts. For example, when predicting the sentiment polarity of food, the context "sufficient" and "delicious" play an important role so that R1 can help SC tasks. We will use M^O2A as the representation of R1, and the specific operation of R2 is defined as:

$\begin{matrix} M_{i, j}^{ctx} \leftarrow M_{i, j}^{ctx} + M_{i, j}^{O 2 A} \end{matrix}$ (19)

M^O2A reflects the terminology and context dependence from the extraction perspective, while M^ctx reflects the dependence between them from the classification perspective. Of course, we should have the same effect in theory if we change M^O2A to M^A2O.

R3 represents the binary relationship between SC and OE. Because opinions usually express the sentiment polarity, for example, being "too delicious" is usually a positive factor. More attention should be paid to the opinion items extracted by the OE task in the SC task. Similar to R2 and R3, the formula is defined as follows: $\begin{matrix} M_{i, j}^{ctx} \leftarrow M_{i, j}^{ctx} + P_{y_{j}^{O} \in {B, I}} \cdot [\log_{2} (2 + | i - j |)]^{- 1} \end{matrix}$ (20)

where P represents the probability value that we give.

By doing this, the opinion item can obtain a greater weight representation in the sentiment prediction matrix. Then we will recalculate the feature X^S of the SC in formula 10, as in formula 21, express the connection of H and X^S as the final feature of SC for the final sentiment polarity prediction: $\begin{matrix} Y^{S} = Softmax (W^{S} \cdot (H \oplus X^{S})) \end{matrix}$ (21)

where W^S ∈ R^3×2d_h is a transformation matrix, Y^s ∈ R^3×n is the predicted tag sequence of SC.

R4 represents the binary relationship between SC and AE. Only aspect words have their corresponding sentiment polarity. For example, in food reviews, "food" and "environment" usually have specific sentiment polarity. It shows that the AE task is helpful to the supervised training of the SC task. Therefore, we directly use the label YA corresponding to AE to improve the marking process in SC, specifically as follows: $\begin{matrix} y_{i}^{S} \leftarrow I (y_{i}^{A}) \cdot y_{i}^{S} \end{matrix}$ (22)

where $I (y_{i}^{A})$ equals to 1 if w_i is an aspect term and to 0 if not. During the training process, only the predicted labels for the real terms are calculated.

3.2.3 Model stacking

As shown in Fig. 4, we use BERT-PT for sentence encoding as the input of the first RACL module because a single RACL module may only extract lower semantic features, so we stack the RACL modules. Specifically, the features extracted by the RACL module of the first layer are used as the input of the RACL module of the second layer, thereby stacking the RACL module to the L layer. Finally, the output results of each layer are averaged and pooled as the prediction result.

$\begin{matrix} Y^{T} = Avg ([Y^{T (1)}, Y^{T (2)}, . . ., Y^{T (L)}]) \end{matrix}$ (23)

Where T∈ { AE, OE, SC } represents a specific subtask, and L is the number of layers. In actual experiments, we also use 6 layers (L=6).

4 Experiment

4.1 Training procedure

The list of pipelines from the input layer to the last layer is shown in Table 2. After generating the tag sequences Y^A, Y^O and Y^S for the sentence S_e, one of the examples shown in Table 3, we compute the cross-entropy loss of each subtask:

$\begin{matrix} L^{T} = - \sum_{i = 1}^{N} \sum_{j = 1}^{J} y_{ij}^{T_{p}} \cdot \log (y_{ij}^{T_{g}}) \end{matrix}$ (24) $\begin{matrix} L = \sum L^{T} + λ \cdot L^{R} \end{matrix}$ (25)

where T∈ {A, O, S} denotes the subtask, N is the length of S_e, J is the category of labels, $y_{i}^{T_{g}}$ and $y_{i}^{T_{p}}$ are the predicted tags and ground truth labels. The final loss L of RACL-BERT-PT is the combination of the loss for subtasks and the loss for regularization, as shown in formula 24, where λ is a coefficient. We then train all parameters with back propagation.

Table 1

RRC in EM(Exact Match) and F1

Domain	Laptop		Restaurant
Methods	EM	F1	EM	F1
DrQA	38.26	50.99	49.52	63.73
DrQA+MRC	40.43	58.16	52.39	67.77
BERT	39.54	54.72	44.39	58.76
BERT-MRC	47.01	63.87	54.78	68.84
BERT-PT	48.05	64.51	59.22	73.08

Table 2

Pipeline of model

Task Name	Input	Layer Name	Output
AE \| OE	S _e	Embedding layer	E
	E	Hidden layer	H
	H	CNN layer	X^A \| X^O
	X^A \| X^O	Interaction layer	M^O2A \| M^A2O
	M^O2A \| M^A2O	Concat layer	X^O2A \| X^A2O
	X^O2A \| X^A2O	FC layer	Y^A \| Y^O
SC	S _e	Embedding layer	E
	E	Hidden layer	H
	H	CNN layer	X _ctx
	X _ctx	Attention layer	M _ctx
	M _ctx	Concat layer	X ^S
	X ^S	FC layer	Y ^S

Table 3

Input Example

S _e		This	is	a	consistently	great	place	to	dine	for	lunch
Y ^A ¹		0	0	0	0	0	0	0	1	0	1
Y ^O ²		0	0	0	0	1	0	0	0	0	0
Y ^S ³		0	0	0	0	0	0	0	1	0	3
Model Input	[CLS]	This	is	a	consistently	great	place	to	dine	for	lunch	[SEP]
Token ^*	101	2023	2003	1037	10862	2037	2173	2000	10672	2005	6266	102

¹ Y^A contains the aspect term tag sequences, where 0=O, 1=B, 2=I. ² Y^O contains the opinion term tag sequences, where 0=O, 1=B, 2=I. ³ Y^S contains the sentiment tag sequences, where 0=background, 1=positive, 2=negative, 3=neutral. ^* Token represents the word form for bert to perform token expressed.

4.2 Datasets and setting

Datasets Our three real-world datasets from the computer field and the restaurant field in the public datasets [31], and [32] verify the effectiveness of our model. For the opinion term label, someone has already annotated it [33]. We use 20% of the training set for adjusting hyperparameters and the remaining 80% for training. Table 4 shows the statistics of the data set.

Table 4
Datasets

Datasets Type Aspect Opinion

Restaurant14 Train 3699 3484

Test 1134 1008

Laptop14 Train 2373 2504

Test 654 674

Restaurant15 Train 1199 1210

Test 542 510

Datasets	Type	Aspect	Opinion
Restaurant14	Train	3699	3484
	Test	1134	1008
Laptop14	Train	2373	2504
	Test	654	674
Restaurant15	Train	1199	1210
	Test	542	510

To show more details of the data set, we randomly selected a real example in the data set, as shown in Table 3. We first add special tags [CLS] and [SEP] to the sentence, then we use Bert’s own vocabulary for token representation, and finally use it as the input of the model, as shown in Fig. 4. The three tags {Y^A, Y^O, Y^S} are used for fine-tuning the model.

Setting We will compare two pre-training models: the pre-training encoder BERT-Large in the general field and the pre-training encoder BERT-PT with domain knowledge for comparison. For BERT-Large, we set d_w to 1024, d_h = 400 and d_c = 256. For BERT-PT, we set d_w to 768 (because BERT-PT is post-training based on BERT-base), and other parameters are the same as BERT-Large. In addition, for the CNN used for feature extraction, the kernel size and layer number we set for the three data sets are {3, 3, 5} and {4, 3, 4} respectively. In the training process, in order to prevent over-fitting, we use the dropout strategy, and the dropout rate is set to 0.5. Finally, the Adam optimizer is used to train a fixed-period model at the learning rate 0.0001.

4.3 Baseline

The model that achieves the smallest loss on the development set is used to evaluate the test set. The evaluation index uses the F1 index to represent the performance of each subtask, and the specific results are shown in Table 5. In order to facilitate the comparison of experimental results to verify the effectiveness of the method, we use the following model for comparison.

Table 5
Comparison of different methods

Model Restaurant14 (Res14) Laptop14 (Lap14) Restaurant15 (Res15)

AE(F1) OE(F1) SC(F1) ABSA(F1) AE(F1) OE(F1) SC(F1) ABSA(F1) AE(F1) OE(F1) SC(F1) ABSA(F1)

M1^* MNN 83.05 84.55 68.45 63.87 76.94 77.77 65.98 53.8 70.24 69.38 57.9 56.57

E2E-TBSA 83.92 84.97 68.38 66.6 77.34 76.62 68.24 55.88 69.4 71.43 58.81 57.38

DOER 84.63 - 64.5 68.55 80.21 - 60.18 56.71 67.47 - 36.76 50.31

IMN-D 84.01 85.64 71.9 68.32 78.46 78.14 69.92 57.66 69.8 72.11 60.65 57.91

RACL-GloVe 85.37 85.32 74.46 70.67 81.99 79.76 71.09 60.63 72.82 78.06 68.69 60.31

M2^* SPAN-BERT 86.71 - 71.75 73.68 82.34 - 62.5 61.25 74.63 - 50.28 62.29

IMN-BERT 84.06 85.1 75.67 70.72 77.55 81 75.56 61.73 69.9 73.29 70.1 60.22

RACL-BERT-Large 86.38 87.18 81.61 75.42 81.79 79.72 73.91 63.4 73.99 76 74.91 66.05

M3^* RACL-BERT-PT 87.36 88.12 80.14 75.87 85.71 81.88 76.57 67.68 74.6 73.83 75.3 67.28

	Model	Restaurant14 (Res14)	Laptop14 (Lap14)	Restaurant15 (Res15)
		AE(F1)	OE(F1)	SC(F1)	ABSA(F1)	AE(F1)	OE(F1)	SC(F1)	ABSA(F1)	AE(F1)	OE(F1)	SC(F1)	ABSA(F1)
M1^*	MNN	83.05	84.55	68.45	63.87	76.94	77.77	65.98	53.8	70.24	69.38	57.9	56.57
	E2E-TBSA	83.92	84.97	68.38	66.6	77.34	76.62	68.24	55.88	69.4	71.43	58.81	57.38
	DOER	84.63	-	64.5	68.55	80.21	-	60.18	56.71	67.47	-	36.76	50.31
	IMN-D	84.01	85.64	71.9	68.32	78.46	78.14	69.92	57.66	69.8	72.11	60.65	57.91
	RACL-GloVe	85.37	85.32	74.46	70.67	81.99	79.76	71.09	60.63	72.82	78.06	68.69	60.31
M2^*	SPAN-BERT	86.71	-	71.75	73.68	82.34	-	62.5	61.25	74.63	-	50.28	62.29
	IMN-BERT	84.06	85.1	75.67	70.72	77.55	81	75.56	61.73	69.9	73.29	70.1	60.22
	RACL-BERT-Large	86.38	87.18	81.61	75.42	81.79	79.72	73.91	63.4	73.99	76	74.91	66.05
M3^*	RACL-BERT-PT	87.36	88.12	80.14	75.87	85.71	81.88	76.57	67.68	74.6	73.83	75.3	67.28

^*We will separate the GloVe-based method (M1) and the BERT-based method (M2) for fair comparison and the BERT-PT-base method (M3). The best scores are shown in bold, with "-" indicating the method does not include subtask OE.

MNN In order to overcome the error propagation caused by the upstream tasks of the pipeline model to the downstream tasks, a joint representation framework that uses the attention mechanism to learn sentiment relationships is proposed.

E2E-ABSA [26] The goal of this method is to solve the complete ABSA task in an end-to-end manner. In order to achieve this goal, a new unified model is proposed, and a joint annotation scheme is adopted in the model.

DOER [23] To model the R4 relationship. A bidirectional recurrent neural network is designed to extract the feature representation of each task and a cross-sharing unit to consider the relationship between tasks.

IMN-D [22] An interactive multi-task learning framework is proposed, which can learn word-level and document-level tasks simultaneously, and uses the relationship between R3 and R4, and uses a message-passing mechanism to allow AE and SC subtasks to exchange information.

IMN-BERT [22] It is an extension method of IMN-D. Specifically, we replaced the encoder with BERT-Large, the purpose of which is to facilitate task comparison.

SPAN-BERT [35] A pipeline framework based on span, extracting first and then sorting, is proposed, and BERT-Large is used as the backbone.

RACL-BERT-Large [28] It also uses BERT-large as the backbone encoder and the joint framework RACL but does not incorporate domain knowledge into the pre-training model.

4.4 Results

The results reported for all models are the average over 10 runs. It is noteworthy that our model results are very stable, and the results of ten experiments hardly fluctuate. As shown in Table 5, we used three methods for comparison: M1 represents the method based on Glove, M2 is based on the BERT-Large method, and M3 is based on the BERT-PT method.

First, in the M1 method, we can observe that RACL-Glove outperforms all baselines in the overall indicator ABSA-F1 and achieves 2.12%, 2.92%, and 2.40% surpasses on the three data sets, respectively. Secondly, in the M2 method, RACL-BERT-Large surpasses the method based on the same type of encoder in ABSA-F1. On the three data sets, 4.70%, 1.67%, and 3.76% are achieved, respectively. It shows that the use of joint training of all sub-tasks and a comprehensive model of their interaction is beneficial to improve the performance of ABSA tasks. Finally, the M3 method using the pre-trained model BERT-PT with domain knowledge has achieved the latest results in ABSA-F1. It shows that the pre-trained model with domain knowledge has advantages in solving aspect-oriented sentiment classification tasks.

4.5 Analysis

The effect of K and L There are two main parameters in the model, the kernel size K of the CNN encoder and the number of layers L of the RACL module. While we first fix the size of L, K increases in steps of 2, and then we fix the size of K, and L gradually increases in steps of 1.

From Fig. 5, we can see that when K=1, the model’s performance is inferior because the extracted feature is a single word; when the K=3 and K=5, the size of the features we extract is more reasonable. When K exceeds 5, the model’s performance deteriorates because other irrelevant words will be added during feature extraction. In Fig. 6, we can see that when we increase the number of RACL layers to 5, better performance can be achieved, but when it exceeds 5, the performance of the model does not change much, and increasing the number of layers means an increase in parameters. The cost will increase.

Fig. 5

Effects of K.

Fig. 6

Effects of L.

Cross-validation In order to prevent the model from overfitting and verify the robustness of the model, the experiments are using five-fold (k=5) cross validation. In 5-fold cross validation, the dataset is partitioned into two sest, where 4 folds (k-1) for training the model and 1-fold for testing the model. And for the convenience of comparison, we only use the closest RACL-BERT-Large model and the IMN-BERT model for cross-validation. The results of the crossover experiment are shown in Table 6, and calculation formula for cross-validation is as follows:

$\begin{matrix} {CV}_{k} = \frac{1}{k} \cdot \sum_{i}^{k} L_{i} \end{matrix}$ (26) where i represents the i-th fold cross-validation experiment, L_i refers to formula 25.

Table 6

Corss validation result

Model	Restaurant14 (Res14)				Laptop14 (Lap14)				Restaurant15 (Res15)
	AE(F1)	OE(F1)	SC(F1)	ABSA(F1)	AE(F1)	OE(F1)	SC(F1)	ABSA(F1)	AE(F1)	OE(F1)	SC(F1)	ABSA(F1)
IMN-BERT	83.76	85.12	74.67	68.72	76.55	80.56	74.32	61.83	67.88	72.29	70.19	63.22
RACL-BERT-Large	85.88	87.08	80.61	74.82	80.69	78.72	72.91	62.43	72.06	75.55	73.91	65.05
RACL-BERT-PT	86.92	88.22	80.2	75.57	84.91	81.06	75.67	66.68	73.65	74.73	74.9	66.78

From the experimental results, we can see that three models through cross-validation, the results of the three tasks have a slight decline. For example, our method is in two data sets (Res14, Lap14), and the F1 values of ABSA decreased by 0.5, 1, but rose 0.5 in the data set Res15.

5 Conclusion

In this paper, we propose a joint model RACL-BERT-PT, which combines the relationship-aware collaborative learning framework RACL and the pre-training model BERT-PT with domain knowledge. Experiments on three real-world datasets demonstrate that our RACL-BERT-PT model outperforms the state-of-the-art pipeline and unified baselines for the complete ABSA task. In order to prove the universality of the model, in future work, we will explore and study our model in a broader data set.

Footnotes

Acknowledgments

We thank the anonymous reviewers for their valuable comments. This research was funded by [the National Natural Science Foundation of China] grant number [No.U1603115], [the National Key Research and Development Program of China] grant number [No.2017YFBO504203], [the Science and Technology Planning Project of Sichuan Province] grant number [No.18SXHZ0054] and [National Engineering Laboratory for Public Safety Risk Perception and Control by Big Data] grant number [PSRPC:No.XJ201810101].

References

and Liu

, Mining and summarizing customer reviews, Tenth Acm Sigkdd International Conference on Knowledge Discovery & Data Mining, (2004).

, Liu

, Yu

P.S.

, et al., BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis, NAACL-HLT (1), (2019).

Vicente

I.S.

, Saralegi

and Agerri

, EliXa: A Modular and Flexible ABSA Platform, (2015).

Liu.

, Sentiment analysis: Mining opinions, sentiments, and emotions, Cambridge university press, (2020).

Chen

, Mukherjee

and Bing

, Aspect Extraction with AutomatedPrior Knowledge Learning, Proceedings of the 52nd AnnualMeeting of the Association for Computational Linguistics 1(2014), 347–358.

Dang

, Liao

, Cheng

and Liu

, MF-COTE: A chinese opinion target extraction model based on multiple features, Journal of Intelligent & Fuzzy Systems (2020), 1–16.

Chernyshevich

, IHS RandD Belarus: Cross-domain extraction of product features using CRF, Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), (2014).

Qian

, Liu

, Zhang

, Kim

D.S.

and Gao

, Improving opinionaspect extraction using semantic similarity and aspect associations, Proceedings of the AAAI Conference on Artificial Intelligence 30(1) (2016).

Poria

, Cambria

and Gelbukh

, Aspect extraction for opinionmining with a deep convolutional neural network, Knowledge-Based Systems 108 (2016), 42–49.

10.

Clercq

O.D.

, Lefever

, Jacobs

, Carpels

, et al., Towards an integrated pipeline for aspect-based sentiment analysis in various domains, Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (2017), 136–142.

11.

Xin

and Lam

, Deep Multi-Task Learning for Aspect Term Extraction with Memory Interaction, Proceedings of the 2017 conference on empirical methods in natural language processing (2017), 2886–2892.

12.

Shuai

, Mazumder

, Bing

, Zhou

and Yi

, Target-sensitivememory networks for aspect sentiment classification, Proceedings of the 56th Annual Meeting of the Association forComputational Linguistics 1 (2018), 957–967.

13.

Tulkens

and Cranenburgh

A.V.

, Embarrassingly Simple Unsupervised Aspect Extraction, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020), 3182–3187.

14.

Wei

, Hong

, Zou

, Cheng

and Yao

, Don’t Eclipse Your Arts Due to Small Discrepancies: Boundary Repositioning with a Pointer Network for Aspect Extraction, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020), 3678–3684.

15.

Qiu

, Liu

, Bu

and Chen

, Opinion word expansion and targetextraction through double propagation, Computationallinguistics 37(1) (2011), 9–27.

16.

Long

, Mo

, Ming

, Liu

and Zhao

, Target-dependent twitter sentiment classification, Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies (2011), 151–160.

17.

, Wei

, Tan

, Tang

and Ke

, Adaptive recursive neural network for target-dependent twitter sentiment classification, Proceedings of the 52nd annualmeeting of the association for computational linguistics (2) (2014), 49–54.

18.

D.T.

and Zhang

, Target-dependent twitter sentiment classification with rich automatic features, Twenty-Fourth International Joint Conference on Artificial Intelligence (2015).

19.

Zhu

and Qian

, Enhanced aspect level sentiment classification with auxiliary memory, Proceedings of the 27th International Conference on Computational Linguistics (2018), pages 1077–1087.

20.

Zhu

, Chen

, Zheng

and Qian

, Aspect Aware Learning forAspect Category Sentiment Analysis, ACM Transactions onKnowledge Discovery from Data 13(6) (2019), 1–21.

21.

Wang

, Shen

, Yang

, Quan

and Wang

, Relational Graph Attention Network for Aspect-based Sentiment Analysis, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020), 3229–3238.

22.

, Lee

W.S.

, Ng

H.T.

and Dahlmeier

, An Interactive Multi-Task Learning Network for End-to-End Aspect-Based Sentiment Analysis, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019), 504–515.

23.

Luo

, Li

, Liu

and Zhang

, DOER: Dual Cross-Shared RNN for Aspect Term-Polarity Co-Extraction, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019), 591–601.

24.

Zhao

, Huang

, Zhang

, Lu

and Xue

, SpanMlt: A Span-based Multi-Task Learning Framework for Pair-wise Aspect and Opinion Terms Extraction, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020), pages 3239–3248.

25.

Tang

, Ji

, Li

and Zhou

, Dependency Graph Enhanced Dual-transformer Structure for Aspect-based Sentiment Classification, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020), 6578–6588.

26.

, Bing

, Li

and Lam

, A Unified Model for Opinion TargetExtraction and Target Sentiment Prediction, Proceedings of theAAAI Conference on Artificial Intelligence 33(1) (2019), 6714–6721.

27.

Zhang

, Han

, Liu

, Jiang

, Sun

and Liu

, ERNIE: Enhanced Language Representation with Informative Entities, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019), 1441–1451.

28.

Chen

and Qian

, Relation-aware collaborative learning for unified aspect-based sentiment analysis, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020), 3685–3694.

29.

Chen

, Fisch

, Weston

and Bordes

, Reading Wikipedia toAnswer Open-Domain Questions, Proceedings of the 55th AnnualMeeting of the Association for Computational Linguistics 1(2017).

30.

, Liu

, Shu

and Yu

P.S.

, Double Embeddings and CNN-basedSequence Labeling for Aspect Extraction, Proceedings of the56th Annual Meeting of the Association for ComputationalLinguistics 2 (2020), 592–598.

31.

Pontiki

, Galanis

, Pavlopoulos

, Papageorgiou

, Androutsopoulos

and Manandhar

, SemEval-2014 Task 4: Aspect Based Sentiment Analysis, Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014) (2014), 27–35.

32.

Pontiki

, Galanis

, Pavlopoulos

, Papageorgiou

, Manandhar

and Androutsopoulos

, SemEval-2015 Task 12: Aspect Based Sentiment Analysis, Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015) (2015), 486–495.

33.

Wang

, Pan

S.J.

, Dahlmeier

and Xiao

, Coupled multi-layerattentions for co-extraction of aspect and opinion terms, Thirty-First AAAI Conference on Artificial Intelligence 31(1) (2017).

34.

Wang

, Lan

and Wang

, Towards a One-stop Solution to Both Aspect Extraction and Sentiment Analysis Tasks with Neural Multi-task Learning, 2018 International Joint Conference on Neural Networks (IJCNN) (2018), 1–8.

35.

, Peng

, Huang

, Li

and Lv

, Open-domain targeted sentiment analysis via span-based extraction and classification, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019).

Aspect-based sentiment analysis via relation-aware collaborative learning

Abstract

Keywords

1 Introduction

2 Related work

3 Methodology

3.1 Task definition

3.2 Model architecture

4.1 Training procedure

Table 4 Datasets Datasets Type Aspect Opinion Restaurant14 Train 3699 3484 Test 1134 1008 Laptop14 Train 2373 2504 Test 654 674 Restaurant15 Train 1199 1210 Test 542 510

4.5 Analysis

Footnotes

Acknowledgments

References

Table 4
Datasets

Datasets Type Aspect Opinion

Restaurant14 Train 3699 3484

Test 1134 1008

Laptop14 Train 2373 2504

Test 654 674

Restaurant15 Train 1199 1210

Test 542 510