Evaluation of end-to-end aspect-based sentiment analysis methods employing novel benchmark dataset for aspect,and opinion review analysis

Aspect and opinion term extraction

In the past, aspect extraction was commonly performed by rule-based approaches [36]. However, recent work has shown that there is some potential for improvement in aspect extraction techniques. Xu et al. [48] proposed a double embedding mechanism where general and domain embeddings are combined. Wang et al. [45] utilized a multi-layer network where each layer contains aspect and opinion attention. Following the previous work, Li and Lam [25] employed a memory network to improve performance by memory interaction between AE and OE. Co-extraction of aspects and opinions was also performed by Dai and Song [7] where the authors enhanced the performance with mined rules as weak supervision. While all previous works deal with labeled data in supervised learning, several studies attempted to perform aspect extraction in an unsupervised manner utilizing auto-encoders [15] and weakly supervised with the help of seeded words [2].

Aspect term polarity classification

Recent works for aspect term polarity classification focus either on the research of brand-new methods or incorporate other data sources and transfer knowledge from other tasks. Xue and Li [49] point out the inefficiencies in the training time and parallelization of Long Short-Term Memory network (LSTM) architectures and propose gated convolutional networks that train faster and also outperform LSTMs. Li et al. [22] state that Convolutional Neural Networks (CNNs) are unable to fully explore the target information as done by Recursive Neural Networks (RNN)-based methods such as LSTM. Hence, the authors introduce transformation networks that combine both RNN and CNN. As one of the first works, He et al. [17] show the advantages of performing auxiliary tasks (i.e. document-level sentiment and domain classification) with a vanilla LSTM and attention. The study was followed by Du et al. [11], who use capsule networks with RNN and attention as an alternative to pooling. Similarly, Chen and Quian [5] also use transfer capsule networks to improve the exchange of information between the tasks. Another work utilized capsules both for aspect term polarity classification and aspect category polarity classification without the necessity of using auxiliary data [19]. This model tries to tackle the problem of predicting different polarities for multiple aspects in one sample.

E2E-ABSA and pipeline approaches

While previous models focused on either aspect term (opinion target) extraction or aspect term (opinion) polarity classification (or handled them in a pipeline manner), Li et al. [23] address both tasks with a unified model formulated as a sequence labeling problem using a unified tagging scheme. Expressing ABSA in such a way enables the exploitation of correlations among AOE and ATC tasks. Later, the authors also engaged a BERT encoder to improve the performance of their model [24]. Building on their previous work, He et al. [18] have recently introduced an interactive multi-task learning network (IMN) for E2E-ABSA based on two document-level auxiliary tasks, in addition to AE, ATC, and OT while dealing with AE and OE as an aspect and opinion term co-extraction task (AOE). In detail, the authors’ proposed model introduces a message passing mechanism that provides each classification task in step t with the concatenated outputs of all tasks from step t - 1 . Besides, the opinion transmission mechanism unidirectionally passes the output of the AOE task to the ATC task. As suggested by Wang et al. [43], He et al. [18] employ Convolutional Neural Networks (CNN) encoders in their architecture; nonetheless, past works argued with the suitability of other neural networks. Following He et al. [18], Chen and Quian [6] proposed mechanism called relation-aware collaborative learning where relation propagation mechanism stacks in the multi-layer network. Previous works focus on a fully integrated solution for ABSA, [32] proposed a two-stage framework. In the first stage, the model predicts aspects, opinions, and polarities separately. In the second stage, it pairs up predictions into triplets with ATE-OTE-ATP.

Recent methods of aspect-based sentiment analysis focus on end-to-end solutions for ABSA while also incorporating additional sources of knowledge to deal with the unavailability of sufficiently large datasets. Thus, we target the addressed lack of data, stated as one of the reasons to introduce auxiliary tasks [17], and introduce a novel ABSA dataset annotated with a unified tagging scheme and 5x larger than the currently available datasets.

Finally, based on analysis of related work and availability of models, we selected works [18, 6, 24, 48, 45, 17, 5, 19] as state-of-the-art methods for evaluation of our novel dataset (see Section 4.2).

E2E-ABSA

For the task of integrated end-to-end aspect-based sentiment analysis, the following four state-of-the-art models, including eight pipeline approaches, as well as two baselines are used for comparison:

•

Simple CNN – shared encoder built from two stacked convolutional layers and self-attention layer followed by separate linear layers for AE/OE and ATC

•

Simple RNN – shared encoder built from simple bidirectional LSTM and self-attention layer followed by separate linear layers for AE/OE and ATC

•

IMN^-D [18] – is a unified method with joint training for AOE and ATC with separate labels while ATE and OTE are fused into AOTE,

•

RACL-glove and RACL-BERT[6] – is a unified method employing collaborative learning for AT, OT and ATC with GloVe and BERT version of embeddings,

•

BERT-E2E-TBSA[24] – is a unified method using the collapsed tagging scheme for ATE and ATP, it is a BERT version of previous model E2E-TBSA[23], we evaluated two versions with GRU and Transformer layers over BERT referred as BERT-E2E-TBSA-gru and BERT-E2E-TBSA-tfm respectively,

•

pipeline approach as a combination of following models {DE-CNN, CMLA} + {dTrans, TransCap, CapsNet, CapsNet-BERT} where DE-CNN[48] and CMLA[45] are employed for aspect/opinion extraction and dTrans[17], TransCap[5], CapsNet and CapsNet-BERT[19] are employed for aspect term polarity classification.

Aspect/opinion term extraction

For both tasks, aspect term extraction and opinion term extraction, we employ two state-of-the-art models as well as two baselines. All models are trained with fused ATE and OTE labels.

•

Simple CNN – employs two stacked convolutional layers and self-attention layer with fused ATE and OTE labels,

•

Simple RNN – employs simple bidirectional LSTM and self-attention layer with fused AT and OT labels,

•

DE-CNN[48] – is the state-of-the-art model for ATE, it utilizes a multi-layer CNN, we performed fused ATE and OTE labels as proposed by He et al. [18],

•

CMLA[45] – was designed to perform co-extraction of aspect and opinion terms by modeling their inter-dependencies.

Aspect term polarity classification

•

dTrans[17] employs attention-based long short-term memory which originally utilizes knowledge from document corpora which was not used for our experiments,

•

TransCap[5] refers to transfer capsule network which originally tries to transfer knowledge from document-level data which was not used for our experiments,

•

CapsNet and CapsNet-BERT models [19] employ capsule networks while CapsNet-BERT employs the BERT model.

4.3 Train and test data

We utilize both versions of the dataset (sentence-level and review-level) in the model evaluation; however, the evaluation of the ATP models and pipeline models on the review-level is not done due to the way the models for aspect term polarity classification are designed. This is because the original idea works with repetition; each sample is repeated for each aspect. While the number of aspects, hence repetitions, in sentences is usually limited, whole reviews typically contain much higher numbers. Originally, the models were not designed for this. In addition, we are not able to evaluate BERT-based models on the review-level dataset version since the maximum input length for BERT is 512 which is shorter than many reviews.

To provide further information on DAORA, we show detailed statistics of training (Train), development (Dev) and testing (Test) sets along with their overall numbers in Table 7. In addition to the previous comparison with different datasets, we present more detailed information about the polarity of the aspects (negative, neutral, positive, conflict).

All experiments were performed on an NVIDIA Titan RTX with hyper-parameters set as proposed in original papers. We only had to change the transformation of input data and evaluation measures. To ensure the comparability of the methods, we did not use any external data, such as document-level data for any of the experiments, as well as domain embeddings. The textual input has been fed into a word embedding layer which was either initialized using standard Glove embeddings [33] with 300 dimensions, or with uncased BERT [8]; the use of BERT large or BERT small depended on its use by the original papers. Moreover, the textual input was transformed to lowercase without any additional changes for all experiments.

5.1 E2E-ABSA

Table 9 provides an overview of the model performances for end-to-end aspect-based sentiment analysis on the sentence-level. For the integrated models as well as selected pipeline models we conduct polarity classification with and without conflict polarity labels. The dataset version without conflict labels follows the work of He et al. [18] and only keeps the aspect annotation for the AOE task.

According to recent research works, it is common practice to omit conflict labels given their usually low number of samples in the data [40, 46]. In contrast, we perform experiments with and without conflict labels. However, the results show a minor impact. Comparing the results for the acc-s measure, we can even observe a higher accuracy for the versions with conflict labels. Integrated models bring advantages and generally outperform pipeline models; they can better model correlations between the individual sub-tasks and, hence, are able to share and exploit information. Analyzing the performance of CapsNet-BERT, we observe that it performs better than integrated models on the sentiment task, however, it suffers from a limited performance on the aspect extraction stage.

Table 10 shows the performance of the models for end-to-end aspect-based sentiment analysis on the review-level. Similarly to the previous table, polarity classification is conducted with and without the conflict polarity labels. For this comparison, only models compatible with such setup were selected (i.e. models using BERT were omitted as it does not process more than 512 words per input).

All evaluated models perform better on the sentence-level dataset although the review-level dataset provides more context and the annotation was performed at the review-level. The differences in performance can relate to two causes: First, although providing more context, whole reviews might contain unrelated opinions towards other hotels or restaurants in the neighborhood. Second, there is a large difference in the sample length when considering whole reviews and sentences; sentence lengths are in the range of 2–90 words while reviews contain 2–50 sentences. This extensive length might lead to the loss of context in the network. In addition, many models have a limited length of the input sequence, however, we predict the label for each token when employing a unified tagging scheme.

Table 11 shows the performance of the models on aspect/opinion term extraction on the sentence-level and review-level. Across all approaches and on both evaluation metrics, we can see that the performance of the models trained on the sentence-level is superior to those trained on the review-level. We hypothesize this is related to the problem of keeping context in long sequences. Further, the CNN based models achieve higher F1 results on the review-level data, while the RNN is better on the sentence level. Comparing the aspect extraction with the opinion extraction, the latter seems to be a slightly easier task for the sentence-level approaches, however, this is not clear on the review-level dataset. On the one hand, it is possible that AE is, in general, easier to achieve than OE. On the other hand, having the characteristics of DAORA in mind, we know its advantage of coming with a high and diverse number of opinion phrases. Hence, models are provided with some rich information which can lead to an increased performance.

In Table 12, we can observe the performance of the aspect term polarity classification models on the sentence-level dataset, with and without consideration of the conflict labels. While the results, evaluated in accuracy, are comparable between the models, varying around 1 point, the differences in terms of F1-score are higher, varying between 10 and 14 points. In addition, we can observe the similarity between the models when it comes to the performance on runs with and without conflict labels; the ones doing well without conflict label also do well with conflict label. The best model, CapsNet-BERT, outperforms the other approaches across all set-ups and evaluation metrics. Generally, the results for this task are in line with previous research [17, 5, 19]. While the results are different for the particular state-of-the-art datasets (SemEval 2014 [35] and SemEval 2015 [34]), the rank of the models is the same as reported in the original works for sentiment accuracy (acc-s) and sentiment F1-score (F1-s).

DAORA consists of a variety of sentence typologies, namely No ATE, ATE, Single ATE, Multiple ATE, and Different ATP, which differ in the number of aspects and corresponding polarities. To provide insight into the models performance, we evaluated these separately for each typology except for No ATE. Table 13 presents the outcome of this evaluation. The results show a higher performance for the AE and OE tasks for sentences with Multiple ATE than for Single ATE sentences. Results are higher between 5–7 points and 3–5 points for the AE and OE prediction when sentences without ATE are not present in the dataset. With these findings, we can conclude that false predictions on ATE and OTE for No ATE sentences have a significant negative impact on the overall performance. The ATC models perform better for sentences with Multiple ATE than the Single ATE sentences. However, on Different ATP sentences the performance considerably drops by 7–20 points for the sentiment accuracy. We hypothesize this occurs since, on the one hand, the ATC models take into account the whole sentence for their prediction and, on the other hand, assign the same sentiment prediction to all aspects (irrespective of their polarity) in the sentence. Our evaluations confirm that model performance suffers from incorrect sentiment prediction on different ATP sentences (drop on acc-s) and incorrect aspect extraction on No ATE sentences (improvement on F1-a on ATE sentences). Assessing the sentences No ATE also showed that the overall performance is negatively affected when these samples are present in the dataset, thus, pointing at the models’ difficulty in sentiment classification in cases without ATE. We selected the best pipeline model (DE-CNN+CapsNet-BERT), the best joint model with glove embeddings (RACL-glove) and the best BERT-based model (RACL-BERT). In addition, we wanted to explore differences between IMN and RACL to see the actual contribution of such a model to specific cases with the original idea of relation modeling.

While DAORA is larger than the previous state-of-the-art datasets for aspect-based sentiment analysis, it comes also with terms that have much higher frequency than others. To provide an overview of this, we present its 10 most frequent ATE and OTE in Section 3. Given that terms can be labeled or unlabeled, depending on their occurrence in a review, the models have to learn to differentiate terms between unlabeled, ATE, and OTE. To assess the difficulty to distinguish terms, Table 14 provides an evaluation (in precision, recall, and F1-score) of the classification of the top ten terms.

5.6 Polarity prediction

Many datasets contain polarities that are not balanced. In DAORA, we consider positive and negative classes as the major ones while neutral and conflict are the minor classes. Table 15 shows the average number of polarity predictions by the selected models for each polarity class. While the models polarity predictions of the majority classes is close to the distribution in the test data, the same does not occur for the minority classes. In this case, all models under-perform. Further, this effect is the strongest for the conflict polarities which are not predicted at all by the DE-CNN and CapsNet-BERT models. Although the performance score does not allow us to determine the correctness (i.e. true positives) of the predictions, we can still infer that predicting the conflict polarity is challenging for most models. Despite this, we are not fully able to comprehend such behavior and some people could claim that conflict polarities were not sufficiently represented in the train set. In addition, we hypothesize another explanation for the models difficulty in predicting conflict polarities. While positive, negative, and neutral are single polarities, the conflict polarity is much more complex, consisting of combinations of single polarities. Given that the majority of the models use the arguments of the maxima (argmax) to obtain the most probable prediction, we assume that the models predict single polarities with a higher confidence, which in turn impedes the selection of the conflict polarity. Further, we hypothesize that predicting conflict polarities (i.e. the combination of single polarities) comes with a higher difficulty than predicting single polarities, hence, relative to the single polarity training samples, the models would need many more conflict polarity samples in the training set.

In this section we analyze the models predictions and explore the nature of the erroneous observations. In Tables 16–19, we present the results for selected samples. Similarly to previous error analysis, we selected the best pipeline model (DE-CNN + CapsNet-BERT), the best joint model with glove embeddings (RACL-glove) and the best BERT-based model (RACL-BERT). In addition, we wanted to explore differences between IMN and RACL to see the actual contribution of such a model to specific cases with the original idea of relation modeling. To present such predictions, we applied a voting mechanism to obtain one single prediction from the runs of each model. Generally, we can define four error types: (1) Incorrect Sentiment Prediction, (2) Incorrect ATE/OTE Prediction, (3) Non-opinionated Sentences, and (4) Special Sentence Structures.