Research on aspect-based sentiment analysis of movie reviews based on deep learning

Abstract

Aspect-based sentiment analysis aims to extract the sentiment polarity of different aspects within a text. In recent years, most methods have relied on pre-trained language models such as BERT and Roberta to learn semantic representations from the context. However, in texts with ambiguous sentiment expression, the absence of domain knowledge guidance may lead pre-trained language models to miss critical information, and the attention mechanism might incorrectly focus on text that is irrelevant to the aspect categories. To address these issues, this study integrates the ontology of movie reviews to construct an aspect-based sentiment analysis model based on the ERNIE(OMR-EBA). We annotated a new Chinese data set focused on movie reviews to evaluate the model’s performance. Experimental results show that our model achieves 86% accuracy in aspectual sentiment analysis, which is better than other baseline models. The movie review domain ontology and aspect-based sentiment analysis model proposed in this study can provide valuable reference and guidance for research in the field of online movie reviews. It can also help movie production teams understand genuine user sentiments, aiding in subsequent marketing and production efforts.

Keywords

Aspect recognition aspect-based sentiment analysis attention mechanism domain ontology ERNIE

1. Introduction

In the vast amount of online data, review data provides crucial insights into public opinion, market development and marketing strategies. Early sentiment analysis tasks primarily focused on identifying the overall sentiment at the sentence or document level [1], which is a coarse-grained sentiment analysis that cannot locate and describe different aspects of a sentence at a finer level. In real life, user comments on the Internet contain affective attitudes towards multiple objects, for example, in movie reviews, a comment like ‘The leading actress is beautiful, but the story is too cheesy, the narrative pace is too slow, and the background music is not bad’. This comment involves four evaluation objects: the actress, the story, the narrative, and the music. Each evaluation object has different sentiment polarities, and focusing only on the overall sentiment of the sentence may obscure the user’s true emotional attitude. In recent years, Aspect-based Sentiment Analysis (ABSA) has become a hot issue in the field of natural language processing [2], aiming at detecting the sentiment polarity of different aspects in the same sentence [3], which can provide feedback information with more dimensions, thus increasing the overall understanding of users’ needs.

Early aspect-based sentiment analysis tasks often used machine learning methods, which involve training sentiment classifiers based on feature engineering to predict the sentiment tendencies of different aspects [4]. Commonly used machine learning techniques include Naive Bayes [5], Support Vector Machines [6], Decision Trees [7], etc. Al-Smadi et al. [8] used different classifiers such as Support Vector Machines (SVMs), K-Nearest Neighbours (KNN), Decision Trees (DTs), and Bayesian Networks (BNs) to perform the ABSA task. Their experiments demonstrated that Support Vector Machines (SVM) exhibited the best performance. Although machine learning algorithms are highly scalable and reusable for aspect-based sentiment analysis tasks, Sivakumar et al. [9] found that these algorithms struggle to handle and analyse multiple aspects within a single text and cannot extract implicit aspect sentiments from sentences [10]. In recent years, as the architecture and performance metrics of deep learning models have progressively improved, researchers have increasingly leaned towards using deep learning methods for aspect-based sentiment analysis tasks [11].

Currently, deep learning methods such as Convolutional Neural Networks [12,13], Long Short-Term Memory Networks [14,15], and Pre-trained Language Models [16,17] have all been widely used in aspect-based sentiment analysis tasks. Among these, pre-trained language models based on the Transformer architecture, such as BERT and Roberta, demonstrate superior generalisation ability and robustness in aspect-based sentiment analysis tasks [18]. Li et al. [19] demonstrated that BERT-based models outperform early neural models specifically created for the end-to-end ABSA challenge. Zhao and Yu [20] showed the practicality of combining domain knowledge from sentiment knowledge graphs with the BERT language representation model, achieving excellent performance even with small-scale training data. Although pre-trained language models such as BERT are promising, they face two major challenges: (1) The pairing information between evaluation objects and aspect terms in review texts is not always clear [20]. In texts with ambiguous language expression, the lack of domain knowledge guidance may lead pre-trained language models to incorrectly treat context words that are unrelated to the given aspect. (2) Review texts often contain a large amount of noise information unrelated to sentiment. Most existing methods use BERT as an embedding layer for feature extraction [21 –23], combined with various deep learning algorithms (such as BiLSTM, CNN, etc.) to enable the model to learn contextual information. However, these methods do not focus on the local sentiment information of specific aspects, resulting in reduced performance of the model in multi-aspect sentiment classification.

In terms of experimental data sets, the most widely used data sets for aspect-based sentiment analysis tasks are the English data sets from various domains such as laptops, restaurants, and social media, released by the International Workshop on Semantic Evaluation (SemEval [24 –26]). Compared with the abundance of English data sets, high-quality Chinese data sets are still relatively scarce [27]. Wang et al. [28] proposed a Chinese annotated corpus for aspect-based sentiment classification in question-answering tasks based on new ABSA research tasks. Bu et al. [27] released a Chinese restaurant review data set containing 18 predefined aspect categories for aspect-level sentiment analysis and rating prediction tasks. However, Zhuang et al. [29] found that compared with restaurant and electronics reviews, movie reviews contain more related elements and involve a wider range of aspect categories. Different users may use various vocabularies to express opinions on the same aspect [30]. For example, the term ‘plot’ can be replaced with ‘story’ or ‘film’, which poses a challenge to the accuracy of aspect sentiment classification. Regatt et al. [31] created a Telugu-language movie review data set, categorising aspect terms into six categories: story, performance, direction, music, technical aspects, and general, but these six categories do not fully encompass all aspects of movie evaluations from users. Chinese movie reviews differ from reviews in other languages in terms of linguistic characteristics, corpus size, and annotation methods. These differences require in-depth exploration in aspect-based sentiment analysis tasks.

Inspired by the above observations, there is a need to provide a more explanatory and nuanced approach to aspect-based sentiment analysis for online movie reviews. Based on this, the goal of our work is to construct an aspect-based sentiment analysis model for online movie reviews. This model is designed to integrate external domain knowledge into pre-trained language models, extract evaluation aspects from movie review texts, and analyse the sentiment polarity of each aspect. To achieve this, we introduced a domain ontology specific to online movie reviews, covering six main aspects: story, role, actor, audiovisual, genre, behind the scenes, along with 23 sub-ontologies. This allows for a more multi-dimensional and in-depth summarisation of online movie reviews. In addition, considering the linguistic particularities of Chinese movie reviews, we designed an aspect-based sentiment analysis model based on the ERNIE 2.0 framework. Compared with pre-trained language models like BERT and Roberta, ERNIE 2.0 is more suitable for Chinese sentiment analysis tasks, offering significant improvements in semantic understanding and representation capabilities. It better captures the relationships between words within a sentence. Experimental results show that our method outperforms baseline models.

The main contributions of this paper can be summarised as follows. First, we constructed a movie review domain ontology following the seven-step methodology for ontology construction. We systematically reviewed and integrated the ontology classes, attributes, aspects, and relationships within the domain of movie reviews. Second, based on the movie review domain ontology, we annotated 400,000 user movie reviews according to six predefined aspects and three predefined sentiment polarities. This resulted in the creation of a Chinese aspect-based sentiment analysis data set. Third, we proposed an aspect-based sentiment analysis model that integrates the ontology of movie reviews (OMR-EBA), addressing the issues of insufficient domain knowledge guidance and inadequate aspect feature extraction in sentiment analysis tasks. Experimental results have confirmed the superiority of this method. Our code is publicly available at https://github.com/HYM98/ABSA-Model. Overall, the proposed method can assist movie marketing teams in understanding audience sentiment towards different aspects of a movie, enabling the development of more targeted marketing strategies. It can also help filmmakers identify strengths and areas for improvement, guiding future production decisions and enhancing the quality of films.

The remaining sections of this paper are organised as follows. Section 2 reviews and discusses related work on aspect-based sentiment analysis. Section 3 describes the framework of this study, including the construction method of the movie review domain ontology and the structure of the aspect-based sentiment analysis model. Section 4 presents the approach for building the aspect-based sentiment analysis model. In Section 5, we have demonstrated the effectiveness of our approach through comparative experiments. Section 6 provides a summary of the findings and future prospects are presented.

2. Related work

This section provides a brief overview of the task types in aspect-based sentiment analysis and analyses literature most relevant to aspect category sentiment analysis for this study. In addition, it offers a brief review of existing research on aspect-based sentiment analysis in movie reviews and discusses their limitations.

2.1. Subtasks of aspect-based sentiment analysis

Aspect-based sentiment analysis aims to detect the sentiment polarity of different aspects within the same sentence [3], providing valuable information for user needs analysis, opinion monitoring, or business decision-making. Pontiki et al. [26] identified three important subtasks of aspect-based sentiment analysis: (1) Aspect Category Detection (ACD), which is used to identify the category of aspects discussed in each sentence; (2) Opinion Term Extraction (OTE), which extracts viewpoint words and opinion terms related to entities or aspects; (3) Aspect-based Sentiment Classification (ASC), which makes sentiment tendency judgements about different aspects. Taking ‘The plot of the movie is very interesting and the background music is very good’ as an example, the analysis results of the three subtasks are shown in Figure 1.

Figure 1.

Example of aspect-based sentiment analysis subtask.

In sentence S, ‘plot’ and ‘background music’ are evaluative entities, ‘interesting’ and ‘good’ are opinion words. The aspectual sentiment of ‘plot’ and ‘background music’ is positive, as shown by the opinion words. In addition to the above three important subtasks, Zhang et al. [32] proposed a new classification method for ABSA, further dividing aspect-based sentiment analysis methods into single ABSA tasks and compound ABSA tasks, as shown in Table 1.

Table 1.

ABSA task classification, and representative studies for each task.

Classification	Subtasks	Description	Type	Research
Single ABSA Tasks	Aspect Category Detection(ACD)	Identify the category of aspect discussed in each sentence	Classification	Zhou et al. [33]; Kumar et al. [34]; Miao et al. [35]; Hu et al. [36]; Shi et al. [37]
	Opinion TermExtraction (OTE)	Extraction of opinion words and opinion terms expressed in the text	Extraction	Wang et al. [38]; Yu et al. [39]; Veyseh et al. [40]; Mensah et al. [41]
	Aspect-based SentimentClassification (ASC)	Identify the emotional polarity of the text	Classification	Davidov et al. [42]; Jiang et al. [43]; Wang et al. [44]; Meng et al. [45]; Chen et al. [46]; Dai et al. [47]
Compound ABSA Tasks	Aspect CategorySentiment Analysis (ACSA)	Joint detection of aspect categories contained in the text and their corresponding sentiment polarity	Classification	Schmitt et al. [48]; Wang et al. [49]; Cai et al. [50]; Li et al. [51]; Liu et al. [52]
	Aspect-Opinion PairExtraction (AOPE)	Pairwise extraction of aspects and corresponding viewpoint words in the text	Extraction	Chen et al. [53]; Zhao et al. [54]; Wu et al. [55]; Wu et al. [56]
	Aspect Sentiment TripletExtraction (ASTE)	Simultaneous extraction of ternary groups of aspect, opinion words and sentiment polarity in the text	Classification &Extraction	Peng et al. [57]; Xu et al. [58]; Chen et al. [59]
	Aspect SentimentQuad Prediction (ASQP)	Predicting quaternions of aspect, category, point of view and affective polarity in a text	Classification &Extraction	Cai et al. [60]; Li et al. [61]

In ABSA research, there has been a great deal of related work done by researchers on tasks such as opinion term extraction and aspect sentiment triplet extraction. The following will describe the work most relevant to us and discuss the current state of research in aspect category sentiment analysis.

2.2. Aspect category sentiment analysis

Aspect category sentiment analysis aims to jointly detect the aspect categories present in the text and their corresponding sentiment polarities [32]. For example, as illustrated in Figure 1, ACSA is able to predict the sentiment (plot, POS) and (background music, POS) for all aspectual categories involved in the text. The pipeline approach is the most straightforward way to deal with ACSA, which sequentially channels multiple models with potentially different modelling paradigms to obtain the result [32]. In a pipeline solution, aspect category detection is performed first to identify the aspect categories mentioned in the text, followed by aspect category sentiment classification to predict the sentiment polarity for the detected aspect categories. In this subsection, we will separately review the current research status of the ACD and ASC tasks.

The goal of aspect category detection is to classify reviews based on product features or entity aspects [62]. It primarily consists of two subtasks: (a) aspect term extraction and (b) classification/normalisation of aspect terms into aspect categories. Common implementation methods for ACD include rule-based and pattern-matching approaches, statistical machine learning approaches, neural network-based approaches. Rule-based and pattern-matching methods rely on manually defined rules and patterns to identify aspect categories in the text. Kumar et al. [34] proposed a hybrid rule-based approach for aspect category detection that uses association rules and word embeddings to capture statistical and semantic associations. Experimental results showed that when using domain-specific word embeddings to combine semantic similarity, this method achieved better accuracy compared with general word embeddings. Such methods are simple to implement but have limited generalisation capabilities of the rules. Statistical machine learning methods use algorithms to learn features of aspect categories from large amounts of annotated data. Jakob et al. [63] implemented opinion target extraction using Conditional Random Fields (CRF) and conducted experiments in both single-domain and cross-domain environments. Their results demonstrated that the CRF model could mitigate the negative impact of errors in opinion expression recognition. While these methods can learn features of aspect categories, they do not effectively address issues such as polysemy and missing contextual semantic information. In recent years, neural network models such as Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) networks, and their variants have been widely applied to ACD tasks [64]. Miao et al. [35] used a BiLSTM-CRF fusion model to identify digital products in user reviews, demonstrating that the BiLSTM-CRF model significantly outperforms both LSTM and CRF models. Ma et al. [65] employed a BERT-BiLSTM-CRF model for identifying temporal entities in social media texts, showing that this model performs better than BiLSTM-CRF. These neural network models can automatically learn deep representations of the text and capture long-distance dependencies, but they require large amounts of annotated data and substantial training time.

Aspect sentiment classification also known as aspect-based/-targeted/-level sentiment classification [32], aims to predict the sentiment polarity of specific aspects within a sentence. Early ASC tasks were treated as text classification problems and typically implemented using supervised machine learning algorithms. These methods required training a sentiment classifier based on feature engineering to predict the sentiment orientation of aspects or targets. Representative methods include K-Nearest Neighbours [42] and Support Vector Machines [43] among others. Although machine learning algorithms offer strong scalability and reusability for aspect-based sentiment analysis tasks, training sentiment classification models requires large-scale manually annotated data sets and complex feature engineering, which can lead to issues such as underfitting or overfitting. Current research primarily focuses on using neural networks to align each aspect with sentences to learn aspect-related sentence representations [46]. To provide multiple attentions to the target vocabulary and build richer semantic information, many studies employ attention mechanisms to obtain representations specific to each aspect [44,66,67]. For example, Wang et al. [44] proposed ATAE-LSTM, which connects aspect embeddings with input sentence representations, allowing aspect terms to participate in the calculation of attention weights for aspect sentiment classification. Beyond LSTM, researchers have designed more complex network structures to better learn aspect-specific representations. For instance, Meng et al. [45] introduced a feature-enhanced attention model, CNN-BiLSTM, where CNN extracts higher-level phrase representations from the embedding layer, and the FEA attention module encodes both the target and the full sentence to better distinguish the relevance between the target and sentiment. To explore the correlation between text sentiment polarity and contextual information, Zeng et al. [68] proposed a local context focus mechanism based on multi-head self-attention and aspect sentiment classification, which makes the model more attentive to local contextual information in the text. In recent years, pre-trained language models such as BERT have achieved significant results in several natural language processing tasks, and the latest ASC tasks started to use pre-trained language models. Sun et al. [16] fine-tuned the pre-trained BERT model on an aspect-based sentiment classification task, and proved experimentally that the classification based on sentence pairs was better than the single-sentence classification based on fine-tuned BERT. Li et al. [19] used BERT as an embedding layer in combination with various neural network models for end-to-end aspect-based sentiment analysis. Although Transformer-based pre-trained language models show promise, the lack of syntactic knowledge guidance may lead to incorrect handling of context words unrelated to the given aspect [69]. In addition, generating aspect-level training data requires manual annotation of aspect targets, which can be quite challenging in practice [70].

In summary, aspect category sentiment analysis based on deep learning has stronger generalisation and expressive capabilities. However, it still faces significant issues such as handling polysemy and incomplete contextual semantic information in aspect recognition tasks. In addition, aspect sentiment classification methods often struggle to simultaneously process and analyse sentiment polarities for multiple aspects, and relying solely on pre-trained language models like BERT to learn contextual semantic information may overlook key information related to aspect categories. To address these issues, this paper proposes an aspect-based sentiment analysis model based on ERNIE 2.0. This model is better suited for Chinese sentiment analysis tasks and incorporates an attention module to better capture key information within sentences, more accurately identify user evaluation entities and aspects, and effectively determine the sentiment polarity of users regarding specific aspects.

2.3. Sentiment analysis based on movie reviews

Currently, aspect-based sentiment analysis has been widely applied in areas such as electronic product reviews (e.g., laptops, cameras) and dining reviews (e.g., restaurants, hotels). However, research in the field of movie reviews is relatively sparse. With the proliferation and development of the Internet, film and television resources have become an important way for people to acquire knowledge and for entertainment. In this context, analysing user needs and uncovering user interests through ABSA becomes particularly significant. This subsection will explore research in aspect-based sentiment analysis within the domain of movie reviews and provide insights into the current progress in this area.

Ontologies are considered one of the widely accepted structures for representing contextual knowledge. Sejwal et al. [71] integrated movie ontologies with rule-based and pattern-matching sentiment analysis techniques to capture users’ sentiment polarities towards various aspects and contexts of movies. Penalver et al. [72] used ontologies to model predefined domain features of movies and combined them with the SentiWordNet sentiment lexicon for sentiment polarity identification. Another linguistically based approach is dependency parsing, which describes sentence structure by identifying the dependencies of each word in a sentence. Analysing these links and dependencies can help recognise emotions related to specific entities [73,74]. While these methods can accurately reflect the unstructured features of the text and are relatively easy to analyse and understand, they rely heavily on manually constructed ontology rules and sentiment lexicons, which can overlook the semantic relationships between different contexts within the text.

In recent years, deep neural networks have been widely applied to sentiment analysis of movie reviews. Du et al. [75] used CBOW to learn aspect word vectors and combined them with deep convolutional neural networks to model aspect sentiment and syntactic context in movie reviews. Experiments demonstrated that this model outperforms traditional word embedding methods in sentiment analysis. Ibrahim et al. [76] proposed an ABSA model that combines convolutional neural networks with an attention mechanism. This model outperforms traditional machine learning models in both aspect term recognition and sentiment polarity prediction for movie reviews. In addition, some researchers have used the pre-trained BERT model to obtain dynamic word vectors and combined it with CNN to extract local features and BiLSTM to capture global features of word vectors. Attention mechanisms are employed to highlight key information in the Douban movie review data set for sentiment classification. However, this method is limited to short texts containing only one evaluation aspect and is designed for binary classification rather than multi-class classification.

From the current research on aspect-based sentiment analysis in movie reviews, it is evident that while some methods have utilised movie ontologies, the existing movie ontologies are quite limited in terms of contextual representation. They often only include hierarchical concepts related to movie classification and fail to effectively encapsulate the overall user evaluation of movies. In addition, most studies rely on traditional heuristic methods to extract text features and then apply machine learning classifiers for sentiment analysis, which restricts the depth of text data exploration. To address these limitations, this paper proposes an aspect-based sentiment classification model that integrates a movie domain ontology. This approach aims to effectively address the lack of domain knowledge guidance and insufficient aspect feature extraction in text sentiment analysis, thereby improving the accuracy of aspect-category sentiment analysis.

3. Our methodology

3.1. Main framework

The main work of this paper is to incorporate a movie review domain ontology to enhance the effect of aspect category sentiment analysis. Our research framework is shown in Figure 2, which mainly includes the following main components: data collection, ontology construction, aspect category detection and aspect-based sentiment classification. The research content of each part is as follows.

Figure 2.

The research framework of aspect-based sentiment analysis.

Data collection: Douban Movie is a leading domestic platform with a rich collection of film resources and television resources, which has accumulated a substantial amount of accessible movie and user review data. This study crawls user review information from this platform to construct a Chinese movie review corpus.

Ontology construction: This study adopts the seven-step method to construct a domain ontology for movie reviews, systematically organising and integrating ontology classes, attributes, aspects, and relationships. First, all reviews in the corpus are tokenized, and the top 100 most important nouns are manually annotated. Then, the domain ontology is described using the OWL language and created using the Protégé tool.

Aspect category detection: Referring to the content of the movie review domain ontology, the data sets are labelled using the BIO tagging method. An aspect identification model based on ERNIE is proposed to achieve joint extraction of evaluative entities and explicit aspects. The effectiveness and accuracy of the model are validated through comparative experiments.

Aspect sentiment classification: This study constructs an aspect-based sentiment classification model, incorporating an attention mechanism to enhance the effectiveness of aspect-level sentiment analysis. The attention mechanism helps the model focus on local sentiment information specific to certain aspects, improving both the performance and accuracy of sentiment analysis. The effectiveness and accuracy of this method are validated through comparative experiments.

3.2. Data collection

Currently, the commonly used data sets in the research field of movie review sentiment analysis are IMDb movie review data set [77] and Netflix movie review data set from Kaggle, which are coarse-grained English data sets, and the data are labelled with only negative and positive sentiments, which are not applicable to aspect-based sentiment analysis tasks. In order to develop an aspect-based sentiment analysis model for online movie reviews, we use the platform Douban Movie (https://movie.douban.com/), one of the famous movie review websites in China, as a data collection source. Among the many film review websites in China, Douban, Mtime and Maoyan Movie are the three most representative. Mtime and Maoyan mainly serve as platforms for online ticket sales, with movie ratings and reviews acting as supplementary features [78]. Douban, established in 2005, attracts over 100 million active monthly visitors [79], cataloguing hundreds of thousands of films and housing tens of millions of users and a vast amount of movie review data. As a result, Douban has become a valuable data source for many previous studies[80 –82].

The steps of crawling Douban’s movie review data are as follows. (1) Start by entering the browser’s debugging mode to obtain the URL of Douban’s movie index page. Use Python’s Request method to access the movie index page and retrieve the HTML source code. Utilise Beautiful Soup along with regular expressions to extract the Douban IDs of movies from the page. (2) For each movie’s Douban ID, access the corresponding movie review page. Again, use regular expressions to match and extract each short review’s user ID. Since Douban limits the number of reviews available for crawling, a maximum of 200 short reviews and user IDs can be retrieved per movie. (3) With the user IDs, access each user’s movie review page and apply the same method to scrape all the reviews posted by that user. Through the above process, a total of 12,000 film details and about 400,000 short comments from 2,130 Douban users are crawled, constituting the Chinese film comment corpus D1.

3.3. Ontology development

Unlike common aspect identification tasks such as product names, place names, and organisation names, aspect identification in movie reviews is more abstract, diverse, and complex. On one hand, evaluative entities may not be explicitly mentioned. For example, in the comment ‘It’s so boring, makes me want to fall asleep’ there is no explicit mention of the evaluative entity. However, with the context and surrounding text, one can infer that the comment is about the plot of the movie. It is difficult to identify such implicit evaluative entities directly. On the other hand, movie reviews are highly subjective, as viewers usually express their personal preferences when writing reviews. Unlike professional film critics who use specialised terms in the movie domain, viewers rarely provide comprehensive and objective film analysis from multiple perspectives. To reduce the subjectivity in movie reviews and extract objective and consistent evaluative entities, it is necessary to establish a set of standardised rules for movie review entity identification, including conceptual relationships and the classification of entities.

Ontologies can be subdivided according to the degree of dependence on specialised domain knowledge: top-level ontology, task ontology, domain ontology and application ontology [83]. Among them, Domain ontology is one of the hot issues in ontology research. As a formal and standardised way to describe knowledge in a specific domain [84], from a conceptual perspective, domain ontology not only defines the concepts and classes of entities within a given domain but also describes the relationships, attributes, and rules between these concepts. Following the seven-step method for ontology construction [85], this paper developed the movie review domain ontology using the Protégé tool. The specific steps of ontology construction are shown in Figure 3.

Figure 3.

Seven-step approach to domain ontology construction.

Based on the Chinese movie review corpus D1, the high-frequency word information in the comments was obtained using the term frequency-inverse document frequency (TF-IDF) method. After calculating TF-IDF for the 400,000 short comments, the top 100 movie review vocabulary items were manually selected in a bottom-up manner, forming the Movie Review Terminology Corpus (N1). Subsequently, annotators conducted aspect annotation based on corpus N1. To provide a comprehensive and multi-dimensional description of the movie review domain ontology, covering all possible dimensions of a user’s evaluation of a movie, annotators paid particular attention to the following issues during the annotation process. (1) What are the most commonly considered dimensions or aspects by users when evaluating a movie? (2) Are the classification dimensions mentioned in the movie domain ontology model sufficiently represented in the user’s comment texts? (3) Do certain dimensions of movie comments have a professional nature, being present in both movie-related books and comments from ordinary viewers? (4) Is the level of user interest sufficient for aspects such as movie release time, language, shooting location, and distribution region? (5) To what extent do user evaluations of movie marketing, box office performance, and audience numbers reflect their emotional attitudes towards the movie?

Based on the comprehensive analysis above, dimensions strongly correlated with movie comments were manually selected, becoming the classification dimensions of the movie review domain ontology in this study. Table 2 below provides each dimension and its description.

Table 2.

Dimensions of the movie review domain ontology.

Aspects	Description	Example
Story	Descriptions related to film theme, plot, story, etc.	Story, plot, themes, climax, ending, etc.
Role	Descriptions related to film characters, identities, etc.	Main character, protagonist, hero, heroine, father, mother, girl, etc.
Actor	Descriptions related to film actors, stars, performances, etc.	Actors, Jackie Chan, Stephen Chow, Wu Jing, acting, performance, etc.
Audiovisual	Descriptions related to film editing, scenes, sound effects, etc.	Camera work, editing, music, special effects, tones, scenes, etc.
Genre	Descriptions related to film themes, styles, countries, genres, etc.	Subject matter, genre, style, romance, science fiction, etc.
Behind the scenes	Descriptions related to film directors, screenwriters, production teams, etc.	Directors, writers, camera crews, editors, etc.

In order to visualise the abstract ontological model of the film criticism domain, the Protégé tool was used to create classes. For example, the results for the dimensions of ‘genre’ and ‘story’ are illustrated in Figure 4.

Figure 4.

Creation of categories for genre and story aspects.

‘Movie Review’ serves as the top-level parent class in the entire ontology model, encompassing six evaluation subclasses: Story, Role, Actor, Audiovisual, Genre, and Behind the Scenes. Each subclass can also serve as a parent class for other categories. For example, the ‘Genre’ subclass includes subcategories like Science Fiction, Drama, Thriller, Mystery, and more. Based on the inheritance and hierarchy between classes, the movie review domain ontology model is depicted in Figure 5, consisting of six aspects and 23 sub-ontologies. These aspects are combined to form the movie review domain ontology model used in aspect-based sentiment classification tasks.

Figure 5.

Movie review domain ontology model.

3.4. Aspect category detection model

Although previous research on aspect category detection in movie reviews has also focused on and utilised movie ontologies, the content of these ontologies is quite limited and often fails to accurately capture users’ overall evaluations of films. Considering this, our study explores whether integrating the hierarchical structure of domain ontologies can help identify the aspect categories to which the evaluated objects belong, thereby providing additional domain knowledge to the model. Moreover, existing research has demonstrated that injecting prior knowledge from ontologies can improve model performance [86]. Inspired by this, we have developed an aspect identification model that incorporates movie review domain ontologies, as shown in Figure 6.

Figure 6.

Aspect category detection model.

Using ERNIE2.0 as the model embedding layer, each word in the input text sequence is encoded. The hidden vector corresponding to the special [CLS] token in the last layer of the encoding sequence is extracted as the feature vector of the entire text sequence. Finally, this vector is fed into a fully connected layer to obtain the layer output. The BiLSTM layer performs sequential and reverse calculations on the input sequence vectors separately. The forward LSTM model generates the forward representation, while the backward LSTM model generates the backward representation. These two different hidden states are concatenated to obtain the final hidden layer representation of the BiLSTM model. Although BiLSTM can learn contextual semantic information and accurately determine entity boundaries, it cannot handle dependency relationships between adjacent entities. Therefore, the output of BiLSTM is passed to the CRF layer, which introduces constraints on neighbouring labels to ensure the validity of the output labels and obtain an optimal label prediction sequence.

3.5. Aspect-based sentiment classification model

Aspect-based sentiment classification can detect the user’s sentiment polarity towards a particular aspect of a film, helping more moviegoers to understand the content of the film and improving the production and promotion of the film. In order to extract the multifaceted sentiment polarity that exists in the text, this paper constructs the aspect-based sentiment classification model ERNIE-BiLSTM-Attention, as shown in Figure 7, which introduces an attention mechanism to make the model focus on the local sentiment information of a specific aspect to improve the performance and accuracy of sentiment analysis.

Figure 7.

Aspect-based sentiment classification model.

Using ERNIE2.0 as the model embedding layer to transform the input text sequence into word vector form. In order to give full play to the role of ERNIE in model training, in this paper, the original input sequences are processed into the form of ‘[CLS] + aspect + [SEP] + sentence + [SEP]’. Among them, [CLS] is a special classification vector marker, which aggregates classification-related information, and [SEP] is a separator, which is used to separate different sequences when multiple sequences are input. After obtaining the hidden vector representation of the input sequence through the ERNIE embedding layer, feeding this vector into the BiLSTM model to obtain hidden layer semantics. BiLSTM helps to address the issues of vanishing or exploding gradients in RNN and enables learning of contextual semantic features in the sequence. Applying attention mechanism to better capture the dependencies between the output and input sequence, thereby learning which words are crucial for distinguishing positive sentiment, negative sentiment, or neutral sentiment. This allows the model to focus on the local feature information of specific words and entities in the text, further improving the accuracy of sentiment analysis.

After training the aforementioned two models, given an input sequence S, the aspect identification model identifies the aspects contained in S, while the sentiment analysis model analyses the sentiment orientation for specific aspects.

4. Sentiment analysis model and algorithm design

This section specifies the structure of each layer of the aspect-based sentiment analysis model and how it is calculated.

4.1. Problem formulation

This study represents a user’s movie review by $S = (t_{1}, t_{2}, \dots, t_{n})$ , designs a model that aims to predict all aspect categories $a_{i}, i \in {1, 2, \dots, N}$ contained in the review S and the sentiment polarity $s_{i} \in {0, 1, 2}$ of the aspect categories. Where $n$ denotes the length of the review S, $N$ is the number of predefined aspect categories (i.e. the 6 dimensions in this paper), and sentiment polarity has a value of 2 if it is positive, 0 if it is negative, and neutrality is noted as 1.

4.2. Creation of word vectors based on ERNIE

This study uses the ERNIE2.0 model, which is open-sourced on the Hugging Face website by Baidu PaddlePaddle, as the embedding layer for the ACSA model. For aspect-based sentiment classification tasks, the input text sequence $S = (t_{1}, t_{2}, \dots, t_{n})$ is first formatted as ‘[CLS] + aspect + [SEP] + context + [SEP]’. Next, the pre-trained model is used to encode the word vectors, and the hidden vector corresponding to the [CLS] special token in the encoded sequence is extracted as the feature vector for the entire text sequence. This vector is then fed into a fully connected layer, as shown in Equation (1), to obtain the output of this layer. The output $x = (x_{1}, x_{2}, \dots, x_{n})$ is then used as the input sequence for the subsequent BiLSTM layer. In Equation (1), E represents the feature hidden vector including the [CLS] token, $W_{a}$ denotes the weight matrix, and $b_{a}$ is the bias vector.

x_{t} = W_{a} \cdot E + b_{a}

(1)

4.3. Obtaining contextual information based on BiLSTM

Compared with LSTM, the Bidirectional Long Short-Term Memory (BiLSTM) network combines both forward and backward LSTMs. This approach allows for processing the input sequence in both directions and concatenating their outputs in subsequent layers. By considering both preceding and succeeding context information, the model can better capture the semantic dependencies in the context and improve the performance of entity recognition. The structure of the BiLSTM is illustrated in Figure 8.

Figure 8.

BiLSTM model structure diagram.

BiLSTM consists of two LSTM models that perform sequential calculations in both the forward and backward directions for each vector in the sequence $x = (x_{1}, x_{2}, \dots, x_{n})$ . The forward LSTM model computes the forward representation $h_{R}$ of the input sequence by processing it from left to right. The backward LSTM model computes the backward representation $h_{L}$ of the input sequence by processing it from right to left. The calculation method for a single LSTM is as follows:

i_{t} = σ (W_{i} \cdot [h_{t - 1}, x_{t}] + b_{i})

(2)

f_{t} = σ (W_{f} \cdot [h_{t - 1}, x_{t}] + b_{f})

(3)

o_{t} = σ (W_{o} \cdot [h_{t - 1}, x_{t}] + b_{o})

(4)

{\tilde{C}}_{t} = \tanh (W_{C} \cdot [h_{t - 1}, x_{t}] + b_{C})

(5)

C_{t} = f_{t} * C_{t - 1} + i_{t} * {\tilde{C}}_{t}

(6)

h_{t} = o_{t} * \tanh (C_{t})

(7)

The forward and backward hidden states are concatenated to obtain the final hidden representation h of the BiLSTM model, as shown in Equation (8). The final hidden state $\tilde{h}$ is calculated by combining the forward and backward hidden states, as shown in Equation (8).

{\tilde{h}}_{t} = [h_{Rt}, h_{Lt}]

(8)

H = {{\tilde{h}}_{1}, \dots, {\tilde{h}}_{t}, \dots, {\tilde{h}}_{T}}

(9)

4.4. CRF-based label prediction

Although BiLSTM can learn the contextual semantic information and determine the boundary of entities more accurately, it cannot handle the dependency relationship between neighbouring entities. Therefore, the output results of BiLSTM are fed into the CRF layer, and the CRF layer is introduced to add some constraints to the neighbouring labels, so as to ensure the validity of the output labels and obtain an optimal label prediction sequence, and the training process of the CRF layer is as follows.

First, the analysis in Section 4.3 obtained the final hidden layer representation ñ of the BiLSTM model. The dimension of ñ is (n, hidden_size), where n is the number of words in the input sequence and hidden_size is the dimension of the BiLSTM hidden layer. Next, the hidden layer output ñ from the BiLSTM is fed into a linear layer for encoding, mapping the hidden state to a matrix P. The size of P is (n, tagset_size), where n is the number of words in the input sequence and tagset_size is the number of label categories. Finally, $P_{i, j}$ to is defined as the probability that the ith word in the sentence corresponds to the jth label. The transition matrix A is used as the parameter for the CRF layer, where $A_{i, j}$ represents the probability of transitioning from label i to label j. The CRF layer is trained using maximum conditional likelihood estimation. For an input sequence $X = {x_{1}, x_{2}, . ., x_{n}}$ , the calculation of its output label sequence $Y = {y_{1}, y_{2}, . ., y_{n}}$ is as follows.

P (y | X) = \frac{\exp (score (X, y))}{\sum_{\tilde{y} \in Y_{X}} \exp (score (X, \tilde{y}))}

(10)

score (X, y) = \sum_{i = 0}^{n} A_{y_{i}, y_{i + 1}} + \sum_{i = 1}^{n} P_{i, yi}

(11)

In the above Equation (10) and Equation (11), $\tilde{y}$ represents the true label value. $Y_{X}$ represents the set of all possible labels, including label sequences that do not conform to the BIO labelling rules. During training, maximum likelihood estimation is used, and the corresponding loss function is given in Equation 12.

- \log (P (y \ X)) = - score (X, y) + \log (\sum_{\tilde{y} \in Y_{X}} \exp (score (X, \tilde{y}))

(12)

Effective and reasonable output sequences are obtained by Equation (10) and (11). Then, the Viterbi algorithm is applied to output a group of predicted label sequences with the maximum overall score probability, as shown in Equation (13).

y^{*} = \underset{\tilde{y} \in Y_{X}}{argmax} (score (X, \tilde{y})) Y_{X}

(13)

4.5. Focus on local information based on attention mechanism

Using attention mechanism can help the model better capture the dependencies between the output and input sequences, thereby learning which words are more important for distinguishing positive, negative, or neutral sentiments. This allows the model to focus on specific words and entities in the text, capturing local feature information and enhancing the semantic representation of the model, thus improving the performance and accuracy of sentiment analysis. In the aspect-based sentiment classification model, the hidden layer $H = {{\tilde{h}}_{1}, \dots, {\tilde{h}}_{t}, . ., {\tilde{h}}_{T}}$ obtained from the BiLSTM layer is used as input for the attention mechanism calculation, as shown in Equation (14) to (16).

Q = W^{q} H

(14)

K = W^{K} H

(15)

V = W^{v} H

(16)

Here, $W^{q}$ , $W^{k}$ , $W^{v}$ represents the weight matrix and $H$ is the hidden layer representation. The formula for the self-attention mechanism is shown in Equation (17). $\sqrt{d_{k}}$ is the dimension of the key, which is used to scale the dot product to avoid getting too large a value, and $R$ is the output of the attention mechanism layer, which represents the final response of the model to the current focus or query.

R = softmax (\frac{Q K^{T}}{\sqrt{d_{k}}}) V

(17)

Next, the fully connected layer maps the feature representation to the sentiment label space by multiplying the weight matrix with the output $R$ of the self-attention mechanism module plus a bias, which is equivalent to the action of a classifier. The output results z from the fully connected layer are then subjected to softmax normalisation, as calculated in Equation (18) and (19), to obtain the probability values $\hat{y}$ for each sentiment category. The sum of all $\hat{y}$ values is equal to 1. Here, $x$ represents the input of the fully connected layer, $W$ is the weight matrix of the fully connected layer, b is the bias vector, and $\hat{y}$ denotes the output probabilities from the softmax function.

\hat{y} = softmax (z) = softmax (W^{T} x + b)

(18)

softmax (z_{j}) = \frac{e^{z_{i}}}{\sum_{j = 1}^{K} e^{z_{j}}}

(19)

5. Experimental results

5.1. Experimental data set

This study used a data set consisting of 400,000 user movie short reviews crawled from the Douban movie platform during the construction process of the movie review domain ontology in Section 3. User short reviews often contain not only evaluations of the movie but also abstract and personal comments unrelated to the movie itself. To avoid the interference caused by such comments, we extracted 10,000 valid comments containing explicit entities from 1,000 users. These comments include at least one aspect and sentiment. The specific scale of the aspect category detection experiment corpus is shown in Table 3, and the scale of the aspect-based sentiment classification experiment corpus is shown in Table 4.

Table 3.

Introduction to data size for aspect category detection (unit: sentence).

Corpus	Training set	Validation set	Testing set
Douban Movie Reviews	6000	2000	2000

Table 4.

Introduction to Douban Movie data size for sentiment analysis (unit: sentence).

Aspects	Data set	Training set	Validation set	Testing set
Story	d1	4096	1365	1366
Role	d2	2527	842	843
Actor	d3	2361	787	787
Audiovisual	d4	1928	642	643
Genre	d5	1935	644	645
Behind the scenes	d6	1824	608	608

To ensure a fair comparison between the proposed method and other methods, this study also used the open-source Chinese restaurant review data set ASAP proposed by Bu et al. [27] The processed corpus scale is shown in the Table 5.

Table 5.

Introduction to ASAP data size for sentiment analysis (unit: sentence).

Aspects	Data set	Training set	Validation set	Testing set
Location	a1	14064	1883	1921
Food	a2	35536	4768	4764
Service	a3	25562	3462	3392
Price	a4	24793	3362	3264
Ambience	a5	22180	3037	2935

This data set consists of 46,730 real reviews from China’s leading online and offline e-commerce platforms, which were manually annotated with five aspectual categories and sentiment polarity by the researcher. Aspect categories were represented as ‘coarse-grained categories # fine-grained categories’ (e.g. ‘environment # traffic’, ‘environment # distance’). The sentiment polarity of the aspect categories was labelled as -2(Not Mentioned),1 (Positive),0 (Neutral) or -1 (Negative). In order to use the data set for this experimental research, we performed a simple processing of the data set, consolidating the sentiment polarities according to predefined aspect categories (e.g. location, food, service, price, and ambience).

5.2. Data set annotation

In the aspect category detection experiment, combining with the movie review domain ontology, the evaluation entities appearing in the review texts can be classified into one of the six ontology dimensions: Story, Character, Audiovisual, Actor, Genre, and Behind the scenes. Considering the characteristics and categories of movie review entities, the movie review data set was manually annotated using the BIO tagging scheme. The rules for BIO sequence tagging are as follows: the beginning of an evaluation entity is marked as ‘B-X’ the middle and end are marked as ‘I-X’ and words that do not belong to an evaluation entity are marked as ‘O’. The specific method for data annotation is as follows: First, the open-source website Label Studio is used to annotate the ontology dimension to which the evaluation entity belongs, as shown in the above Figure 9. Then, code is written to convert the annotation information into BIO tagging format.

Figure 9.

Label-studio annotation methods.

In the sentiment analysis experiment, manual analysis was employed to determine the sentiment orientation of users towards each aspect. This task involved annotating the sentiment of the user comments. If the sentiment was positive, it was labelled as 2; if negative, it was labelled as 0; and if neutral, it was labelled as 1. As for evaluation aspects that were not mentioned, no label classification was applied. The annotated results are presented in the following Table 6.

Table 6.

Example of sentiment analysis corpus annotation.

Comment text	Story	Role	Actor	Audiovisual	Genre	Behind the scenes
Through a simple narrative, the film depicts a sublime and complex human nature and emotions, full of philosophical, literary and life significance of the struggle theme. The director shoots poetic documentary-style shots in the first half of the film, and the second half shows the violent and ruthless real world. In the darkness before the dawn of the revolution, anger is magnified in the close-up eyes of the actors. The execution sequence is so godlike that you can’t stop imagining where the plot is going to go next.	2		1	2	2	1

5.3. Experimental configuration

The hardware and software environments involved in this experiment are configured as shown in Table 7 below.

Table 7.

Experimental environment configuration information.

Software and hardware environment	Configuration
Operating systems	Windows10 x64
Development languages	Python3.7
Deep learning frameworks	Pytorch1.10
CPU	Intel Core i7-13700 CPU
GPU	NVIDIA GeForce RTX 4060Ti

During model training, the pre-trained language model used in this study, ERNIE2.0, has 12 hidden layers with a dimension of 768. It employs 12 attention heads and utilises the ReLU activation function for the hidden layers. The model has a total of 110 million parameters. The BiLSTM hidden layer has a dimension of 256.

5.4. Baseline methodology

5.4.1. Baseline models for ACD task

In order to demonstrate the effectiveness of the proposed model in aspect category detection task, four different methods were deployed on the movie review data set, including the proposed method in this paper and three baseline methods.

BiLSTM-CRF [87]: this method uses BiLSTM to generate word embeddings and then utilises CRF to model the transition probabilities between labels [87]. By considering contextual information and label transition constraints, it optimises the aspect category detection task to obtain more accurate entity boundaries and label predictions.

W2V-BiLSTM-CRF [88]: this method employs the Word2Vec model to represent words in the text as dense vectors [88]. It then utilises a BiLSTM model to learn contextual information and uses a CRF layer to model the transition probabilities between labels, resulting in an output sequence.

BERT-BiLSTM-CRF [89]: this method leverages the BERT model to obtain hidden vector representations [89]. It then utilises BiLSTM to capture the dependencies between words in the text and employs a CRF layer for label prediction, generating the output sequence.

ERNIE-BiLSTM-CRF: the proposed method in this paper uses the ERNIE2.0 model to train word embeddings. It then utilises both forward and backward LSTMs to learn contextual information in the text. Finally, a CRF layer is used to obtain the output sequence.

5.4.2. Baseline models for ASC task

In the aspect-based sentiment classification task, this paper deployed three different types of modelling approaches.

Foundation Models: including Bert-Base-Chinese (referred to as Chinese BERT), ERNIE 2.0 (referred to as ERNIE), BERT-WWM and RoBERTa models.

Bert-Base-Chinese [90]: is developed by the Natural Language Processing Laboratory of Tsinghua University based on Google’s BERT model, using a large-scale Chinese corpus for training, and mainly consists of two pre-training tasks, Masked Language Modelling (MLM) and Next Sentence Prediction (NSP).

ERNIE2.0 [91]: model further optimises the masking process of BERT with entity-level masking and phrase-level masking, where entity-level masking differs from randomly selecting words in the input by masking named entities.

BERT-WWM [92]; changes the training sample generation strategy of the original BERT in the pre-training phase, and applies the whole-word Mask approach to Chinese, and uses Chinese Wikipedia (both simplified and traditional) for training.

RoBERTa [93]: is an enhanced version of the BERT model, which uses dynamic masks to train the model and improves on the original BERT model in terms of the number of parameters and optimisation.

Enhanced Models: this study trained the following four models based on the original pedestal model itself, including BERT-BiLSTM, ERNIE-BiLSTM, BERT-WWM-BiLSTM and RoBERTa-BiLSTM. The hidden vector representation was obtained from the original base model and then BiLSTM was used to capture the dependencies between words in the text.

Attention Enhanced Models: this study trained the following four models based on the original base model itself, including BERT-BiLSTM-ATT, ERNIE-BiLSTM-ATT, BERT-WWM-BiLSTM-ATT, and RoBERTa-BiLSTM-ATT. The augmentation-based model added a multi-attention mechanism to allow the model to focus on local sentiment information in specific aspects, thus enhancing the model’s performance on multifaceted sentiment classification.

5.5. Results and analysis

This study conducted extensive experiments to evaluate the performance of our aspect-level sentiment analysis model on Douban Movie and ASAP data sets. In addition, ablation experiments were performed to discuss the impact of incorporating the attention module.

5.5.1. Aspect category detection results

Table 8 reports the performance of the different models on the task of aspect category detection. It can be found that the ERNIE-BiLSTM-CRF model achieves the best results in precision, recall and F1 metrics with 89.94%, 92.49% and 91.19% respectively. The reason for this is that the ERNIE model can effectively capture the relationship features at the character, word, and sentence levels, which enhances the expressive power of the text vectorisation model. As a result, it performs better in learning entity information compared with the Word2Vec model in the Audiovisual dimension. Compared with the BERT model, ERNIE employs a masked language modelling mechanism with prior knowledge, which masks phrases, words, and entities in the text. This implicitly learns the relationships between entities, words, and objects, enabling better semantic representation of the review text. As a result, the ERNIE-based model achieved improvements of 1.77%, 0.13%, and 0.97% in precision, recall, and F1 score, respectively, compared with the BERT-based model.

Table 8.

Results of aspect category detection.

Entity-owned aspects	Model	Precision (P)	Recall (R)	F1
Audiovisual	BiLSTM-CRF [87]	76.91%	74.25%	75.56%
	W2V-BiLSTM-CRF [88]	79.63%	80.38%	80.00%
	BERT-BiLSTM-CRF [89]	88.17%	92.36%	90.22%
	ERNIE-BiLSTM-CRF	89.94%	92.49%	91.19%

The ERNIE-BiLSTM-CRF model for entity recognition in the context of deep learning relies on a large amount of annotated corpus data. Due to time constraints, this study only annotated 10,000 comments. There is still considerable room for improvement in the model’s performance by increasing the amount of manually annotated data. On the other hand, the quality of data annotation also plays a role in model training. Different annotators may have different judgments regarding the evaluation dimensions of comment entities, and this human judgement error can also affect the training results of the model.

5.5.2. Aspect-based sentiment classification results

In the aspect-based sentiment classification task, this study used four different base models to create word embeddings and described how the performance of different models changes with increasing training epochs. Figure 10 shows the test performance of the models on the Douban Movie data set. Since BERT-like models are pre-trained, we found that the models start to overfit on the training data set after more than 5 epochs. Therefore, we set the number of epochs to 4 for the experiment.

Figure 10.

ACC (left) and LOSS (right) performance of foundation models with increasing number of epochs.

Figures 10 –12 shows that different models require varying numbers of iterations to achieve optimal prediction performance. For foundation models, the best performance is achieved within 2 to 3 epochs, with the loss values being lowest at the 3rd epoch for BERT and ERNIE, and the highest accuracy at the 2nd epoch for RoBERTa and BERT-WWM model. Enhanced models, which include the BiLSTM module to capture bidirectional semantic dependencies, require more training iterations to reach optimal accuracy, generally achieving this by the 4th epoch. Attention-enhanced models, which incorporate self-attention mechanisms to learn local contextual dependencies, show that our proposed ERNIE-BiLSTM-ATT model performs well by the 2nd epoch, while other models reach better performance around the 3rd epoch.

Figure 11.

ACC (left) and LOSS (right) performance of enhanced models with increasing number of epochs.

Figure 12.

ACC (left) and LOSS (right) performance of attention enhanced models with increasing number of epochs.

To illustrate the effectiveness of our proposed aspect-based sentiment classification model, we conducted further experiments on two different data sets, and the performance of the three types of models is shown in Table 9. In the foundation model, the ERNIE model has an accuracy of more than 75% on both data sets, which is better than the BERT, BERT-WWM and RoBERTa models because it is pre-trained based on a massive Chinese corpus, and the model focuses on the information of words and entities, so it is better than the other models on the Chinese data set. In the enhanced model, the addition of BiLSTM layer gives better access to the bidirectional dependencies of the text and learns more semantic information, so the augmentation models all work better than the base model alone. The attention enhancement model adds the self-attention mechanism module, which will directly link the relationship between any two words in a sentence through one computational step, making it easier for the model to capture the features of long-distance interdependence in a sentence. In summary, the ERNIE-BiLSTM-ATT model constructed in this study achieved the best results on both data sets.

Table 9.

Results of aspect-based sentiment classification model on Douban Movie data set and ASAP data set.

Category	Model	Douban Movie		ASAP
		Macro-F1	Acc	Macro-F1	Acc
Foundation model	BERT	70.89%	75.33%	69.41%	72.53%
	BERT-WWM	68.45%	72.00%	70.35%	74.00%
	RoBERTa	68.92%	71.33%	68.93%	73.00%
	ERNIE	77.82%	78.00%	72.30%	75.87%
Enhanced model	BERT-BiLSTM	74.26%	78.67%	72.48%	75.87%
	BERT-WWM-BiLSTM	70.46%	74.00%	70.83%	75.00%
	RoBERTa-BiLSTM	68.66%	73.33%	69.25%	73.47%
	ERNIE-BiLSTM	81.48%	82.00%	74.90%	77.93%
Attention enhanced model	BERT-BiLSTM-ATT	77.08%	79.33%	73.08%	76.20%
	BERT-WWM-BiLSTM-ATT	74.06%	77.33%	70.90%	75.47%
	RoBERTa-BiLSTM-ATT	71.89%	74.67%	70.83%	74.60%
	ERNIE-BiLSTM-ATT	84.26%	86.00%	77.16%	81.00%

Figure 13 shows the effect of introducing the self-attention mechanism. The user comment mentions, ‘Watching the HD restored version is really a visual enjoyment, I found the feeling of watching the drama when I was a kid, but the characterisation is a bit thin now’ (). This sentence involves two aspects audiovisual and role. The attention module in Figure 13(a) focuses on the statement ‘audiovisual’, while the attention module in Figure 13(b) focuses on the ‘role’ aspect. It is evident that by using the self-attention mechanism, our proposed model can pinpoint the key aspects’ local information and focus on the sentiment analysis of those aspects, thereby achieving better sentiment analysis results.

Figure 13.

Attentional visualisation. Aspects in (a) and (b) are audiovisual and role, respectively. The colour depth indicates the importance of the attention vector weights. From (a), attention dynamically detects important audiovisual-related utterances from the whole sentence. From (b), attention focuses more on role-related words.

To further validate the effectiveness of the aspect-based sentiment classification model ERNIE-BiLSTM-Attention, this study conducted ablation experiments on the Douban Movie data set. The experimental results are shown in Table 10.

Table 10.

Results of ablation experiments.

Model	Precision (P)	Recall (R)	Accuracy	Macro F1
ERNIE	79.14%	78.74%	78.00%	77.82%
ERNIE-BiLSTM	81.43%	81.74%	82.00%	81.48%
ERNIE-BiLSTM-Attention	86.72%	83.30%	86.00%	84.26%

By analysing the data in the table, the following observations can be made: (1) The ERNIE-BiLSTM model performs better than the standalone ERNIE model. This can be attributed to the fact that while using the ERNIE model enriches the semantic representation of the text, it may not capture the global features of the text effectively. By incorporating the BiLSTM layer, the model can better capture the contextual dependencies and learn more contextual semantic information. (2) The ERNIE-BiLSTM-Attention model shows significantly better sentiment prediction performance compared with the standalone ERNIE model and ERNIE-BiLSTM model. The reason behind this improvement is that the ERNIE-BiLSTM model may struggle to accurately identify the sentiment of a specific aspect in the presence of multiple aspect entities with different sentiment orientations in the text. By incorporating the self-attention mechanism, the model is able to compute the correlation between different parts of the input text, thus focusing on the sentiment tendency judgement of one aspect dimension and achieving better sentiment analysis results.

6. Conclusion

To address the issues of limited application of aspect-based sentiment analysis in the movie review domain, lack of domain knowledge guidance leading to the model’s inability to capture key information, and insufficient aspect feature extraction, this study proposes an aspect-based sentiment analysis method based on the ERNIE2.0 model. The main contributions of this work include (1) Developed a domain ontology for movie reviews based on a seven-step ontology construction method, providing a reference for aspect-based sentiment analysis tasks. (2) Created a model tailored for online movie reviews that integrates external domain knowledge into a pre-trained language model. This model can extract evaluation aspects from review texts and analyse the sentiment for each aspect. Experimental results demonstrate that this model outperforms baseline models in sentiment analysis effectiveness. (3) Addressed the issue of lacking Chinese movie review corpora for aspect-based sentiment analysis by constructing a corpus based on Douban movie reviews. However, it should be noted that the domain-specific ontology model and sentiment analysis corpus proposed in this paper are limited to movie review research. Therefore, in the future, we will focus on expanding sentiment analysis applications in different domains, sharing and transferring knowledge of aspect-based sentiment analysis between different domains to improve the performance of the models in new domains. Furthermore, as aspect-based sentiment analysis research progresses, we will further explore the research of aspect-based sentiment triplets to enhance the accuracy and applicability of aspect-based sentiment analysis.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iDs

Hanyue Mao

Mingwen Tong

Availability of data and materials

The data supporting the results of this study were crawled from the Douban movie platform, but the data was not published because of the privacy issues of the platform users. If researchers and readers promise not to use the data set commercially, but only for personal research, the data that support the findings of this study are available upon reasonable request.

References

Liu

Sentiment analysis and opinion mining. Synth Lect Hum Lang Technol 2012; 5(1): 1–167.

Schouten

Frasincar

Survey on aspect-level sentiment analysis. IEEE Trans Knowl Data Eng 2015; 28(3): 813–830.

Liang

Gui

, et al. Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks. Knowl-Based Syst 2022; 235: 107643.

Wagner

Arora

Cortes

, et al. DCU: aspect-based polarity classification for SemEval task 4. In: Proceedings of the 8th international workshop on semantic evaluation, Dublin, August 2014, pp. 223–229. New York: Association for Computational Linguistics.

Ruz

Henríquez

Mascareño

Sentiment analysis of Twitter data during critical events through Bayesian networks classifiers. Future Gener Comput Syst 2020; 106: 92–104.

Ahmad

Aftab

Ali

Sentiment analysis of tweets using SVM. Int J Comput Appl 2017; 177(5): 25–29.

Naz

Ahuja

Kumar

, et al. DT-FNN based effective hybrid classification scheme for Twitter sentiment analysis. Multimed Tools Appl 2021; 80: 11443–11458.

Al-Smadi

Al-Ayyoub

Jararweh

, et al. Enhancing aspect-based sentiment analysis of Arabic hotels’ reviews using morphological, syntactic and semantic features. Inf Process Manag 2019; 56(2): 308–319.

Sivakumar

Reddy

. Aspect based sentiment analysis of students opinion using machine learning techniques. In: 2017 international conference on inventive computing and informatics (ICICI), Coimbatore, India, 23–24 November 2017, pp. 726–731. New York: IEEE.

10.

Girija

Sudha

. A comparative review on approaches of aspect level sentiment analysis. In: 2023 Third international conference on artificial intelligence and smart energy (ICAIS), Coimbatore, India, 2–4 February 2023, pp. 770–777. New York: IEEE.

11.

Huang

Guo

Zhu

, et al. Aspect-level sentiment analysis with aspect-specific context position information. Knowl-Based Syst 2022; 243: 108473.

12.

Gan

Wang

Zhang

Multi-entity sentiment analysis using self-attention based hierarchical dilated convolutional neural network. Future Gener Comput Syst 2020; 112: 116–125.

13.

Ray

Chakrabarti

A mixed approach of deep learning method and rule-based method to improve aspect level sentiment analysis. Appl Comput Inf 2022; 18(1–2): 163–178.

14.

Nguyen

Le Nguyen

. Effective attention networks for aspect-level sentiment classification. In: 2018 10th International conference on knowledge and systems engineering (KSE), Ho Chi Minh City, Vietnam, 1–3 November 2018, pp. 25–30. New York: IEEE.

15.

Peng

Cambria

Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. Proc AAAI Conf Artif Intell 2018; 32(1): 5876–5883.

16.

Sun

Huang

Qiu

. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In: Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, Minneapolis, MN, June 2019, vol 1, pp. 380–385. New York: Association for Computational Linguistics.

17.

Ong

DC.

Context-guided BERT for targeted aspect-based sentiment analysis. Proc AAAI Conf Artif Intell 2021; 35(16): 14094–14102.

18.

Kandhro

Ali

Uddin

, et al. Exploring aspect-based sentiment analysis: an in-depth review of current methods and prospects for advancement. Knowl Inf Syst 2024; 66: 3639–3669.

19.

Bing

Zhang

, et al. Exploiting BERT for end-to-end aspect-based sentiment analysis. In: Proceedings of the 5th workshop on noisy user-generated text, Hong Kong, China, November 2019, pp. 34–41. New York: Association for Computational Linguistics.

20.

Zhao

Knowledge-enabled BERT for aspect-based sentiment analysis. Knowl-Based Syst 2021; 227: 107220.

21.

Jiang

Song

, et al. Research on sentiment classification for netizens based on the BERT-BiLSTM-TextCNN model. PeerJ Comput Sci 2022; 8: e1005.

22.

Talaat

AS.

Sentiment analysis classification system using hybrid BERT models. J Big Data 2023; 10(1): 110.

23.

Jain

Quamer

Saravanan

, et al. Employing BERT-DCNN with sentic knowledge base for social media sentiment analysis. J Ambient Intell Humaniz Comput 2023; 14(8): 10417–10429.

24.

Pontiki

DGM

Pavlopoulos

HPJ

Androutsopoulos

. SemEval-2014 task 4: SemEval-2014 Task 4: aspect based sentiment analysis. In: Proceedings of the 8th international workshop on semantic evaluation, Dublin, August 2014, pp. 23–24. New York: Association for Computational Linguistics.

25.

Pontiki

Galanis

Papageorgiou

, et al. SemEval-2015 Task 12: aspect based sentiment analysis. In: Proceedings of the 9th international workshop on semantic evaluation, Denver, CO, June 2015, pp. 486–495. New York: Association for Computational Linguistics.

26.

Pontiki

Galanis

Papageorgiou

, et al. SemEval-2016 Task 5: aspect based sentiment analysis. In: Proceedings of the 10th international workshop on semantic evaluation, San Diego, CA, June 2016, pp. 19–30. New York: Association for Computational Linguistics.

27.

Ren

Zheng

, et al. ASAP: a Chinese review dataset towards aspect category sentiment analysis and rating prediction. In: Proceedings of the 2021 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, June 2021, pp. 2069–2079. New York: Association for Computational Linguistics.

28.

Wang

Sun

, et al. Aspect sentiment classification towards question-answering with reinforced bidirectional attention network. In: Proceedings of the 57th annual meeting of the Association for Computational Linguistics, Florence, July 2019, pp. 3548–3557. New York: Association for Computational Linguistics.

29.

Zhuang

Jing

Zhu

XY.

Movie review mining and summarization. In: Proceedings of the 15th ACM international conference on Information and knowledge management, Arlington, VA, 6–11 November 2006, pp. 43–50. New York: ACM.

30.

Zhao

. Ontology based opinion mining for movie reviews. In: International conference on knowledge science, engineering and management, Vienna, 25–27 November 2009, pp. 204–214. Berlin: Springer-Verlag.

31.

Regatte

Gangula

RRR

Mamidi

. Dataset creation and evaluation of aspect based sentiment analysis in Telugu, a low resource language. In: Proceedings of the twelfth language resources and evaluation conference, Marseille, May 2020, pp. 5017–5024. Paris: European Language Resources Association.

32.

Zhang

Deng

, et al. A survey on aspect-based sentiment analysis: tasks, methods, and challenges. IEEE Trans Knowl Data Eng 2022; 35(11): 11019–11038.

33.

Zhou

Wan

Xiao

Representation learning for aspect category detection in online reviews. Proc AAAI Conf Artif Intell 2015; 29(1): 417–423.

34.

Kumar

Saini

Sharan

Aspect category detection using statistical and semantic association. Comput Intell 2020; 36(3): 1161–1182.

35.

Miao

Cheng

, et al. Aspect-based sentiment analysis in Chinese based on mobile reviews for BiLSTM-CRF. J Intell Fuzzy Syst 2021; 40(5): 8697–8707.

36.

Zhao

Guo

, et al. Multi-label few-shot learning for aspect category detection. In: Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing, 1–6 August 2021, pp. 6330–6340. New York: Association for Computational Linguistics.

37.

Shi

Wang

, et al. A simple and effective self-supervised contrastive learning framework for aspect detection. Proc AAAI Conf Artif Intell 2021; 35(15): 13815–13824.

38.

Wang

Pan

Dahlmeier

, et al. Recursive neural conditional random fields for aspect-based sentiment analysis. In: Proceedings of the 2016 conference on empirical methods in natural language processing, Austin, TX, November 2016, pp. 616–626. New York: Association for Computational Linguistics.

39.

Jiang

Xia

Global inference for aspect and opinion terms co-extraction based on multi-task neural networks. IEEE/ACM Trans Audio Speech Lang Process 2018; 27(1): 168–177.

40.

Veyseh

APB

Nouri

Dernoncourt

, et al. Introducing syntactic structures into target opinion word extraction with deep learning. In: Proceedings of the 2020 conference on empirical methods in natural language processing, November 2020, pp. 8947–8956. New York: Association for Computational Linguistics.

41.

Mensah

Sun

Aletras

. An empirical study on leveraging position embeddings for target-oriented opinion words extraction. In: Proceedings of the 2021 conference on empirical methods in natural language processing, Punta Cana, Dominican Republic, 7–11 November 2021, pp. 9174–9179. New York: Association for Computational Linguistics.

42.

Davidov

Tsur

Rappoport

Enhanced sentiment learning using Twitter hashtags and smileys. In: Coling 2010: posters, Beijing, China, August 2010, pp. 241–249. Beijing, China: Coling2010Organizing Committee.

43.

Jiang

Zhou

, et al. Target-dependent Twitter sentiment classification. In: Proceedings of the 49th annual meeting of the Association for Computational Linguistics: human language technologies, Portland, OR, June 2011, pp. 151–160. New York: Association for Computational Linguistics.

44.

Wang

Huang

Zhu

, et al. Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 conference on empirical methods in natural language processing, Austin, TX, November 2016, pp. 606–615. New York: Association for Computational Linguistics.

45.

Meng

Wei

Liu

, et al. Aspect based sentiment analysis with feature enhanced attention CNN-BiLSTM. IEEE Access 2019; 7: 167240–167249.

46.

Chen

Sun

Wang

, et al. Aspect sentiment classification with document-level sentiment preference modeling. In: Proceedings of the 58th annual meeting of the Association for Computational Linguistics, July 2020, pp. 3667–3677. New York: Association for Computational Linguistics.

47.

Dai

Yan

Sun

, et al. Does syntax matter? A strong baseline for aspect-based sentiment analysis with Roberta. In: Proceedings of the 2021 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, June 2021, pp. 1816–1829. New York: Association for Computational Linguistics.

48.

Schmitt

Steinheber

Schreiber

, et al. Joint aspect and polarity classification for aspect-based sentiment analysis with end-to-end neural networks. In: Proceedings of the 2018 conference on empirical methods in natural language processing, Brussels, October 2018, pp. 1109–1114. New York: Association for Computational Linguistics.

49.

Wang

Sun

Huang

, et al. Aspect-level sentiment analysis using as-capsules. In: The World Wide Web conference, San Francisco CA, 13–17 May 2019, pp. 2033–2044. New York: ACM.

50.

Cai

Zhou

, et al. Aspect-category based sentiment analysis with hierarchical graph convolutional network. In: Proceedings of the 28th international conference on computational linguistics, Barcelona, December 2020, pp. 833–843. New York: International Committee on Computational Linguistics.

51.

Yin

Zhong

, et al. Multi-instance multi-label learning networks for aspect-category sentiment analysis. In: Proceedings of the 2020 conference on empirical methods in natural language processing, November 2020; 3550–3560. New York: Association for Computational Linguistics.

52.

Liu

Teng

Cui

, et al. Solving aspect category sentiment analysis as a text generation task. In: Proceedings of the 2021 conference on empirical methods in natural language processing, Punta Cana, Dominican Republic, 7–11 November 2021, pp. 4406–4416. New York: Association for Computational Linguistics.

53.

Chen

Liu

Wang

, et al. Synchronous double-channel recurrent network for aspect-opinion pair extraction. In: Proceedings of the 58th annual meeting of the Association for Computational Linguistics, July 2020, pp. 6515-6524. New York: Association for Computational Linguistics.

54.

Zhao

Huang

Zhang

, et al. SpanMlt: a span-based multi-task learning framework for pair-wise aspect and opinion terms extraction. In: Proceedings of the 58th annual meeting of the Association for Computational Linguistics, July 2020, pp. 3239–3248. New York: Association for Computational Linguistics.

55.

Ying

Zhao

, et al. Grid tagging scheme for aspect-oriented fine-grained opinion extraction. In: Findings of the Association for Computational Linguistics: EMNLP, November 2020, pp. 2576–2585. New York: Association for Computational Linguistics.

56.

Fei

Ren

, et al. Learn from syntax: improving pair-wise aspect and opinion terms extraction with rich syntactic knowledge. In: IJCAI international joint conference on artificial intelligence, Montreal, QC, Canada, 19–26 August 2021, pp. 3957–3963. https://www.ijcai.org/proceedings/2021/0545.pdf

57.

Peng

Bing

, et al. Knowing what, how and why: a near complete solution for aspect-based sentiment analysis. Proc AAAI Conf Artif Intell 2020; 34(5): 8600–8607.

58.

, et al. Position-aware tagging for aspect sentiment triplet extraction. In: Proceedings of the 2020 conference on empirical methods in natural language processing, November 2020, pp. 2339–2349. New York: Association for Computational Linguistics.

59.

Chen

Wang

Liu

, et al. Bidirectional machine reading comprehension for aspect sentiment triplet extraction. Proc AAAI Conf Artif Intell 2021; 35(14): 12666–12674.

60.

Cai

Xia

Aspect-category-opinion-sentiment quadruple extraction with implicit aspects and opinions. In: Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing, 1–6 August 2021, pp. 340–350. New York: Association for Computational Linguistics.

61.

, et al. An aspect-category-opinion-sentiment quadruple extraction with distance information for implicit sentiment analysis. Inf Technol Control 2023; 52(2): 445–456.

62.

Chebolu

SUS

Rosso

Kar

, et al. Survey on aspect category detection. ACM Comput Surv 2022; 55(7): 1–37.

63.

Jakob

Gurevych

Extracting opinion targets in a single and cross-domain setting with conditional random fields. In: Proceedings of the 2010 conference on empirical methods in natural language processing, Cambridge, MA, October 2010, pp. 1035–1045. New York: Association for Computational Linguistics.

64.

Colón-Ruiz

Segura-Bedmar

Comparing deep learning architectures for sentiment analysis on drug reviews. J Biomed Inf 2020; 110: 103539.

65.

Tan

Tian

, et al. Extraction of temporal information from social media messages using the BERT model. Earth Sci Inf 2022; 15(1): 573–584.

66.

Liu

Zhang

. Attention modeling for targeted sentiment. In: Proceedings of the 15th conference of the European chapter of the Association for Computational Linguistics, Valencia, April 2017, vol 2, pp. 572–577. New York: Association for Computational Linguistics.

67.

Song

Wang

Jiang

, et al. Attentional encoder network for targeted sentiment classification. arXiv 2019, https://arxiv.org/pdf/1902.09314

68.

Zeng

Yang

, et al. LCF: a local context focus mechanism for aspect-based sentiment classification. Appl Sci 2019; 9(16): 3389.

69.

Zhou

Shen

Chen

, et al. Hierarchical dual graph convolutional network for aspect-based sentiment analysis. Knowl-Based Syst 2023; 276: 110740.

70.

Zhang

, et al. Aspect-level sentiment classification based on attention-BiLSTM model and transfer learning. Knowl-Based Syst 2022; 245: 108586.

71.

Sejwal

Abulaish

CAMO: a context-aware movie ontology generated from LOD and movie databases. Multimed Tools Appl 2021; 80(5): 7247–7269.

72.

Penalver-Martinez

Garcia-Sanchez

Valencia-Garcia

, et al. Feature-based opinion mining through ontologies. Expert Syst Appl 2014; 41(13): 5995–6008.

73.

Thet

Khoo

CSG

. Aspect-based sentiment analysis of movie reviews on discussion boards. J Inf Sci 2010; 36(6): 823–848.

74.

Bancken

Alfarone

Davis

Automatically detecting and rating product aspects from textual customer reviews. In: Proceedings of the 1st international conference on interactions between data mining and natural language processing, Nancy, 15 September 2014, pp. 1–16. Aachen: CEUR-WS.org.

75.

Cheng

, et al. Aspect-specific sentimental word embedding for sentiment analysis of online reviews. In: Proceedings of the 25th international conference companion on World Wide Web, Montréal, QC, Canada, 11–15 April 2016, pp. 29–30. Geneva: International World Wide Web Conferences Steering Committee.

76.

Ibrahim

Zandam

Adam

, et al. A deep convolutional neural network-based model for aspect and polarity classification in Hausa movie reviews. arXiv 2024, https://arxiv.org/pdf/2405.19575

77.

Maas

Daly

Pham

, et al. Learning word vectors for sentiment analysis. In: Proceedings of the 49th annual meeting of the Association for Computational Linguistics: human language technologies, Portland, OR, June 2011, pp. 142–150. New York: Association for Computational Linguistics.

78.

Hao

Kim

Typical opinions mining based on Douban film comments in animated movies. Entertain Comput 2021; 36: 100391.

79.

Yang

Yecies

Mining Chinese social media UGC: a big-data framework for analyzing Douban movie reviews. J Big Data 2016; 3: 1–23.

80.

Abisado

Text sentiment analysis of Douban film short comments based on BERT-CNN-BiLSTM-Att model. IEEE Access 2024; 12: 45229–45237.

81.

Cai

, et al. A graph convolutional network based on object relationship method under linguistic environment applied to film evaluation. Inf Sci 2022; 608: 1283–1300.

82.

Zhang

Movie recommendation algorithm based on sentiment analysis and LDA. Procedia Comput Sci 2022; 199: 871–878.

83.

Guarino

Understanding, building and using ontologies. Int J Hum-Comput Stud 1997; 46(2–3): 293–310.

84.

Borst

WN.

Construction of engineering ontologies for knowledge sharing and reuse. PhD Dissertation, Institute for Telematica and Information Technology, University of Twente, Enschede, 1997.

85.

Noy

McGuinness

DL.

A guide to creating your first ontology. Biomed Inf Res 2001; 2: 14.

86.

Zhao

Zhang

Zeng

, et al. Construction of an aspect-level sentiment analysis model for online medical reviews. Inf Process Manag 2023; 60(6): 103513.

87.

Lample

Ballesteros

Subramanian

, et al. Neural architectures for named entity recognition. In: Proceedings of the 2016 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, San Diego, CA, June 2016, pp. 260–270. New York: Association for Computational Linguistics.

88.

Zhang

Huang

, et al. Named entity recognition in Chinese electronic medical record using attention mechanism. In: 2019 international conference on Internet of Things (Ithings) and IEEE Green Computing and Communications (Greencom) and IEEE Cyber, Physical and Social Computing (Cpscom) and IEEE Smart Data (Smartdata), Atlanta, GA, 14–17 July 2019, pp. 649–654. New York: IEEE.

89.

Dai

Wang

, et al. Named entity recognition using BERT BiLSTM CRF for Chinese electronic health records. In: 2019 12th international congress on image and signal processing, biomedical engineering and informatics (CISP-BMEI), Suzhou, China, 19–21 October 2019, pp. 1–5. New York: IEEE.

90.

Devlin

Chang

Lee

, et al. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, Minneapolis, MN, June 2019, vol 1, pp. 4171–4186. New York: Association for Computational Linguistics.

91.

Sun

Wang

, et al. ERNIE2.0: a continual pre-training framework for language understanding. Proc AAAI Conf Artif Intell 2019; 34(5): 8968–8975.

92.

Cui

Che

Liu

, et al. Pre-training with whole word masking for Chinese BERT. IEEE/ACM Trans Audio Speech Lang Process 2021; 29: 3504–3514.

93.

Liu

Ott

Goyal

, et al. RoBERTa: a robustly optimized BERT pretraining approach. arXiv 2019, https://arxiv.org/pdf/1907.11692