Word embedding attention and balanced cross entropy technique for sentiment analysis

Abstract

Customer feedback is useful for product development and increases the sales of the product. Reviews on e-commerce websites provided by the user provide valuable information about the product. Sentiment analysis on the text review helps to analyze the sentiment of users about the product and predict the sales of a product. The existing techniques in sentiment analysis use Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) models that have limitations of vanishing gradient problem and overfitting problem. Initially, the Amazon review dataset is collected and processed in word embedding stage. The CNN is utilized to extract the features from the input dataset for sentiment analysis. The Word Embedding Attention (WEA) technique provides higher weight to the words having strong relation with class. The CNN feature helps to provide higher performance for a smaller number of training data. This technique helps to increase the performance of the model related class-wise, thus increasing the precision and recall value. Finally, WEA technique in Bi-directional LSTM (BiLSTM) is used to increase the classification performance. The Balanced Cross-Entropy is proposed to maintain the gradient and solves the vanishing gradient problem in the network. The WEA-BiLSTM model has 97.4% accuracy, and 86.8% precision, and the existing CNN model has 97.1% accuracy and 85.4% precision in sentiment analysis. In this study, WEA-LSTM is used for the sentimental analysis of user reviews. This technique solves the vanishing gradient problem in the network by using Balanced Cross Entropy and helps to increase the performance of the model.

Keywords

Balanced cross-entropy bi-directional long short-term memory convolutional neural network sentiment analysis word embedding attention

1. Introduction

Consumer feedback is useful information in business to assess the quality and improve the product for the benefit of customers that provides an idea of expectation of new products. Various deep learning techniques were applied to accurately predict the customer’s opinion on mobile phone reviews of Amazon products [1]. On e-commerce websites such as Flipkart, eBay, and Amazon, thousands of users leave reviews about the products and services provided by the websites such as Yelp, Rotten Tomatoes, and Trip Advisor. Some users leave reviews about the product or services on social media. Therefore, customer reviews and feedback shared on the common platform about services or products influence new customers’ perspectives towards institutions, organizations, services, and products [2]. In natural language processing, one of the basic tasks is to learn low-dimensional word vector representation from a large dataset. The existing word embedding techniques learns word vectors from context semantic information and grammar while ignoring words sentiment information. Some techniques of sentiment information in reviews don’t consider certain words in various domains [3]. Sentiment analysis exploits the grouping of text mining, computational linguistics, and natural language processing to evaluate, calibrate, derive and analyze textual data in terms of documents, phrases, sentences, etc. Recently, natural language processing gained a huge consideration due to its efficiency in extracting useful information [4]. Sentiment analysis aims to classify the given text into three categories of user emotions: Positive, Neutral, and Negative [5].

Sentiment analysis is applied to input text to determine the sentiment polarity such as positive, neutral, and negative. Recently, online merchants and retailers ask their customers to share their opinion about the product. Consequently, more opinions are created every day and this is hard to process the data to extract useful information about the customer review [6]. Recently, deep learning technologies were applied to sentiment analysis to provide more promising results. Various neural network-based architectures such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) were applied to improve the efficiency of sentiment analysis [7]. Deep learning models of Convolutional Neural Network (CNN) and LSTM provide higher efficiency in sentiment analysis [8, 9, 10]. The objectives and contribution of this study are given below:

1)
The word embedding technique is proposed to provide higher weight value to the words having strong relation to the classes. This helps to increase the learning rate of the model and increases the class-wise performance of the model.
2)
The Balanced Cross Entropy is applied to maintain the gradient in the network to reduce the vanishing gradient problem. The gradient is maintained in the network based on the loss calculation.
3)
The CNN-based feature extraction technique is applied to generate more features in the convolutional layer and helps the model to provide higher performance for less training data.
4)
The Word Embedding Attention-Bi-directional Long Short-Term Memory (WEA-BiLSTM) model has higher performance in sentiment analysis classification compared to existing techniques. The WEA-BiLSTM model solves the vanishing gradient problem and decreases the overfitting problem in the classification.

This research study is structured as follows: Sentiment analysis researches are reviewed in Section 2 and the Word Embedding Attention (WEA) and Balanced Cross Entropy explanation are given in Section 3, the results of WEA-BiLSTM model are specified in Section 4. Lastly, Section 5 provides the conclusion.
2. Literature review

Sentiment analysis helps to find the response of the user related to the product and this technique can be used for sales prediction or to improve the product quality. Sentiment analysis is a challenging process in large e-commerce review data due to the presence of unstructured data. Recent research related to sentiment analysis was reviewed in this Section.

2.1 Feature extraction methods

Banbhrani et al. [11] applied the sentiment analysis technique for the prediction of review ratings based on feature extraction techniques. The model provides significant features such as number of sentences, emoticons, hashtags, elongated words, punctuation marks, numerical words, number of capitalized words, Term Frequency Inverse Document Frequency (TF-IDF), and SentiWordNet-based statistical features. Sentiment classification was performed using random multimodal deep learning and extracted features. The developed technique was a supervised technique and deep learning technique has an overfitting problem.

Wei and Song [12] construct a classification model based on text sentiment and a multi-input matrix was used to combine sentiment features that provide input of multi-channel CNN to extract sentiment features for text sentiment classification. The model also developed an image sentiment classification model using face images and global image merging. The supervision module on CNN with weighted loss was applied to extract features of facial sentiment, whole image sentiment was fused with facial target sentiment to determine image sentiment polarity. The text and image sentiment output were fused for the decision fusion method for the classification technique. The fusion of features creates an overfitting technique and irrelevant features affect the classification performance.

Shobana and Murali [13] applied the skip-gram technique for semantic and contextual feature extraction of words. LSTM model was applied for learning the complex textual pattern and the Adaptive Particle Swarm Optimization (APSO) model was applied for weight parameters. The Adaptive Particle Swarm Optimization model has a local optima trap and lower efficiency in the classification.

2.2 Feature selection methods

Zhang et al. [14] applied the attention model of interactive attributes to consider relevant attributes and interactive relationships to enhance performance for reviews. Multiple interactive attribute encoder deliberates all the attributes to increase the sentiment classification performance. After the local text encoder, multiple interactive attribute encoders were exploited to extract hidden data for text representations of aligning attribute features with self-attention of bilinear interaction. The multi-loss objective function was applied to improve the performance and tested on three datasets such as Amazon, Yelp and IMDB. The developed technique was a supervised model and has a vanishing gradient problem. Gokalp et al. [15] applied Iterated Greedy (IG) technique for feature selection in sentiment classification. The Iterated Greedy model was tested on the Amazon product dataset and displays significant results. The developed technique has limitations of imbalance data problems and overfitting.

2.3 Classification methods

Chen et al. [16] applied sentiment classification techniques to analyze user’s review habits to increase the performance of Hierarchical Neural Networks. Based on users, training sets are partitioned in the model and each user review are aggregated that is user’s historical reviews. The LSTM-based Hierarchical Neural Network was applied to train target and historical reviews for document representations. The similarities between multiple historical reviews and document representation of target review. The LSTM model has a vanishing gradient problem and the Hierarchical Neural Network model has an overfitting problem in classification.

Zhao et al. [17] applied an attention sharing mechanism and parameter transferring technique for cross-domain sentiment classification that consists of Target Domain Network and Source Domain Network. The pre-training language model was applied as training data in Hierarchical Attention Network such as global vectors for Bidirectional Encoder Representations from Transformers (BERT) and global vectors for word representation. To fine-tune the words, the parameter transferring technique in word and sentence levels was applied from Target Domain Network to Source Domain Network. The system with overfitting degrades the model’s performance.

Sazzed and Jayarathna [18] applied a hybrid technique of Self-supervised Sentiment Analyzer that uses lexicon-based technique and machine learning technique for sentiment classification from unlabelled data. The Lexical Rule-based Sentiment Analyzer was applied to predict review semantic orientation for prediction confidence score. The confidence score helps to provide highly accurate pseudo labels for Self-supervised Sentiment Analyzer and this consists of machine learning techniques to improve classification performance for complex reviews and less polarized. The Self-supervised Sentiment Analyzer and Lexical Rule-based Sentiment Analyzer performance was compared with existing unsupervised and achieved higher performance.

Rezapour [19] applied three machine learning classifiers such as Decision Tree, Support Vector Machine (SVM) and Naïve Bayes for sentiment analysis. Sparse word removal was applied for data in the removal of almost sentiments of negative reviews due to positive reviews being applied for reviews majority. Training the model individually for positive and negative reviews helps to increase learning capacity. Maintaining stop words, applications of various N-grams, separate tokenization process, and review title combination with their contents were applied to improve model performance. The learning capacity of the model was less and the feature selection process was not effective.

Kumar et al. [20] applied a hybrid technique of a three-step semi-supervised model that jointly learns aspects and sentiment from review sentences. Each aspect of seed words in a small set was considered and constructed sentiment class for respective semantics in vocabularies of coherent class. The Part-of-Speech (POS) tags were used to construct vocabularies to label sentences subset from training datasets. The semi-automatic technique was applied to label the data and this induced noise in the label during annotation. The overfitting problem in the joint learning method degrades the performance of classification.

Geetha and Renuka [21] performed sentiment analysis on consumer review data to classify positive and negative feelings. Various models such as SVM, LSTM and Naïve Bayes were used for review classification. The deep learning technique of the BERT Base Uncased model was used to solve the problem of sentiment analysis. An improved performance was provided by the BERT model with high accuracy and good prediction than the existing technique. The training of the BERT model generates more weights that create overfitting in classification.

Dadhich and Thankachan [22] perform automatic identification of sentiment analysis using K-Nearest Neighbor, Random Forest, SentiWordNet, Logistic Regression, and Naïve Bayes on English text from products of Flipkart and Amazon. Five key parameters of methods and existing sentiment analysis were presented in this study. The Product Comment Summarizer and Analyzer system was presented in this paper. A generic and automatic comment analyzer was used to find sentiment polarity and provide effective comments. This summarizes and classifies the comments as neutral, negative and positive very effectively. Rakshit et al. [23] applied analytics-based statistical techniques on primary data from samples of millennial samples to effective communication in selling e-marketers on individual human connections. Twitter platform of Twitter data was used for sentiment analysis to analyze the performance of Amazon and Flipkart during Big Billion Day Sales 2019 in India. The Naïve Bayes applied in the model consider the factors are independent and lower efficiency in learning feature importance.

Priyadarshini and Cotton [24] applied LSTM-CNN with grid search of Deep Neural Network. Baseline models including CNN-LSTM, LSTM-CNN, Neural Networks, K-Nearest Neighbors, and CNN were considered in this research. The developed model displays significant improvement, but the overfitting issue disturbs the effectiveness of developed model. Zhang et al. [25] applied a hybrid model for the amazon review dataset and neural networks were applied for the collaborative filtering process. This model applies embedding techniques to perform the model for a smaller number of training data. This model suffers from the limitation of overfitting problems to degrade the performance.

3. Proposed Word Embedding Attention-Bi-directional Long Short-Term Memory (WEA-BiLSTM) model

Here, in this research, the input Amazon Review dataset was used and given to CNN model for feature extraction. The attention layer provides weight value to the features and is applied to the BiLSTM model for sentiment classification. The Balanced Cross-Entropy is applied to maintain gradient in the network. The flow of the WEA-BiLSTM model for sentiment classification is illustrated in Fig. 1.

Figure 1.

The flow of Word Embedding Attention-Bi-Directional Long Short-Term Memory (WEA-BiLSTM) model for sentiment classification.

3.1 Dataset description

This is a sizable crawl of Amazon merchandise reviews. From about 20 million users, this dataset includes 82.83 million unique evaluations. Additionally, it contains 9.35 million pieces, covering the period from May 1996 to July 2014. It contains the meta data: , category, helpfulness votes, item-to-item relationships, price, product image, reviews and ratings, sales Rank, timestamps.

3.2 Word embedding

Word embeddings are calculated in various ways. The placement of each word takes place in a three-dimensional area. The data scientist decides how many variables to use in this area. Also experiment with various dimensions to see which one yields the finest outcome. Dense vectors with a lot less dimension are word embeddings. In addition, the vectors’ direction and distance represent the semantic connections between words. The embedding technique tends to give rules to words that are only occasionally encountered in training because it has a larger vocabulary.

Existing pre-trained word embeddings never succeed in sentiment analysis tasks because some sentiment words have similar syntactic and semantic characteristics in the corpus. This study suggests a word embedding technique to enhance the sentence-level emotion classification’s efficiency. The mapping between word embeddings and the corresponding sentiment orientations is fully utilized by this technique of sentiment enhancement. Words are first transformed into word embeddings using this technique, and then emotion mapping vectors are applied to each word embedding. Then, word embeddings and the emotion mapping vector that go with them are combined to create sentiment. The predicted sentiment orientations are obtained after reducing the dimensions of sentients through a completely connected layer. Then this output is processed for feature extraction stage where CNN is employed.

3.3 Convolutional Neural Network (CNN)

CNN is excellent at extracting features from images and has been demonstrated to be highly effective at identifying patterns that are challenging to identify using conventional techniques. CNN employs a feature extractor in the training process. The feature extractor used by CNN is made up of unique neural network classes, the weights of which are determined during training. Multiple convolution layers are followed by max-pooling and an activation function in the feature extraction process [26]. A crucial part of the CNN architecture is the convolution layer, which carries out feature extraction. Feature extraction usually entails combining linear and nonlinear operations, such as the convolution operation and the activation function [27]. Humans find the method for comprehending images fascinating, and it is a realistic option for individuals. However, there are more unnoticed difficulties with the machine’s ability to comprehend an image. The CNN is a deep learning that aims to mimic the visual technology of an individual. It is stimulated by the visual cortex of the brain. In image processing, which includes detection, localization, segmentation, and classification, among other tasks, CNNs represent a significant advancement.

Figure 2.

Architecture of Convolutional Neural Network (CNN) for feature extraction.

The major factor driving the model’s widespread use is the great CNN effectiveness in classification. Implementing the convolutional layer of learnable weights and biases similar to real neurons is part of the CNN. As demonstrated in Fig. 2, the fundamental components of CNN are convolutional layers, activation functions, and fully linked layers. This article provides a succinct explanation of the CNN, as well as the report includes a thorough description of CNN [28].

Convolutional Layer: The visual system of brain contains neuronal cells that are involved in retrieving image features [29]. Each neural cell extracts different features that aid in the comprehension of an image. Convolutional layers are used to simulate neuronal cells, which allows for the extraction of properties including gradient perception, texture, colors, and edges. Convolutional filters, also known as kernels, have a size of $n\times m\times d$ , where $d$ is the image depth, and they are learnable filters in convolutional layers. Following the forward pass, the kernels are convolved throughout the input volume’s height and width, as well as the dot product is calculated for filter entries and input. CNN learns which filters to use for texture, color, edge, etc. The output of the convolution layer is used as a layer for the activation function [30].

Activation Function: While non-linearity predominates in actual statistics, non-linear data transformation is applied. This guarantees that specified representations of the input space for diverse output spaces according to the necessities.

This requires real-value number $x$ and converting into 0 and 1. Input with large negative and positive are positioned near 0 and unity, individually which is stated in Eq. (1).

$\displaystyle f(x)=\frac{1}{1+e^{-x}}$ (1)

In a non-linear function, the real value number $x$ is taken into account and, if $x$ is negative, converts to 0. As demonstrated in Eq. (2), the ReLU activation function is a frequently utilized non-linear function that is faster than those with the sigmoid and $\tanh$ functions and takes less computing time.

$\displaystyle f(x)=\max({0,x})$ (2)

Pooling: It does a non-linear down-sampling with a combined feature. As a result, dimensional reduction data processing requires less computational resources. To lower the spatial size, data are aggregated based on feature type or space; rotational variance of images outperforms translation and regulates overfitting. A rectangle patch set is created when inputs are separated to use a pooling method. Depending on pooling procedure, a single value is computed to replace every patch. Maximum and average pooling are the two types of pooling that are widely frequently used [31].

Fully Connected Layer: Inputs are interconnected to every node in the subsequent layer and weight values are assigned to every node, similar to a neural network [32]. The model’s final result is the sum of the inputs times the respective weights. To carry out the classification task, a fully connected layer is coupled to the sigmoid activation function. In Fig. 3, CNN’s completely linked layer is displayed.

Figure 3.

Convolutional Neural Network (CNN) with fully connected layer.

3.4 Attention layer

Three region CNN networks are applied in a weighted word representation layer to use knowledge of the text. The word representation is given as input and applied to a convolutional layer built on every tweet. $F_{n}$ weighted matrix is used to calculate convolutional word vector matrix that is stated as $w\in R^{t\times m}$ regularly to measure local and inherent features and a word vector $t$ is designated in $F_{n}$ matrix, as in Eq. (3):

$\displaystyle h_{i}=f({V_{i:j+t-1}\times W[i]+b_{i}})$ (3)

where word vector $t$ generates a feature map of $h\in R_{n-t+1}$ , bias is $b$ , the matrix’s weight is denoted as $W$ , and the non-linearity activation function is denoted as $f$ . Feature maps are produced in convolutional layers, the max-pooling layer is applied to reduce dimension of dataset and extract import features, as displayed in Eq. (4).

$\displaystyle p_{i}=\textit{Max}[{h_{i}}]$ (4)

Here, max-pooling layer is applied in the feature map to denote $P_{i}\in R^{n-t+\frac{1}{2}}$ . Max pooling is utilized to various CNN layers filters to extract correspondingly vital features and not focus on polarity and semantic status. The attention layer is applied to provide the significance of every feature on CNN-generated features, as in Eq. (5).

$\displaystyle A_{i}=\frac{\exp({p_{i}})}{\mathop{\sum}\nolimits_{i}\exp({p_{i}% })}$ (5)

The max-pooling attention calculation is given in Eq. (5) above and generated attention score $A_{i}$ is applied for each feature context $p_{i}$ .

The attention score output is applied in BiLSTM to learn the feature context. The bi-LSTM model performs sequential maps to generate final features. The final feature context $p_{i}$ is used for the last feature map from CNN and attention scores are denoted as $A_{i}$ .

3.5 Bi-directional Long Short-Term Memory (BiLSTM) model

The LSTM model is widely used in sentiment prediction because it provides superior performance on data sequences [33]. The input $x_{k}$ at the current time, step $k$ and hidden layer output is denoted as $h_{k-1}$ as the previous time step of the hidden layer output. The LSTM cell state $c_{k}$ and LSTM unit network architecture is shown in Fig. 4. The cell state propagates the gradient and thus LSTM stores the relevant features for the long term than Recurrent Neural Network (RNN) model.

Figure 4.

The Long Short-Term Memory unit consists of input, output and forget gate in Bi-directional Long Short-Term Memory (BiLSTM).

Forget gate $f_{k}$ is the LSTM model first unit, and information is specified in the input gate to update. Temporarily save the $g_{k}$ new candidate value before the update a new cell state value. Input is received as $x_{k}$ and $h_{k-1}$ , the output is calculated with biases, weight parameter and ReLU activation function at each stage. The ReLU activation function overcomes vanishing gradient problem and gates are measured using Eqs (6)–(8):

$\displaystyle f_{k}=\sigma({W_{x}^{f}x_{k}+W_{h}^{f}h_{k-1}+b^{f}})$ (6) $\displaystyle i_{k}=\sigma({W_{x}^{i}x_{k}+W_{h}^{i}h_{k-1}+b^{i}})$ (7) $\displaystyle g_{k}=\textit{ReLU}({W_{x}^{g}x_{k}+W_{h}^{g}h_{k-1}+b^{g}})$ (8)

Element-wise product of cell state is applied with forget gate $f_{k}$ at previous time step $c_{k-1}$ , and new candidate value $g_{k}$ is applied with the element-wise product for the input gate $i_{k}$ . Two determined sums is used for cell state $c_{k}$ , as in Eq. (9).

$\displaystyle c_{k}=f_{k}\odot c_{k-1}+i_{k}\odot g_{k}$ (9)

The output gate $o_{k}$ is the last unit. The output information is determined using this gate. The output gate is calculated similarly to the previous steps, as in Eq. (10).

$\displaystyle o_{k}=\sigma({W_{x}^{o}x_{k}+W_{h}^{o}h_{k-1}+b^{o}})$ (10)

Equation (11) is used to compute the hidden state $h_{t}$ .

$\displaystyle h_{k}=o_{k}\odot\textit{ReLU}({c_{k}})$ (11)

The prediction is performed using LSTM model, this model uses input as historic capacity data $\{{C_{k}}\}$ , the size of input and output are the same. The same size of input and output provides input data type and identical output. The LSTM prediction is based on output and input formats.

Two LSTMs layers is applied to learn the sequence of each token-on-token past and future context. The LSTM model manages the order from left to right and another one from vice versa, as in Fig. 5. A hidden forward layer and $\overrightarrow{h}$ hidden unit function at each time step $t$ is calculated in terms of current input step $x_{t}$ and previous hidden state $\overrightarrow{h}_{t-1}$ . Current step input $x_{t}$ and the future hidden state $\overleftarrow{h}_{t+1}$ are used to compute hidden unit function $\overleftarrow{h}$ in a hidden backward layer. The context representations of forward and backward are used to generate $\overrightarrow{h}_{t}$ and $\overleftarrow{h}_{t}$ , correspondingly that concatenate with a long vector. The target signals of predictions are based on combined outputs.

The bi-directional network is transformed into high-level sentiment representation in fully connected dense layer to predict text sentiment polarity. Equation (12) provides the output.

$\displaystyle h_{i}=\textit{Relu}({w_{i}h_{p}+b_{i}})$ (12)

where BiLSTM network [34, 35] provides feature map $h_{p}$ , the obtained features are $h_{i}$ , and training learned parameters are $w_{i}$ , and $b_{i}$ . The merge feature layer applies sentiment classification performed by the output layer, as shown in Fig. 5. Binary and multi-class datasets are applied with sigmoid and sigmoid classifiers, respectively. Balanced Cross Entropy is utilized to perform difference among actual and predicted text.

Figure 5.

Attention layer in Bi-directional Long Short-Term Memory (BiLSTM) model for sentiment classification.

3.5.1 Balanced cross-entropy

The deep neural network baseline is combined with two distinct losses of optimization and a small replay memory: the distillation loss and standard softmax cross-entropy.

Model is trained using total loss $L$ , to define weighted distillation loss $L_{d}$ and standard softmax cross-entropy loss $L_{c}$ , as in Eq. (13):

$\displaystyle L=\rho L_{d}+({1-\rho})L_{c}$ (13)

where $\rho$ is applied as $N_{t-1}/N_{t}$ with incremental step $t$ and $N_{t}$ as a total number of classes, to balance two losses’ importance. At the training stage, the number of new classes are evaluated with a total number of learning classes to update the significance of distillation loss.

At each incremental step $t$ , the new parameters $\theta_{t}$ are used for the first initialization and the previous step parameters $\theta_{t-1}$ that is used to maintain previously learned knowledge using distillation loss. The distillation loss $L_{d}$ for each training sample $({x,y})\in X_{t}\mathop{\cup}\nolimits X_{M}$ , as in Eqs (14) and (15):

$\displaystyle L_{d}(x)=\mathop{\sum}\limits_{k=1}^{N_{t-1}}-\hat{p}_{k}(x)\log% (p_{x}(x))T^{2},$ (14) $\displaystyle\hat{p}_{k}(x)=\frac{e^{\frac{\hat{z}_{k}(x)}{T}}}{\sum_{j=1}^{N_% {t-1}}e^{\frac{\hat{z}_{j}(x)}{T}}},p_{k}(x)=\frac{e^{\frac{z_{k}(x)}{T}}}{% \sum_{j=1}^{N_{t-1}}e^{\frac{z_{j}(x)}{T}}}$ (15)

where associated ground truth label is $y$ , the input data is $x$ , the current model $\theta_{t}$ output logits is $z(x)=[{z_{1}(x),\ldots,z_{N_{t}}(x)}]$ , and the previous incremental step $\theta_{t-1}$ output logits of model is $z(x)=[\hat{z}_{1}(x),\ldots,\hat{z}_{N_{t}}-1(x)]$ .

4. Results

The Word Embedding Attention-Bi-directional Long Short-Term Memory (WEA-BiLSTM) model is applied on the Amazon dataset for the sentiment analysis and compared with other methods. The performance and error metrics were measured on the results of WEA-BiLSTM model for comparison.

Table 1
Performance of WEA-BiLSTM model for various iterations

Iteration	Accuracy (%)	Precision (%)	Recall (%)	F-measure (%)
0	0	0	0	0
50	91.7	81.4	72.3	76
100	92.5	82.1	73.4	76.5
150	92.8	82.3	73.6	77.2
200	93.4	82.5	73.8	77.8
250	93.7	83.3	74.1	78.1
300	94	84	75.2	78.3
350	94.9	85.7	76	78.7
400	95.6	86	76.1	80.4
450	97	86.8	76.3	80.4
500	97.4	86.8	76.7	81.2

The WEA-BiLSTM model is applied for sentiment analysis and its performance is measured for various iterations, as shown in Table 1. This shows that the model has higher accuracy due to its capacity to detect sentiment. The attention technique uses word embedding for the convolution process which helps to provide a better feature representation. The recall value of the model is low due to the presence of text information related to sarcasm and distinguishing features. In 400th iteration, the WEA-BiLSTM model accuracy is 95.6%, precision is 86%, recall is 76%, and F-Measure is 78.7%.

Figure 6.

Accuracy and F-measure of WEA-BiLSTM model for various iterations.

Figure 7.

Precision and recall of WEA-BiLSTM model for various iterations.

The WEA-BiLSTM model accuracy and F-measure are measured in sentiment analysis for various iterations, as shown in Fig. 6. This shows that the WEA-BiLSTM model has higher accuracy and considerably lower performance than F-Measure. The WEA-BiLSTM model has lower performance in the negative class and this reduces the F-measure of the model. The model has the lesser results in the negative class since the presence of sarcastic information from the user.

Figure 8.

The WEA-BiLSTM model classifier comparison on sentiment analysis.

The precision and recall of the WEA-BiLSTM model is measured for various iterations, as shown in Fig. 7. The WEA-BiLSTM model achieves higher precision and recall values within 50 iterations. The WEA technique increases the learning rate of the BiLSTM model based on weight values provided to input text. The WEA technique provides higher weight to the words having strong relation with class and this helps to enhance the model effectiveness.

Table 2

The WEA-BiLSTM classifier comparison on sentiment analysis

ANN: Artificial Neural Network
APSO: Adaptive Particle Swarm Optimization
CNN: Convolutional Neural Network
LSTM: Long Short-Term Memory
SVM: Support Vector Machine
Comparison	Accuracy (%)	Precision (%)	Recall (%)	F-measure (%)
SVM	91.8	81	73	75.6
ANN	81.9	71.9	72	76.8
LSTM	94.1	83	74.5	78.57
APSO-LSTM	96.8	85.28	76.08	80.04
CNN	97.1	85.6	76.33	80.64
Word Embedding CNN	97.4	86.8	76.7	81.2

Table 3

The WEA-BiLSTM model performance for various categories

Category	Methods	Accuracy (%)	AUC	Precision (%)	Recall (%)	F-measure (%)
Books	Iterated Greedy (IG)	88	0.93	88	88	88
	CNN	90	0.94	90	90	90
	Proposed CNN	93	0.96	92	92	92
DVD	Iterated Greedy (IG)	90.25	0.94	90	90	90
	CNN	91.4	0.95	90	90	90
	Proposed CNN	93.5	0.97	93	93	93
Electronics	Iterated Greedy (IG)	91.63	0.95	92	92	92
	CNN	92.3	0.96	93	93	93
	Proposed CNN	93.7	0.97	95	95	95
Kitchen	Iterated Greedy (IG)	93.09	0.96	93	93	93
	CNN	94.1	0.97	93	93	93
	Proposed CNN	95.2	0.98	94	94	94
AVG	Iterated Greedy (IG)	90.74	0.94	90.75	90.75	90.75
	CNN	91.95	0.95	91.5	91.5	91.5
	Proposed CNN	93.85	0.97	93.5	93.5	93.5

Figure 9.

WEA-BiLSTM precision and recall on sentiment analysis.

The performance of the WEA-BiLSTM model is evaluated with other classifiers, as displayed in Table 2 and Fig. 8. WEA with CNN can generate deep features and provide higher weight to the words having strong relation with class. The existing SVM model has an imbalance data problem, and the LSTM model has a vanishing gradient problem. The APSO-LSTM model has a local optima trap and misses some potential features for the classification. The CNN model has an overfitting issue since the formation of additional features in extraction process. The WEA-BiLSTM model has 97.4% accuracy, and 86.8% precision, and existing CNN contains 97.1% accuracy, and 85.4% precision in sentiment analysis.

The precision and recall value of the WEA-BiLSTM model is calculated and compared with existing classifiers on sentiment analysis, as shown in Fig. 9. The WEA-BiLSTM model uses word embedding to provide a higher weight value to the words having strong relation with dataset categories. This increases WEA-BiLSTM model performance related to the product category and increases the precision and recall value. The existing CNN model has lower efficiency since the formation of additional features in the convolutional layer. LSTM has a vanishing gradient problem and this degrades the model performance.

The WEA-BiLSTM model performance is evaluated on the parameters of accuracy and F-measure for several categories in the dataset, as exposed in Table 3 and Fig. 10. The WEA-BiLSTM contains higher performance in every category in the dataset. The WEA-BiLSTM model applies the word embedding technique to provide higher weight value to words having strong relation with classes. The CNN extracted features and weight value increases the learning rate of the LSTM model in classification. The balanced cross-entropy maintains the gradient values in the model that helps to overcome the vanishing gradient problem. The WEA-BiLSTM model has 93.85% accuracy, 93.5% precision, and CNN model has 91.95% accuracy, and 91.5% precision.

The precision and recall value of WEA-BiLSTM model for various categories in the dataset is shown in Fig. 11. The WEA-BiLSTM model has higher precision and recall value than existing techniques in sentiment analysis. The WEA-BiLSTM model applies a higher weight value for the words having strong relation with classes. This helps to increase the performance of the model class-wise, thus increasing the precision and recall value.

Table 4

WEA-BiLSTM model Mean Absolute Error (MAE) on sentiment analysis

A ${}^{3}$ NCF: aspect attention-based neural collaborative filtering
ItemKNN: item-based k-nearest neighbors’ algorithm
SlopeOne: simplest form of non-trivial item-based collaborative filtering
SVD: singular value decomposition for matrix factorization
UTER: unified text/paragraph embeddings of user reviews and item descriptions [25]
Datasets	SlopeOne	ItemKNN	SVD	A3NCF	UTER	CNN	Proposed CNN
Baby	0.8557	0.8536	0.8123	0.8075	0.7966	0.7854	0.7704
Sports and Outdoor	0.663	0.6771	0.6374	0.633	0.6169	0.6087	0.5843
Pet Supplies	0.8917	0.9202	0.88	0.8746	0.855	0.8542	0.8412
Video Games	0.8251	0.8482	0.7935	0.7916	0.7907	0.7891	0.77

Figure 10.

The WEA-BiLSTM model accuracy and F-measure for several categories.

The Mean Absolute Error (MAE) value of WEA-BiLSTM model for various categories on the Amazon dataset is calculated and evaluated by conventional methods, as displayed in Table 4 and Fig. 12. WEA-BiLSTM contains higher performance than existing method. The WEA technique provides a higher weight value to the words having strong relation with classes. The balanced cross-entropy maintains the gradient value in the network and avoids the vanishing gradient problem.

Table 5

WEA-BiLSTM Root Mean Square Error (RMSE) on sentiment analysis

Datasets	SlopeOne	ItemKNN	SVD	A3NCF	UTER	CNN	Proposed CNN
Baby	1.2012	1.1854	1.0704	1.0324	1.0146	0.9987	0.9862
Sports and Outdoor	1	0.9959	0.9065	0.8548	0.8501	0.8421	0.8369
Pet Supplies	1.2454	1.2658	1.1448	1.1058	1.0627	0.9965	0.9921
Video Games	1.1421	1.1634	1.0511	1.0269	1.0164	0.9988	0.9908

Figure 11.

The WEA-BiLSTM model precision and recall for various categories.

Figure 12.

Mean Absolute Error (MAE) of WEA-BiLSTM model on sentiment analysis.

Table 6

WEA-BiLSTM model for sentiment analysis

Algorithm	User cold start		Item cold start		User-item cold start
	MAE	RMSE	MAE	RMSE	MAE	RMSE
CNN	0.8108	1.1984	0.8065	1.1982	0.8057	1.2008
UTER	0.8123	1.2036	0.8106	1.2033	0.8112	1.2089
A3NCF	0.8274	1.2137	0.8313	1.221	0.8336	1.2227
SVD	0.8409	1.2235	0.8446	1.2252	0.848	1.2276
ItemKNN	0.8653	1.251	0.8714	1.2501	0.8752	1.2532
SlopeOne	0.8726	1.2626	0.8805	1.2639	0.8876	1.2683
Word Embedding CNN	0.8044	1.1920	0.7996	1.1905	0.8003	1.1941

Figure 13.

Root Mean Square Error (RMSE) of WEA-BiLSTM model on sentiment analysis.

Figure 14.

The WEA-BiLSTM model for user cold start.

Figure 15.

The WEA-BiLSTM model for item cold start.

Figure 16.

WEA-BiLSTM model for user Item cold start.

The WEA-BiLSTM model measures Root Mean Square Error (RMSE) value for various categories on the dataset, as displayed in Table 5 and Fig. 13. The WEA-BiLSTM shows better class-wise output due to weight value which is provided related to product category. The conventional CNN consist of overfitting problems due to generation of more features in convolutional layers.

The WEA-BiLSTM model is evaluated with Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) values for various cold start problems, as shown in Table 6. The user cold start has less information about the user information and the item cold start has less information about products. The WEA-BiLSTM model uses the CNN based feature extraction that helps to extract deep features and provide higher weight values to the words having strong relation.

The WEA-BiLSTM model performance is measured with Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) for various user, as shown in Figs 14–16, respectively. The WEA-BiLSTM achieved supreme results in classification over conventional models on sentiment analysis. The WEA-BiLSTM model has higher efficiency for a smaller number of training data due to the model uses CNN feature extraction for classification.

5. Conclusion

Sentiment analysis on review provides the sentiment of the user about the product that is useful for product development. The existing LSTM and CNN based models were applied in sentiment analysis that has limitations of vanishing gradient problem and overfitting problem in classification. This study proposed the WEA technique to provide higher weight values to the words having strong relation with classes. The CNN model is applied for feature extraction due to the generation of more features in the convolutional layer. The Balanced Cross Entropy is applied to maintain the gradient in the network to solve the vanishing gradient problem. The CNN feature extraction helps to provide higher performance for a smaller number of training data for classification. The future work comprises applying the feature selection method to select relevant features and achieve higher performance in imbalance data.

Funding

This research received no external funding.

Data availability

The datasets generated during and/or analysed during the current study are available in the [Amazon Review datasets] repository: [https://www.kaggle.com/datasets/bittlingmayer/amazonreviews].

Footnotes

Conflict of interest

The authors declare that they have no conflict of interest.

Author’s Bios

Vijaya Sagvekar is currently pursuing PhD (Computer Engineering) at PAHER University, Udaipur (Rajasthan) and also working as an Assistant Professor in Information technology department at Vasantdada Patil college of engineering and visual arts, Mumbai. She has teaching, research experience of 16 years. Her area of interest are Database management systems, Software engineering, Web Mining, Data Optimization, Network security.

Prashant Sharma working as an Associate Professor & HOD in Department of Computer Science and Engineering at Pacific (PAHER) University, Udaipur (Rajasthan), India. He has above 9 years teaching experience. He completed PhD (CSE) in 2018 and his research interest area is Artificial Intelligence, Machine Learning and Nature Inspired Algorithms.

References

Alharbi

N.M.

Alghamdi

N.S.

Alkhammash

E.H.

and Al Amri

J.F.

, Evaluation of sentiment analysis via word embedding and RNN variants for Amazon online reviews, Mathematical Problems in Engineering, 2021, 5536560. doi: 10.1155/2021/5536560.

Sivakumar

and Uyyala

S.R.

, Aspect-based sentiment analysis of mobile phone reviews using LSTM and fuzzy logic, International Journal of Data Science and Analytics 12(4) (2021), 355–367. doi: 10.1007/s41060-021-00277-x.

Liu

Zheng

and Lin

, Cross-domain sentiment aware word embeddings for review sentiment analysis, International Journal of Machine Learning and Cybernetics 12(2) (2021), 343–354. doi: 10.1007/s13042-020-01175-7.

Chatterjee

Zhou

Abusorrah

Sedraoui

and Alabdulwahab

, Statistics-based outlier detection and correction method for amazon customer reviews, Entropy 23(2) (2021), 1645. doi: 10.3390/e23121645.

Alsayat

, Improving sentiment analysis for social media applications using an ensemble deep learning language model, Arabian Journal for Science and Engineering 47(2) (2022), 2499–2511. doi: 10.1007/s13369-021-06227-w.

Birjali

Kasri

and Beni-Hssane

, A comprehensive survey on sentiment analysis: Approaches, challenges and trends, Knowledge-Based Systems 226 (2021), 107134. doi: 10.1016/j.knosys.2021.107134.

Basiri

M.E.

Nemati

Abdar

Cambria

and Acharya

U.R.

, ABCDM: An Attention-based Bidirectional CNN-RNN Deep Model for sentiment analysis, Future Generation Computer Systems 115 (2021), 279–294. doi: 10.1016.

Dang

C.N.

Moreno-García

M.N.

and De la Prieta

, Using Hybrid Deep Learning Models of Sentiment Analysis and Item Genres in Recommender Systems for Streaming Services, Electronics 10(20) (2021), 2459. doi: 10.3390/electronics10202459.

Agüero-Torales

M.M.

Abreu Salas

J.I.

and López-Herrera

A.G.

, Deep learning and multilingual sentiment analysis on social media data: An overview, Applied Soft Computing 107 (2021), 107373. doi: 10.1016/j.asoc.2021.107373.

10.

Singh

Jakhar

A.K.

and Pandey

, Sentiment analysis on the impact of coronavirus in social life using the BERT model, Social Network Analysis and Mining 11(1) (2021), 33. doi: 10.1007/s13278-021-00737-z.

11.

Banbhrani

S.K.

Lin

and Sajnani

D.K.

, Spider Taylor-ChOA: Optimized deep learning based sentiment classification for review rating prediction, Applied Sciences 12(7) (2022), 3211. doi: 10.3390/app12073211.

12.

Wei

and Song

, Sentiment classification of tourism reviews based on visual and textual multifeature fusion, Wireless Communications and Mobile Computing 2022 (2022), 1–10. doi: 10.1155/2022/9940817.

13.

Shobana

and Murali

, An efficient sentiment analysis methodology based on long short-term memory networks, Complex and Intelligent Systems 7(5) (2021), 2485–2501.

14.

Zhang

Wang

and Zhang

, Personalized sentiment classification of customer reviews via an interactive attributes attention model, Knowledge-Based Systems 226 (2021), 107135. doi: 10.1016/j.knosys.2021.107135.

15.

Gokalp

Tasci

and Ugur

, A novel wrapper feature selection algorithm based on iterated greedy metaheuristic for sentiment classification, Expert Systems with Applications 146 (2020), 113176. doi: 10.1016/j.eswa.2020.113176.

16.

Chen

Zhao

and Zhang

, User’s review habits enhanced hierarchical neural network for document-level sentiment classification, Neural Processing Letters 53(3) (2021), 2095–2111. doi: 10.1007/s11063-021-10423-y.

17.

Zhao

Wang

Liu

Yang

and Liu

, Cross-domain sentiment classification via parameter transferring and attention sharing mechanism, Information Sciences 578 (2021), 281–296. doi: 10.1016/j.ins.2021.07.001.

18.

Sazzed

and Jayarathna

, SSentiA: A Self-supervised Sentiment Analyzer for classification from unlabeled data, Machine Learning with Applications 4 (2021), 100026. doi: 10.1016/j.mlwa.2021.100026.

19.

Rezapour

, Sentiment classification of skewed shoppers’ reviews using machine learning techniques, examining the textual features, Engineering Reports 3(1) (2021), e12280. doi: 10.1002/eng2.12280.

20.

Kumar

Gupta

Balan

Neti

L.B.M.

and Malapati

, BERT based semi-supervised hybrid approach for aspect and sentiment classification, Neural Processing Letters 53(6) (2021), 4207–4224. doi: 10.1007/s11063-021-10596-6.

21.

Geetha

M.P.

and Karthika Renuka

, Improving the performance of aspect based sentiment analysis using fine-tuned Bert Base Uncased model, International Journal of Intelligent Networks 2 (2021), 64–69. doi: 10.1016/j.ijin.2021.06.005.

22.

Dadhich

and Thankachan

, Social and Juristic challenges of AI for Opinion Mining Approaches on Amazon and Flipkart Product Reviews Using Machine Learning Algorithms, SN Computer Science 2(3) (2021), 180. doi: 10.1007/s42979-021-00554-3.

23.

Rakshit

Srivastava

P.K.

Afjal

and Srivastava

S.K.

, Sentimental Analytics on Indian Big Billion Day of Flip Kart and Amazon, SN Computer Science 2(3) (2021), 204. doi: 10.1007/s42979-020-00441-3.

24.

Priyadarshini

and Cotton

, A novel LSTM-CNN-grid search-based deep neural network for sentiment analysis, The Journal of Supercomputing 77(12) (2021), 13911–13932. doi: 10.1007/s11227-021-03838-w.

25.

Zhang

Liu

and Sang

, Unifying paragraph embeddings and neural collaborative filtering for hybrid recommendation, Applied Soft Computing 106 (2021), 107345. doi: 10.1016/j.asoc.2021.107345.

26.

Tan

and Jiang

, Review on convolutional neural network (CNN) applied to plant leaf disease classification, Agriculture 11(8) (2021), 707. doi: 10.3390/agriculture11080707.

27.

Zhang

Y.-D.

Satapathy

S.C.

Guttery

D.S.

Górriz

J.M.

and Wang

S.-H.

, Improved breast cancer classification through combining graph convolutional network and convolutional neural network, Information Processing and Management 58(2) (2021), 102439. doi: 10.1016/j.ipm.2020.102439.

28.

Niu

Lin

and Zou

, sgRNACNN: Identifying sgRNA on-target activity in four crops using ensembles of convolutional neural networks, Plant Molecular Biology 105(4) (2021), 483–495. doi: 10.1007/s11103-020-01102-y.

29.

Song

Han

x.-y.

Montenegro-Marin

C.E.

and krishnamoorthy

, Secure prediction and assessment of sports injuries using deep learning based convolutional neural network, Journal of Ambient Intelligence and Humanized Computing 12(3) (2021), 3399–3410. doi: 10.1007/s12652-020-02560-4.

30.

Chen

Jiang

Zhang

W.-G.

and Chen

, A novel graph convolutional feature based convolutional neural network for stock trend prediction, Information Sciences 556 (2021), 67–94. doi: 10.1016/j.ins.2020.12.068.

31.

Nguyen

H.D.

Tran

K.P.

Thomassey

and Hamad

, Forecasting and Anomaly Detection approaches using LSTM and LSTM Autoencoder techniques with the applications in supply chain management, International Journal of Information Management 57 (2021), 102282. doi: 10.1016/j.ijinfomgt.2020.102282.

32.

Cao

and Li

, Financial time series forecasting model based on CEEMDAN and LSTM, Physica A: Statistical Mechanics and Its Applications 519 (2019), 127–139. doi: 10.1016/j.physa.2018.11.061.

33.

Park

Choi

W.J.

Ryu

H.-Y.

and Kim

, LSTM-based battery remaining useful life prediction with multi-channel charging profiles, IEEE Access 8 (2020), 20786–20798. doi: 10.1109/ACCESS.2020.2968939.

34.

Aslan

M.F.

Unlersen

M.F.

Sabanci

and Durdu

, CNN-based transfer learning-BiLSTM network: A novel approach for COVID-19 infection detection, Applied Soft Computing 98 (2021), 106912. doi: 10.1016/j.asoc.2020.106912.

35.

Challa

S.K.

Kumar

and Semwal

V.B.

, A multibranch CNN-BiLSTM model for human activity recognition using wearable sensor data, The Visual Computer, 2021. doi: 10.1007/s00371-021-02283-3.

Word embedding attention and balanced cross entropy technique for sentiment analysis

Abstract

Keywords

1. Introduction

2.1 Feature extraction methods

2.2 Feature selection methods

2.3 Classification methods

3. Proposed Word Embedding Attention-Bi-directional Long Short-Term Memory (WEA-BiLSTM) model

3.2 Word embedding

3.3 Convolutional Neural Network (CNN)

Table 1 Performance of WEA-BiLSTM model for various iterations

Funding

Data availability

Footnotes

Conflict of interest

Author’s Bios

References

Table 1
Performance of WEA-BiLSTM model for various iterations