Enhancing predictive modeling for Indian banking stock trends: A fusion of BERT and attention-based BiLSTM approach

2.1 Unsupervised feature-based approaches

The learning representation including neural and non-neural network for words have been an agile area of research for decades. The pre-trained embeddings are considered an integral part of Modern NLP as it has shown significant improvement over the word embeddings that were to be learned from scratch. The pre-training of word embeddings which is the main objective of using language models performs training from left to right and differentiates between incorrect words and context. Unsupervised feature-based approaches which have coarser granularities include generalizing into paragraph embeddings for sentence embeddings. The prior work as discussed in [7] included ranking candidates into previous and next sentence representation including generations from both the directions left and right or by using autoencoder which has a denoising effect for derived objectives. Thus, to make the sentence representation and embeddings clear Language model came into the picture.

ELMo i.e., Embeddings from Language Model consists of LSTM (Long Short-Term Memory) which is trained in both directions, and a language model generated which understands the sense of the word which occurs next as well as previous and is considered as a step ahead. In ELMo, generalization is performed on embeddings of traditional words from different dimensions. In this language model, the representation of each token is based on its context which means representations that are concatenated from right to left and left to right. The architectures which are task-specific and existing, integrate these word embeddings which are contextually based and use advanced ELMo, for many important benchmarks in NLP such as opinion mining, named entity recognition, and question answering. In this paper, LSTMs are used to perform learning and are utilized for contextual representations for predicting a single word in both contexts right and left for the selected task. ELMo is also one such language model, which uses feature-based LSTMs and they are not deep bi-directional models as shown in [9].

2.2 Unsupervised feature-based on fine-tuning

The word embeddings from the unlabeled text are the way of working for feature-based approaches for pre-training. Recently, pre-training of contextual tokens generated from documents or sentences from the text data which is unlabeled are fine-tuned further to a supervised downstream task. Learning requires fewer parameters if learning is performed from scratch, hence, such approaches were proved to be advantageous. This advantage gave much achievement to the OPENAI GPT (Generative Pre-trained Transformer) model as state-of-art which resulted from GLUE (General Language Understanding Evaluation) as a benchmark for various tasks on the sentence level. Auto encoder and left-to-right modeling are the objectives used by such models for pre-training as included in [10].

BERT model was introduced as a bi-directional language model which is pre-trained. This model addressed the constraint of unidirectionality by introducing MLM (Masked language Modelling) and NSP (Next Sentence Prediction) phases which improved the performance of the model. The BERT model presumes that the tokens which are masked are independent and can be predicted using unmasked tokens only. XLNet language model was introduced recently to improve the weakness of the BERT model by applying Transformer-XL which included masked positions with dependency. XLNet uses permutation language modeling as shown in [11].

2.3 Transfer learning from supervised data

The primary intention of Transfer Learning is to enable the developers and researchers to use this technique of pre-training the models which are developed by others and perform a few modifications on them. The main reason for increased accuracy is the use of the methods for Transfer Learning in NLP and mainly for Opinion Mining and Sentiment Analysis. If the training data available is limited, still Transfer Learning leverages the information extracted from a big corpus or dataset to increase efficiency. Opinion Mining summarization and machine translation are wide areas in which transfer learning manifests the effective working of supervised tasks using big datasets. This [12] paper, introduces a method of deep transfer learning for the domain of the stock market based on LSTM for prediction.

In this [13] paper, stock market movement prediction is done using the Selective Transfer Learning with Adversarial Training (STLAT) model which applies transfer learning. This model introduced two aspects namely a method for pre-training and fine-tuning for stock prediction and a module for data selection so that selection of relevant samples for training can be done correctly.

2.4 Trends prediction using only historical stock price information

Forecasting of financial data based on time series using Machine Learning methods is much more popularly studied. The techniques specifically SVM (Support Vector Machines) and (KNN) K-Nearest Neighbor are mostly used for stock domain index prediction. The combination of KNN and SVM was popularly used for the Chinese stock market as discussed in [14]. An optimal solution was introduced by [15] in which a neural network was used to develop the model in which PSO (Particle Swarm Optimization) was introduced which predicted the trends. Further to improve the performance, Generic Algorithm, probabilistic SVM, and AdaBoost were introduced by [16]. There are many more approaches proposed which resulted in a good performance.

The work proposed by I. Markovic et al. [17], developed a hybrid model in which the integration of weighted kernel least squares SVM and the analytical hierarchical process was done. In this L. Lei [18] paper, researchers found that the fusion model developed by fusing rough set and wavelet neural network performed better. The fusion model proposed by Jigar Patel et al. [19] significantly contributed to the prediction of stock market trends. This particular fusion model incorporated Random Forest (RF), Artificial Neural Network (ANN), and Support Vector Machine (SVM) techniques at various stages, focusing on CNX Nifty and S&P indices as part of its analysis. Y. Xu et al. [20] introduced a model that combined Logistic Regression (LR), Support Vector Machine (SVM), and Artificial Neural Network (ANN) to predict the trends of the next day. Similarly, Ying Xu et al. [21] proposed a novel fusion model for Chinese stocks, integrating two prediction models based on K-means clustering and ensemble learning. This fusion approach significantly improved the prediction accuracy by leveraging the strengths of both techniques.

In addition to the previously mentioned techniques, Machine Learning (ML) and Deep Learning (DL) are also extensively employed and have demonstrated superior performance. Deep learning, a subset of Machine Learning, plays a crucial role in extracting high-level features from the represented data. Hence, as these deep learning methods have performed better in comparison with ML (Machine Learning) methods, their usage has been extensive in many applications like computing vision, stock market prediction, image processing, etc.

Many DL (Deep Learning) methods have been introduced based on analyzing financial data with time series. In this paper, [22] researchers developed a model in which historical time series financial data was combined with stock-related technical indicators to perform a prediction of the stock market price movement. In [23], a model is proposed in which a Restricted Boltzmann Machine (RBM) along with many classifiers, was combined for the prediction of the short-term stock market trend. Also, in [24], the model was developed by combining generalized autoregressive conditional heteroscedasticity (GARCH) and LSTM (Long Short-term Memory) to predict the volatility of stock price movement. A model was proposed by [25], which integrated FS (feature selection) and LSTM the for prediction of closing stock price in which four stock data the of Shenzhen Component index were taken and a feature set was constructed using 17 technical indices. It was concluded that the FS-LSTM improved the overall accuracy of the system and the forecast value of stock prices. The stock market prediction was done using three frameworks by [26] based on AM (Attention Mechanism) namely AT-RNN, AT-GRU, and AT-LSTM in which selection and focus on key information from stock-related data was performed. Recently, many methods based on CNN are proposed by researchers which achieve good performance in stock market prediction. For example, [27] proposed a method in which stock technical indicators were converted to 2D images, and a CNN-based novel method was applied for the prediction of stock price. CNN and LSTM were combined by [28] to a new network model in which CNN was adopted for extracting features and LSTM was applied to predict stock price movement from the extracted features. The results depicted that the CNN-LSTM method is much more reliable for stock forecasting and gave better prediction accuracy.

2.5 Trends prediction using both textual and numeric information

As the deep learning techniques got advanced, the focus has been given to the learning models based on a neural network that incorporates both financial textual data and quantitative stock data for stock market price prediction. In this paper, [29] recently proposed a model which included both textual and numerical information for the stock index in Thailand. For textual data, they made use of social media and the polarity associated with it. Finally, a multiple regression model is generated which utilizes both the information and another related variable. The [30] paper presented the concept of using two types of information textual and numeric for the prediction of stock market behavior. News headlines were used as textual data to which Word2Vec and CNN are applied, and for numeric data, they made use of technical indicators for the creation of historic price stock data to which LSTM is applied for learning. Finally, the results from both are combined for the prediction of the stock market movement. It analyzed the index prices and sentiments which depicted that sentiment signals which are incorporated in news are reliable and thus are considered accurate predictors of stock market prices. In this paper [31], the work aimed at combining technical analysis as well as fundamental analysis by applying machine learning techniques. For technical analysis indicators were applied while news sentiments were used for fundamental analysis and the trends were predicted for the NASDAQ100 index. In this [32] study the investor sentiment which was taken from Stocktwits and gold returns along with the S&P 500 index was investigated in which it was observed that volatility of the S&P 500 is influenced by gold returns and the sentiments affected returns of S&P 500. Also, [33] studied a combination of CNN and RNN to predict stock trends which included both financial news headlines and stock indicators and it showed that financial news headlines which are in textual format improved the accuracy as compared to the content in the news. Recently, a combination of three different models was proposed by [33] which included SVM, ANN, and an adaptive neuro-fuzzy inference system that performed prediction using public opinions and stock prices. In [34], an examination of signals of the stock market from different websites and blogs was performed, and it was found that the response of investors is much faster and strong toward the positive sentiment. In [35], an investigation of stock market price movement for four different countries using opinions in tweets was done and a better correlation was found between tweets and stock prices.

Currently, the models which incorporate numeric stock data and textual news data are considered largely to support sentiment polarity extraction and it plays an integral part in the prediction of stock trends. Mostly in previous studies, tweets from Twitter have been a big source of information for better performance of opinion mining. However, considering that the sentiment polarity and the news perceive reality which is usually considered as fuzzy and improves the prediction accuracy depicting that the opinions and the reviews cannot be considered as granted. LDA topic model was proposed by [33] in which keyword extraction from tweets was done and opinion mining was performed using the fusion method in which both opinion convergence and news events were incorporated which in turn extended numeric features, ultimately improving the prediction accuracy.

Hence, as per the methods surveyed and discussed above regarding the prediction of stock market trends, the hybrid method is beneficial because it combines both financial textual information along with the stock price for modeling. The increase in the number of data sources can help to achieve better accuracy for the prediction of stock trends with the help of cutting-edge algorithms which can be applied to multiple data sources to get meaningful results. The model in this paper uses the BERT language model which includes transfer learning as few modifications are only required which helps in increasing the accuracy when opinion mining of financial textual data is performed clubbed with Bi-LSTM which is applied for the actual financial numeric data.

This paper proposes a hybrid information mixing module, which incorporates BiLSTM for time-series analysis, to predict stock price movement. The structure of the hybrid information mixing module is illustrated in Fig. 1, comprising of a feature embedding, hybrid information mixing module, and binary classifier. The feature embedding includes price and text embedding. The price embedding utilizes BiLSTM to extract time-series characteristics from historical price data, allowing for effective analysis of stock market fluctuation signals. BiLSTM, a variant of RNN, is well-suited for analyzing time-series data by utilizing hidden states as memory to capture temporal information history and generate outputs based on network memory. [36]

In order to improve the accuracy of stock price movement prediction, we performed a comparative experiment using various architectures of BiLSTM. The BiLSTM algorithm was chosen due to its ability to effectively learn long-term dependencies and process time-series data. By applying BiLSTM to analyze historical price data, we aimed to capture the sequential dependencies during the trading day and predict stock price movement with greater precision.

To analyze the time-series characteristics of the stock data, the market variables of open, high, low, close (OHLC) prices, and trading volume on trading day i are concatenated into a vector p_i ∈ R5. The BiLSTM model is then applied to analyze the time-series features of pi. The output value of the hidden layer of the BiLSTM, represented as b_i ∈ R^dG, is extracted as the time-series feature for the stock data. Here, bi corresponds to the last hidden state of the BiLSTM for trading day i, and dG represents the hidden dimension of the BiLSTM.

Furthermore, p i O , p i H , p i L , p i C , p i V represent the open, high, low, close prices, and trading volume, respectively. The hidden state is denoted by h p i = [ p i O , p i H , p i L , p i C , p i V ](1) b i = BiLSTM ( p i , h i - 1 ) , t - T ≤ i ≤ t - 1(2)

The reviews containing semantic information are crucial for comprehending the stock context, while a large volume of stock trend information is extracted from both review data and quantitative analysis. Textual reviews provide factual information and reflect user sentiment towards stocks. It should be noted that daily reviews and historical prices have distinct impacts on stock price movement. [37]

All reviews [r₁, r₂, r₃,  …  …  …  …  r_N] uploaded data on trading day i for each firm captures the stock market trend. The contextual information was extracted using BERT [e₁, e₂, e₃,  …  …  …  …  e_N] for all review data on a trading day i, e_v ∈ R^(N ×d) is extracted. The semantic feature was obtained by averaging the contextual word embeddings extracted from all review data on a trading day i using BERT, divided by the number of review data N on that day s_v ∈ R^dB of the review data is generated for a specific trading day i. In addition, the last state of BERT is s_v which is hidden, N represents the number of review data available for each corporation on trading day i, and dB denotes the hidden dimension of BERT. e v = BERT s ( r v ) , v ∈ [ 1 , N ](3) s i = 1 N ( ∑ v = 1 N e v )(4)

The integration of price data and text data can significantly improve the accuracy of stock price prediction models, as observed by [38], as stocks are influenced by various market signals originating from different sources. However, unlike image data, text data can have variable dimensions, which can cause problems when using an MLP-based model [39]. In order to tackle this challenge, [40] proposed a BiLSTM-based model called BERT to extract semantic features from text data. These features were subsequently combined with time-series features obtained from the price data to form a mixed feature vector x_i ∈ R^F ×dF, which captures both types of information. The hybrid information mixing module then utilizes this mixed feature vector for further processing. x i = CONCAT ( b i , s i )(5) where b_i and s_i are the time-series and semantic features, respectively. The feature vector is composed of time-series and semantic features, representing the interaction between price and text data and enhancing the market signals influencing stock price movements. By combining these distinct characteristics, we can capture the multimodal interaction of information and effectively reflect market signals from both data types. This approach enables us to better capture the market signals specific to each data type and enhance our ability to predict stock price fluctuations.

Our study introduces a hybrid information mixing module comprising two MLP blocks, namely the feature-mixing MLP and interaction-mixing MLP. These MLP blocks are designed to learn the mixed feature, which is created by combining the time-series feature extracted from the price data and the semantic feature derived from the text data. Each MLP block consists of a fully connected layer, followed by a nonlinearity function such as GELU, and another fully connected layer. By leveraging this hybrid information mixing module, the mixed feature x_i is learned independently in both row-wise and column-wise directions, effectively generating the mixed features.

3.5 Feature mixing BERT-BiLSTM

The Feature-Mixing BERT-BiLSTM module is specifically designed to extract discriminative information from both the price and text data, which are then combined to form the mixed feature matrix. This matrix is structured such that each row represents the channel information of a particular feature, while each column represents an embedding vector. The module operates on all columns x_i of the matrix, with each token being inputted through the same module with shared weights across all columns. Through dense matrix multiplication, the weights are shared within the same channel for different tokens. By leveraging this sharing of time-series information and meaningful insights contained within the time-series feature and semantic feature, the module effectively mixes and learns channel information. This enables the Feature-Mixing BERT-BiLSTM to facilitate global communication between different spatial locations through matrix transposition.

The mixed feature x _i is then passed through the Feature-Mixing BERT-BiLSTM module to produce the output x Feature T ∈ R^dM ×F. Here, dM represents the hidden dimension of the transposed x _Feature ∈ R^F ×dM. We use skip connections and layer normalization for learning stabilization. The Interaction-Mixing MLP takes as input, where GELU denotes the GELU nonlinearity, and W₁ and W₂ ∈ R^F ×dF are the weights of the first and second fully connected layers, respectively. Finally, the output of the Interaction-Mixing MLP is obtained using Equation (6): x Feature = x i + W 2 ( GELU ( W 1 ( LayerNorm ( x i ) T ) ) ) T(6)

This process of channel information mixing is performed by the Feature-Mixing BERT-BiLSTM module, which facilitates communication between different tokens. By doing so, the module is able to effectively extract and learn discriminative information. These advancements in channel information mixing have been highlighted in the work by [41].

In contrast to the Feature-Mixing BERT and BiLSTM module, the Interaction-Mixing BERT and BiLSTM module operates on all rows of the matrix x_i. Its purpose is to transpose the matrix back to the shape of tokens×channels while sharing weights across all tokens. This module enables the communication between different channels by capturing the correlation between the embedding vectors of each feature. Essentially, the Interaction-Mixing BERT and BiLSTM module treats each token independently, taking individual rows of X as input, thereby facilitating communication across different channels. The module leverages dense matrix multiplication effectively, applied independently to each spatial location, thereby enhancing interaction and promoting information sharing between different channels within each spatial location.

The input x _Feature to the Interaction-Mixing BERT and BiLSTM module generates x _Interaction ∈ RF×dI. Moreover, dI denotes the hidden dimension of the Interaction-Mixing BERT and BiLSTM module, and W₃ and W₄ ∈ R^F ×dE are the weights of the third and fourth fully connected layers of the Interaction-Mixing BERT and BiLSTM module, respectively. In addition, dE is a hidden dimension in the Interaction-Mixing BERT and BiLSTM module: x Interaction = x Feature + W 4 ( GELU ( W 3 ( LayerNorm ( x Feature ) ) ) )(7)

By learning the correlation of two embedding vectors through the Interaction-Mixing BERT and BiLSTM module, the correlation between channels of x _Feature is mixed to x _Interaction .

We obtained the output x _Interaction by leveraging the hybrid information mixing module that incorporates both BERT and BiLSTM features. The output layer is then passed through a series of operations, including layer normalization, global average pooling, and a fully connected layer, as follows: x norm = LayerNorm ( x Interaction )(8) x GAP = GAP ( x norm )(9) x output = FC ( x GAP )(10)

Here, LayerNorm applies layer normalization to x _Interaction and x _norm generates as the output. GAP stands for global average pooling, which takes x _norm as input and generates x _GAP as its output. FC denotes the fully connected layer, which takes x _GAP as input and produces the final classification result, x _output .

In the stock price movement classification task using BERT and BiLSTM, the binary classifier aims to predict the direction of stock price fluctuations. The label is set by comparing the closing price p t c of the corresponding t date with the closing price p t - 1 c of the previous date t-1, where t ∈ {1, 2, 3,..., T}, and T denotes the number of trading days during the period [t – T, T – 1]. If the stock price is ‘up’ compared to the previous day’s closing price, the label is 1, otherwise, it is 0.

The output of the hybrid information mixing module undergoes layer normalization, followed by global average pooling, and finally, a fully connected layer to generate the final classification result. The output of layer normalization is denoted by x _norm , and the output of global average pooling is denoted by x _GAP . The final classification result is obtained by passing x _GAP through a fully connected layer and applying the sigmoid function. The binary cross-entropy loss, denoted as BCE loss, is employed to compute the discrepancy between the predicted value denoted as y ˆ and the actual label value denoted as y. The predicted value is y ˆ obtained by applying the sigmoid function to the output of the fully connected layer. The formulation of the binary cross-entropy loss function is defined by Equation 11. BCELoss ( y ˆ , y ) = - [ y · log ( y ˆ ) + ( 1 - y ) · log ( 1 - y ˆ ) ](11)

4.1 Dataset

In this investigation, we conducted a comprehensive analysis of the hybrid information mixing module by leveraging the Stocktwits dataset and stock price data from two prominent Indian companies, HDFC and SBI. The Stocktwits dataset encompasses a vast collection of social media messages specifically related to stock trading, while the stock price data was sourced from the National Stock Exchange of India.

To ensure data quality, we filtered out samples that were missing price data or social media messages, resulting in a refined dataset comprising 503 trading days spanning from January 1, 2014, to January 1, 2016, for HDFC and SBI. The price data incorporated historical OHLC prices (open, high, low, close) and trading volume.

For model training and evaluation, the dataset was split into training and validation sets in an 8 : 2 ratio. To capture temporal dependencies effectively, we employed a window size of 30, meaning that the binary classifier was tasked with predicting the stock price movement for the closing price of the next trading day based on the data from the previous 30 days.

4.2 Training

In our proposed model, we employ a BiLSTM layer with 768 hidden units to capture the time-series characteristics present in the historical price data. For the BERT layer utilized to extract semantic features from the text data, we set the hidden layer dimension to 768 as well.

Within the hybrid information mixing module, we incorporate an MLP with a hidden dimension of 768 and 8 layers. This MLP facilitates the fusion and interaction of the extracted features from both the time-series and semantic domains.

During training, the model undergoes 30 epochs using the AdamW optimization method with a learning rate of 2e-5 and an epsilon value of 1e-8. We utilize a batch size of 32 for efficient computation. All experiments are conducted using one NVIDIA GeForce RTX 3090 GPU.

4.3 Evaluation measures

For evaluating the accuracy of our stock price movement prediction, we rely on the confusion matrix-based metric. This metric allows us to assess the performance of our model by considering four key elements: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). From these elements, we calculate the accuracy, which measures the proportion of correct predictions in relation to the total number of predictions. Accuracy = ( TP + TN ) ( TP + TN + FP + FN )(12)

5.1 Stock price fluctuations

A comparative experiment was conducted to evaluate the performance of the proposed hybrid information mixing module, which combines BERT and BiLSTM, in predicting stock price movement. The performance of the module was compared with existing state-of-the-art models in the field. The results, as shown in Table 1, indicate that the hybrid information mixing module achieved a performance of 68.98%, surpassing all other models, including those that solely rely on either time series or text features. This outcome demonstrates the module’s ability to effectively capture the multimodal interaction of price and text data features, reflecting market signals and improving stock price movement prediction. The models compared in this study, along with their corresponding experimental results, are presented in Table 1.

5.2 Impacts of hybrid mixing module

In this study, we conducted experiments to compare the performance and effectiveness of classical methods such as LSTM and GRU networks in stock price prediction, as well as the hybrid information mixing module itself. To evaluate the performance, we replaced the hybrid information mixing module with a BiLSTM network and compared it with a transformer encoder to examine the potential of replacing attention mechanisms with MLP architectures in NLP.

The results showed that the hybrid information mixing module outperformed both the BiLSTM network and the transformer encoder, indicating its effectiveness in capturing multimodal information for stock price movement prediction. The hybrid information mixing module achieved a performance improvement of approximately 6.92% compared to BiLSTM. Furthermore, it outperformed the transformer-based method using self-attention by approximately 16.99%, despite having a simpler structure.

These findings demonstrate that the two MLP blocks in the hybrid information mixing module effectively combine and leverage the multimodal information in the mixed feature, leading to improved stock price prediction performance. The hybrid information mixing module proves to be more effective than BiLSTM alone, and it offers a promising alternative to the transformer-based approach by achieving comparable or even superior performance with a simpler architecture.

5.3 Comparative analysis

In this study, we utilized BiLSTM as the algorithm of choice for analyzing price data, replacing the use of GRU and LSTM. BiLSTM, being a variant of LSTM, offers the advantage of processing input sequences in both forward and backward directions, thereby enabling it to capture more comprehensive contextual information and enhance model performance.

To evaluate the performance of BiLSTM in comparison to other algorithms in stock price movement prediction, namely GRU and LSTM, we conducted a comparative experiment for price embedding. The results of this experiment are summarized in Table 2.

The experimental findings demonstrate that BiLSTM achieves higher accuracy compared to LSTM and GRU, with a notable performance of 71.23%. This outcome indicates that BiLSTM serves as a superior algorithm for price embedding and effectively captures the market signals associated with historical price volatility in the prediction of stock price movements.