Using artificial intelligence to analyze and classify music emotion

Abstract

With the rapid development of music digitization and online streaming services, automatic analysis and classification of music content has become an urgent need. This research focuses on music sentiment analysis, which is the identification and classification of emotions expressed by music through algorithms. The study defines and classifies possible emotions in music. Then, advanced artificial intelligence techniques, including traditional machine learning and deep learning methods, were employed to perform sentiment analysis on music fragments. In the process of creating and validating the model, the combination of convolutional neural network and long term memory network shows excellent performance in various performance indicators. However, for some complex or culturally ambiguous music fragments, the model may also suffer from misclassification problems. This provides the direction for further optimization of future research aimed at achieving more accurate music emotion analysis.

Keywords

Music emotion analysis artificial intelligence deep learning music classification cultural difference

1. Introduction

Music, as an important part of human civilization, not only runs through the development of human history, but also has become an indispensable element in life. It spans different cultures, bridging differences on a global scale and becoming a universal language. Each piece of music contains a unique emotional color, which not only reflects the emotional state of the composer and performer, but also is closely connected with the mood of the audience. Emotional understanding in music not only provides aesthetic pleasure for research, but also plays an important role in psychotherapy, filmmaking and other fields.

In recent years, with the rapid development of artificial intelligence technology, especially deep learning, the automatic recognition and classification of music emotions has attracted wide attention. The advancement of this technology provides unprecedented personalized and precise experiences for music recommendation, advertising production, film and television soundtrack and other fields. However, as an interdisciplinary study, music emotion analysis not only faces the challenge of accurately extracting emotional information from complex musical structures (such as melody, harmony, rhythm), but also needs to take into account the influence of different cultural backgrounds and personal experiences on the interpretation of musical emotions. In contrast to traditional analysis methods that rely on the expertise of musicians or musicologists, this research aims to combine advanced artificial intelligence technologies, especially deep learning, to achieve automated, efficient and accurate analysis of musical emotions. This is not only expected to advance the development of large-scale music data analysis, but also to provide new perspectives for understanding and applying music emotion. For example, in a music recommendation system, precise sentiment analysis can provide music choices that are more in line with users’ moods and preferences. In film production, according to the emotional needs of the scene, the appropriate soundtrack is automatically selected to enhance the emotional resonance of the audience. In addition, music emotion analysis also shows great potential in psychotherapy, for example, by analyzing the emotional characteristics of patients’ favorite music, to design more personalized treatment programs.

In exploring the field of musical emotion analysis, several studies have provided important insights and methodologies. Hsu et al. [1] analyzed the influence content of music emotion through electronic brain imaging (EEG), demonstrating the possibility of applying physiological signals to emotion analysis. This approach offers new insights into how music affects human emotions. On the other hand, Koelsch [2], through the coordinate-based meta-analysis, deeply discusses the emotions caused by music, highlighting the influence of music on emotional states, which is of great significance for the theoretical basis of music emotion analysis. Further, Yang [3] proposed the “Musi-ABC” model to predict music emotion, which utilized advanced machine learning technology and demonstrated the application potential of artificial intelligence in music emotion classification. In addition, Yang et al. [4] improved the accuracy of music emotion recognition by combining structural analysis and modal interaction, and the application of this method provided a new perspective for emotion analysis. In another research field, Miranda and Blais-Rochette [5] studied the relationship between neural personality and music listening in regulating emotions through meta-analysis, and proposed the importance of music in regulating emotions. This provides theoretical support for practical applications of music emotion analysis, such as psychotherapy. Similarly, Talamini et al. [6] studied how musical emotion affects the memory of emotional pictures, further confirming the close connection between music and emotion. These studies not only provide rich insights into the theory and practice of music emotion analysis, but also provide important guidance for future research directions and application fields. Through these studies, we can see that music emotion analysis is a multi-dimensional and interdisciplinary research field, involving many fields such as psychology, musicology and computer science. With the development of technology, it is foreseeable that music emotion analysis will have a deeper development in both theory and practice.

The implications of this research are clear in several ways. From an academic point of view, it provides a new theoretical framework and method for music emotion analysis, which helps to advance knowledge in this field. In practical application, the accurate music emotion analysis can improve the accuracy of the music recommendation system and make the recommendation more in line with the user’s mood and mood. In addition, it provides strong support for music selection and content creation in film and television production, advertising, games and other fields. Especially in today’s increasing attention to mental health problems, the application of music mood analysis in psychotherapy, such as the use of music to regulate and treat emotional disorders, is also particularly important.

However, music sentiment analysis faces a number of technical challenges. The first is the subjectivity of music emotion. Different listeners may have different emotional responses to the same piece of music. This requires an analytical approach that takes into account individual differences and cultural contexts. Secondly, effective extraction and analysis of complex features in music, such as melody, harmony, rhythm, etc., is the key to achieve accurate music emotion analysis. In addition, in the face of huge music data sets, designing efficient and accurate data processing and analysis algorithms is another important topic. Finally, as an interdisciplinary field, music sentiment analysis needs to effectively integrate the knowledge and methods of multiple fields such as musicology, psychology, and computer science, which is also an important challenge. By continuing to explore and overcome these challenges, we can expect music sentiment analysis to play an even more important role in its future development and application.

This research focuses on multiple core areas of music sentiment analysis and uses artificial intelligence techniques to delve into key questions in each area. In the section of theory and application, the research discusses the definition and classification of music emotion in depth, and emphasizes the importance of music emotion analysis in the current social and technological context. At the same time, it also summarizes the application status and trend of artificial intelligence, especially deep learning technology in music emotion analysis. Data is the cornerstone of AI applications. As a result, the study provides a detailed description of the data collection, including the selection of data sets, the method of labeling musical emotions, and how to extract key audio features from the music. In the part of model selection and design, the application of traditional machine learning method and deep learning method in music emotion analysis is compared, and then the model structure is designed specifically for music emotion analysis. Next, in the process of model training and verification, the paper deeply studies how to use the selected data set to train the model, how to determine the optimal training strategy, and uses cross-validation methods to ensure the generalization performance of the model. Finally, the results are analyzed in detail, the performance of the model, the possible causes of misclassification, and the advantages and disadvantages of the model in the task of music emotion analysis are discussed. Overall, this study aims to provide a comprehensive and in-depth view of how AI techniques can be combined to solve key problems in music sentiment analysis, and to lay a solid foundation for future research in this field.

2. Theory and application of music emotion analysis

2.1 Definition and classification of music emotion

Musical emotion can be defined as the emotional state or frame of mind triggered or expressed by music [7, 8]. This emotional response may be caused directly by the sound properties of music, or it may be triggered by personal experience or cultural background associated with music. Music emotion is the core component of music experience, and it is the most direct and profound experience when people interact with music [9]. Beethoven’s Moonlight Sonata, for example, is often described as a symbol of sadness and melancholy, reflecting the composer’s inner loneliness and struggle. Mozart’s Violin Concerto in G Major, on the other hand, is often seen as a symbol of joy and lightness, conveying a mood of pleasure and excitement.

There have been many ways to classify musical emotions. Traditionally, musical emotions have been grouped into basic emotional categories such as joy, sadness, fear, anger, relief, and tension. These basic emotional categories can be regarded as the “primary colors” of musical emotions, which are universal across different musical styles, cultures, and individuals.

However, the experience of musical emotion is far richer and more complex than these basic categories. Many scholars have proposed that musical emotions can be represented in a continuous two-dimensional space, which is often described as an “emotional circle” or “emotional plane”. One dimension represents the energy or activation of the emotion (for example, from relaxation to tension), while the other represents the positivity or pleasure of the emotion (for example, from sadness to happiness). This representation can capture more detailed and subtle musical emotional differences [10, 11].

The definition and classification of music emotion not only provide a theoretical basis for the analysis of music emotion, but also provide a deep insight for music creation, performance and appreciation. By deeply understanding the nature and classification of musical emotions, we can better capture and express the rich and diverse emotional experiences that music can bring.

2.2 Importance of music emotion analysis

Musical sentiment analysis plays a crucial role in multiple fields, providing a unique approach to understanding and exploiting the deeper meaning of music. The so-called recognition of music emotion is to automatically identify the connotation of music emotion, and the classification of different emotions is its prerequisite [12].

In music emotion analysis, this study considers using a variant of the Russell ring complex arousal model. Such models often include two – or three-dimensional Spaces to more accurately describe emotional states. The two-dimensional space covers emotional energy (e.g., from relaxation to tension) and emotional positivity (e.g., from sadness to happiness), while the three-dimensional model also includes the dimension of dominance to describe emotional control and influence (e.g., from helplessness to control) [13]. These models are very common in the field of musical emotion analysis and have proven to be effective tools for analyzing and classifying musical emotion.

In artistic creation and music production, an accurate understanding and analysis of musical emotions can help composers and producers create and adapt works more specifically to meet specific emotional and thematic needs. In addition, for singers and musicians, a deeper insight into the emotion of music helps them to convey the emotional content of their works more truthfully and powerfully, thus establishing a deeper emotional connection with the audience.

For music recommendation systems, music sentiment analysis provides an effective strategy to recommend music to users that matches their current emotional state and preferences [14]. This can not only improve the accuracy of recommendations, but also enhance the user’s music listening experience, making it more satisfying and enjoyable.

In the film, advertising and games industries, music is a key element of emotional drive, reinforcing the plot, deepening the emotion and enhancing the audience’s immersion. Music sentiment analysis can provide content creators with valuable advice on how to select or customize music to maximize its emotional impact [15].

In addition, in the field of psychology and therapy, music sentiment analysis is also seen as a potential tool. It could help experts better understand how individuals interact with music and how it affects their emotional state, leading to more effective approaches to mental health interventions.

Overall, music sentiment analysis plays a crucial role in connecting music and emotion, improving the effectiveness of art and technology applications, and enhancing people’s music experience.

2.3 Overview of the application of artificial intelligence in music emotion analysis

Artificial intelligence, especially deep learning technology in recent years, has revolutionized music sentiment analysis. Its high degree of automation and precision has led to unprecedented developments in the scale and depth of music sentiment analysis [16, 17].

Traditional methods of music emotion analysis mainly rely on music theory and the expertise of musicians. These methods may work well when dealing with simple and regulated music content, but they often fall short in the face of complex modern music and large-scale music databases. In contrast, AI technology can automatically extract audio features and identify and classify emotional patterns in music without too much human intervention.

The emotion of music is often multi-layered and multi-dimensional, including multiple elements such as melody, rhythm, harmony, and sound texture. Deep learning models, such as convolutional neural networks (CNNS) and recurrent neural networks (RNNS), have been widely used in music sentiment analysis [18, 19]. These models are able to deal with this complexity of music, automatically capturing its inherent emotional patterns.

In addition to deep learning, other AI techniques, such as support vector machines, decision trees, and random forests, have also found applications in music sentiment analysis [20]. These methods are often used for classification tasks based on hand-designed audio features, providing a more traditional perspective on music sentiment analysis.

With the continuous advancement of artificial intelligence technology, its application in music sentiment analysis is becoming more and more diverse and precise. Not only that, but AI has also brought new research directions and future possibilities to music sentiment analysis, making this field full of unlimited potential for both academic and practical applications.

3. Data collection

3.1 Selection of data sets

Selecting the appropriate data set is a key step in performing music sentiment analysis. To ensure the comprehensiveness and validity of the study, the dataset needed to be representative, diverse, and inclusive of a wide range of musical emotions.

Selection criteria:

Representativeness: The selected dataset should cover a wide range of musical styles and sources to ensure that the findings are universally applicable.

Diversity: The dataset should include different types of music in order to study how different musical styles and elements affect emotional expression.

The breadth of musical emotion: The dataset selected should contain multiple emotional categories to capture the full emotional dimension of the music.

Data quality and reliability: Data sets should be annotated by experts or validated with valid labels to ensure the accuracy of the analysis results.

This study selects three main data sets for music emotion analysis, as shown in Table 1.

Table 1
Data set information

Data set name	Number of musical pieces	Major emotional categories	Data source	Description
EmoMusic	500	Happy, sad, tense, relaxed	Popular songs from around the world	A popular dataset, annotated by experts, covering a variety of musical styles
MoodSwings	1000	Active, quiet, sad, happy	Movie, commercial and game tracks	Designed for multimedia content, tagging is done by crowdsourcing users
AI-MusicFeel	700	Melancholy, excitement, relaxation, thinking	Independent artists and open source music platform	The new data set is initially labeled using artificial intelligence and then verified by musicologists

By combining these three datasets, the ability to cover a wide range of styles and sources of music ensures the comprehensiveness of the study. The EmoMusic and MoodSwings datasets have been widely used in previous studies, and their reliability and validity have been validated. The new AI-MusicFeel dataset brings in more work by independent artists, enhancing the diversity of the dataset.

The selection of these datasets provides a solid foundation for the music sentiment analysis of this study and ensures the wide applicability and practicality of the results.

3.2 Music emotion annotation method

In order to ensure the accuracy of music sentiment analysis, it is essential to properly label the music fragments in the data set. Considering the characteristics and sources of different data sets, a variety of labeling methods were adopted to ensure the accuracy and consistency of labeling.

The music emotion labeling methods of selected data sets in this study are summarized, as shown in Table 2.

Table 2
Music emotion labeling methods

Data set name	Annotation method	Participant	Tagging tool	Description
EmoMusic	Expert annotation	Musicologist and psychologist	Custom interface	It is marked by experienced experts according to the melody, harmony and rhythm of the music
MoodSwings	Crowdsourced annotation	Network user	Web platform	Through the network platform, a large number of users are invited to mark the emotion of the music clips, and then the final label is obtained by statistics
AI-MusicFeel	Hybrid method (AI preliminary labeling $+$ expert review)	The AI models are initially labeled and then reviewed and adjusted by musicologists	AI algorithm $+$ Custom interface	The trained AI model is used to make preliminary annotations, and then these annotations are checked and fine-tuned by experts to ensure the accuracy of the annotations

Academic basis of expert annotation: According to the research of music psychology and cognitive science, expert annotation can dig deep into the structure and emotional characteristics of music, and provide a deep understanding of musical emotion analysis [21].

Technical details of crowdsourcing: Crowdsourcing methods often involve big data analysis and the wisdom of crowds theory, which can effectively capture the emotional responses of audiences in different cultures and contexts.

The innovation of the hybrid approach: combining the computing power of AI with the delicate perception of human experts improves the efficiency and accuracy of the labeling process, while reducing subjective biases and errors.

By adopting these diversified labeling methods, the quality of music emotion labeling is ensured, which provides a solid foundation for subsequent model training and verification.

3.3 Audio feature extraction

Audio feature extraction is a key step in music sentiment analysis because it determines the type of music information that a model can use. To fully capture the emotional content of the music, the study selected a series of features extracted from time, frequency, and statistical domains.

The main audio features extracted from different data sets are shown in Table 3.

Table 3
Audio feature extraction

Data set name	Time domain feature	Frequency domain feature	Statistical field feature	Description
EmoMusic	RMS	MFCC	Jump rate	RMS is the overall loudness of the music; MFCC captures the overall sound of music; Jump rate describes changes in rhythm
MoodSwings	ZCR	Chroma	Entropy	ZCR describes the rhythm of music; Chroma captures the harmonic structure; Entropy measures the complexity of audio
AI-MusicFeel	RMS	Spectral Flux	Jump rate	Spectral Flux describes the change in frequency in the music, with other characteristics identical to EmoMusic

(1)

Time domain features: These features are directly extracted from the waveform of the music. For example, RMS (Root Mean Square) describes the overall loudness of the music, while ZCR (Zero-Crossing Rate) describes the rhythm of the music.

(2)

Frequency domain features: These features are extracted from the spectrum of music. For example, MFCC (Mel-frequency cepstral coefficients) are commonly used to describe the timbre of music, while Chroma features capture the harmonic structure of music.

(3)

Statistical domain features: These features provide statistical information about music. For example, the rate of jump describes the rate of change of rhythm in music, while entropy measures the complexity and uncertainty of the music.

Combining these features, emotional information in music can be captured from multiple dimensions. This provides rich and diverse input data for the subsequent music emotion analysis model, and enhances the model’s discriminant ability.

4. Model selection and design

4.1 Overview of traditional machine learning methods

Traditional machine learning methods have a long history and wide application in the field of music sentiment analysis. These methods, often based on statistics and experience, model extracted audio features to predict the emotional labels of music.

Here are several traditional machine learning methods commonly used in music sentiment analysis:

(1)
Support Vector Machine (SVM)

SVM aims to find a hyperplane that maximizes the boundary between two different classes. Given an audio feature $x$ and an emotional label $y$ , the SVM tries to solve the following optimization problem:

$\displaystyle\min_{w,b}\frac{1}{2}||w||^{2}+C\sum\limits_{i=1}^{N}{\xi_{i}}$

subject to:

$\displaystyle y_{i}\left({w\cdot x_{i}+b}\right)\geqslant 1-\xi_{i},\quad\xi_{% i}\geqslant 0$

Where $w$ is the weight vector, $b$ is the deviation term, $\xi_{i}$ is the relaxation variable, and $C$ is the regularization parameter. Support vector machines (SVMS) perform well in processing high-dimensional feature Spaces and are suitable for complex music emotion classification tasks.
(2)
Decision Tree

Decision trees predict the emotional labels of music through a series of rules. Each node is judged based on a feature and a threshold until it reaches a leaf node, which is the emotion category. For example, a node may decide whether to enter the left or right subtree based on some value in the MFCC.
(3)
Random Forest

A random forest is a collection of decision trees. Each tree is trained on a subset of the data and a subset of the features. For the prediction of musical emotion, the random forest considers the output of all decision trees and takes a majority vote. The formula is expressed as follows Eq. (1):

$\displaystyle y=\text{mode}\left\{{T_{1}(x),T_{2}(x),\ldots,T_{N}(x)}\right\}$ (1)

Where $T_{i}(x)$ represents the output of the $i$ tree.

These traditional methods have their advantages and limitations. For example, SVM can handle linearly indivisible cases well, but may require a lot of computational resources. Decision trees and random forests, on the other hand, are easy to understand and interpret, but may suffer from overfitting. In music sentiment analysis, choosing the best machine learning method needs to take into account the characteristics of the data and the objectives of the research.
4.2 Overview of deep learning methods

Deep learning, as the most concerned branch of machine learning in recent years, has shown great potential in music sentiment analysis. Deep learning models, especially neural networks, can automatically learn complex feature representations, avoiding the need for manual feature engineering in traditional methods.

Here are some common deep learning methods used in music sentiment analysis:

(1)
Convolutional Neural Network (CNN)

CNNs are well suited for working with data that has a local structure, such as a time series of audio. By using convolutional layers, the network can extract local features in music. The basic formula of the model is shown in Eq. (2):

$\displaystyle f(x)=\textit{ReLU}(W\ast x+b)$ (2)

Where $\ast$ represents the convolution operation, $W$ and $b$ are weights and biasing, respectively, and ReLU is the activation function.
(2)
Recurrent Neural Network (RNN)

RNNs are designed to process sequential data, making them ideal for time series audio data. The key idea is that the network has a memory to store information about previous time steps. The basic model formula is as follows: Eqs (3) and (4) are shown:

$\displaystyle h_{t}=\tanh\left({W_{hh}h_{t-1}+W_{xh}x_{t}+b}\right)$ (3) $\displaystyle y_{t}=W_{hy}h_{t}+c$ (4)

Where $h_{t}$ is the hidden state of $t$ at time, $y_{t}$ is the output, $x_{t}$ is the input feature, and $W$ and $b$ are the weights and biases.
(3)
Long Short-Term Memory network (LSTM)

LSTM is a variant of RNN that is designed to handle long sequences to avoid long-term dependency problems. At the heart of the LSTM are its three gates (input gate, forget gate, and output gate), which control how information flows into, retains, and flows out of the memory unit.

The research briefly summarizes the key characteristics of these three methods, as shown in Table 4.

Table 4
Deep learning methods

Model name Main use Advantage Shortcoming

CNN Audio feature extraction Can automatically learn local features; Fast processing speed The global structure of the music may be overlooked

RNN Sequential data processing Able to capture the time dependence of music May experience long-term dependency problems

LSTM Long sequence data processing The long-term dependence problem of RNN is solved. Ability to capture dependencies over a longer time horizon High computational complexity

Comparison of traditional machine learning and deep learning methods:

Traditional machine learning methods such as SVM, decision trees and random forests have their unique advantages in music sentiment analysis, such as the efficiency of SVM in processing high-dimensional data, and the easy interpretation of decision trees and random forests. However, these methods can require significant computational resources and are susceptible to overfitting.

In contrast, deep learning methods such as CNNS, RNNS, and LSTMS are capable of automatically learning complex feature representations, avoiding the need for manual feature engineering. These methods are especially suitable for dealing with music data with rich temporal structure. However, deep learning models typically require more training data and computational resources.

In the specific application of music sentiment analysis, choosing the best machine learning method needs to take into account the characteristics of the data, the required computational resources, and the specific needs of the task. Different musical sentiment analysis tasks may require different model structures or combination strategies.
4.3 Structural design of the model suitable for music emotion analysis

Model name	Main use	Advantage	Shortcoming
CNN	Audio feature extraction	Can automatically learn local features; Fast processing speed	The global structure of the music may be overlooked
RNN	Sequential data processing	Able to capture the time dependence of music	May experience long-term dependency problems
LSTM	Long sequence data processing	The long-term dependence problem of RNN is solved. Ability to capture dependencies over a longer time horizon	High computational complexity

Given the richness and complexity of music, it may be difficult for a single model structure to capture all the emotional information in music. Therefore, a reasonable strategy is to combine different types of models to take advantage of their complementary strengths.

The following is a hybrid model structure specifically designed for music emotion analysis:

(1)
Feature extraction layer:

Use CNNS to extract local features from the original audio. The layer mainly consists of several convolution layers and pooling layers. The following Eq. (5) is shown:

$\displaystyle h_{t}=\tanh\left({W_{hh}h_{t-1}+W_{xh}x_{t}+b}\right)$ (5)
(2)
Sequence processing layer: composed of LSTM, it processes feature sequences processed by CNN and captures long-term dependencies. The following Eq. (6) is shown:

$\displaystyle h_{t}=\tanh\left({W_{hh}h_{t-1}+W_{xh}x_{t}+b}\right)$ (6)
(3)
Full connection layer: used for classification tasks, converting the output of LSTM into the final emotional label. The following Eq. (7) is shown:

$\displaystyle h_{t}=\tanh\left({W_{hh}h_{t-1}+W_{xh}x_{t}+b}\right)$ (7)

The following describes the key parts of this hybrid model, as shown in Table 5.

Table 5
Model structure design for music emotion analysis

Layer name Main function Advantage Shortcoming

Feature extraction layer (CNN) Extract features from the original audio Can automatically learn the local features of music, processing speed is fast Long-term dependencies may be overlooked

Sequence processing layer (LSTM) Process feature sequences to capture long-term dependencies It can capture the time structure of music and process long sequence data High computational complexity

Fully connected layer Convert LSTM output into emotional labels A direct connection to the target output provides a decision boundary It is possible to introduce overfitting

This hybrid model structure integrates the feature extraction capability of CNN and the sequence processing capability of LSTM, providing a powerful and flexible tool for music emotion analysis. The design of this structure ensures a deep understanding of the music data, while taking into account computational efficiency and the generalization ability of the model.
5. Model training and verification

Layer name	Main function	Advantage	Shortcoming
Feature extraction layer (CNN)	Extract features from the original audio	Can automatically learn the local features of music, processing speed is fast	Long-term dependencies may be overlooked
Sequence processing layer (LSTM)	Process feature sequences to capture long-term dependencies	It can capture the time structure of music and process long sequence data	High computational complexity
Fully connected layer	Convert LSTM output into emotional labels	A direct connection to the target output provides a decision boundary	It is possible to introduce overfitting

5.1 Partitioning of data sets

In order to ensure the generalization ability of the model, it is essential to partition the data set properly. Typically, data sets are divided into training sets, validation sets, and test sets.

(1)
Training Set: Parameters used to train the model.
(2)
Validation Set: Used to adjust the hyperparameters of the model, such as learning rate, regularization parameters, etc., and provide a baseline for early stopping, thereby preventing overfitting.
(3)
Test Set: used to evaluate the final performance of the model after model training and verification.

For illustration, the study considers a dataset containing 1,000 musical fragments. As shown in Fig. 1, are common data partitioning strategies.

Figure 1.
Data partitioning strategy.

The partitioning of data sets can be based on simple random sampling or more complex strategies such as stratified sampling, ensuring that each subset has a representative sample of various emotional labels.

Specific to the training of mathematical model, a given data set ${\cal D}=(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{N},y_{N})$ , where $x_{i}$ is the musical feature and $y_{i}$ is the corresponding emotional label, ${\cal D}$ can be divided into three subsets: ${\cal D}_{\textit{train}}$ , ${\cal D}\textit{val}$ and ${\cal D}_{\textit{test}}$ .

During the training process, the model will iterate on ${\cal D}_{\textit{train}}$ to optimize the loss function, such as the cross entropy loss, as shown in the following Eq. (8):

$\displaystyle{\cal L}=-\sum\limits_{i=1}^{N}{y_{i}}\log\left({p\left({y_{i}% \mid x_{i}}\right)}\right)+\left({1-y_{i}}\right)\log\left({1-p\left({y_{i}% \mid x_{i}}\right)}\right)$ (8)

Where $p(y_{i}|x_{i})$ is the prediction probability given by the model.

Validate the model with ${\cal D}\textit{val}$ , adjust hyperparameters, and monitor overfits. Once the model is trained, evaluate its final performance using ${\cal D}\textit{test}$ .

This data partitioning strategy ensures the generalization performance of the model on unknown data and provides a fair evaluation criterion.
5.2 Model training strategies

Model training strategy is the key to ensure efficient and stable learning. The following are the main training strategies designed for music sentiment analysis, each chosen based on rigorous scientific evidence to ensure the validity and repeatability of the experiment:

(1)
Batch Training:

In order to improve training efficiency, the data is usually divided into several small batches for training. For example, from the 700 training music pieces described above, model updates can be performed in subsets of batch size 50. This method can speed up the convergence of the model and reduce the requirement of memory resources.
(2)
Learning Rate Scheduling:

Using a larger learning rate at the beginning can speed up model training, but as training progresses, it may be necessary to gradually reduce the learning rate to ensure convergence. A common strategy is learning rate decay, such as reducing the learning rate by 10% after every 10 epochs. This strategy helps to advance quickly at the beginning of training and make more detailed adjustments as you approach the optimal solution.
(3)
Regularization:

To prevent overfitting, especially when the amount of data is relatively small, regularization techniques can be employed. L2 regularization is the most commonly used method, which adds a penalty term to the loss function that is related to the weight size. This helps stabilize the training process and guarantees the model’s ability to generalize on different data sets.
(4)
Early Stopping:

Verify performance on the set by monitoring, and stop training if performance does not improve significantly over successive epochs. This helps prevent overfitting and reduces unnecessary calculations. For example, training can be stopped when it is verified that the loss has not improved in 5 successive iterations.
(5)
Data Augmentation

New training samples are generated by making slight modifications to the original music snippet. Common audio data enhancement techniques include speed changes, tone variations, and adding background noise. Data enhancement can improve the recognition ability and generalization performance of the model for different musical emotions.

Table 6 provides an example that Outlines the model training strategy and its parameters:

Table 6
Model training

Policy name Parameter value/description Aim

Batch training Batch size: 50 Improve training efficiency and ensure efficient utilization of memory

Learning rate adjustment Initial value: 0.01, attenuation: 10% per 10 epochs The step size of weight updating is controlled to ensure model convergence

Regularization L2 regularization, $\lambda=0.001$ Prevents overfitting and stabilizes training

Early stop Stop condition: Verify that the loss has not improved for 5 consecutive epochs Save computing resources and prevent overfitting

Data enhancement Change speed, change tone, add noise Increase the generalization ability of the model to enhance the model’s understanding of musical emotion by providing diverse data

These strategies show good results in the practical application of music emotion analysis. Batch training and learning rate scheduling strategies ensure the efficiency and stability of training, L2 regularization and early stop strategies effectively prevent overfitting, and data enhancement strategies significantly improve the generalization ability of the model when dealing with different types of music. The comprehensive application of these strategies can significantly optimize the performance of music sentiment analysis and improve the accuracy and robustness of the model in practical applications.
5.3 Cross-verification and performance evaluation

Policy name	Parameter value/description	Aim
Batch training	Batch size: 50	Improve training efficiency and ensure efficient utilization of memory
Learning rate adjustment	Initial value: 0.01, attenuation: 10% per 10 epochs	The step size of weight updating is controlled to ensure model convergence
Regularization	L2 regularization, $\lambda=0.001$	Prevents overfitting and stabilizes training
Early stop	Stop condition: Verify that the loss has not improved for 5 consecutive epochs	Save computing resources and prevent overfitting
Data enhancement	Change speed, change tone, add noise	Increase the generalization ability of the model to enhance the model’s understanding of musical emotion by providing diverse data

In order to ensure the robustness of the model on different data subsets, and to ensure the effectiveness and repeatability of the experiment, cross-validation technology is usually adopted. In addition, carefully selected performance metrics can provide an intuitive representation of the model’s performance and provide direction for further improvements to the model.

(1)
Cross-validation:

k-fold cross-validation is a common technique used to evaluate the generalization ability of a model. In 10x cross validation, the original data set is randomly divided into 10 subsets. Nine subsets are used at a time for training, and the remaining one is used for validation. After each iteration, the average of the evaluation metrics is used to evaluate the overall performance of the model. This method reduces the bias caused by the selection of a specific subset of data and improves the accuracy and reliability of the evaluation.
(2)
Performance Metrics:

Accuracy: Accuracy is the basic indicator for evaluating the performance of the model, representing the ratio of correctly classified musical works to the total number of works. The following Eq. (9) is shown:

$\displaystyle\text{Accuracy }=\frac{\text{Number of correct predictions}}{% \text{Total number of predictions}}$ (9)

Confusion Matrix: The confusion matrix provides a detailed comparison between the model’s predictions and the actual labels, helping to identify the strengths and weaknesses of the model on a particular emotion category.

As shown in Fig. 2, there is a confusion matrix for the three emotional labels: happy, sad, and relaxed.

Figure 2.
Confusion matrix.

From the Fig. 2, you can see, for example, that there are five musical pieces that are actually “happy” but predicted to be “sad.”

Through cross-validation, the average performance of the model can be obtained, allowing a more fair and accurate assessment of its predictive power on previously unseen data. At the same time, selecting the appropriate performance evaluation index can provide a powerful guide for the subsequent model optimization.

Other indicators:

(1)
OC-ROC (Area under the curve):

This index is suitable for binary classification task and can be used to evaluate the classification performance of the model for positive and negative cases. The perfect classifier has an AUC of 1, while the random classifier has an AUC of 0.5.
(2)
Precision, Recall, F1 scores, etc.:

These measures take into account the number of true positives, false positives, and false negatives, providing multiple perspectives on model performance for the study.
(3)
Mean Squared Error (MSE). Commonly used in regression tasks, it is also suitable for models evaluating the prediction of emotional intensity in music. As shown in the following Eq. (10):

$\displaystyle\text{MSE}=\frac{1}{N}\sum\limits_{i=1}^{N}{\left({y_{i}-\hat{y}_% {i}}\right)^{2}}$ (10)

Where $y_{i}$ is the real label and $\hat{y}_{i}$ is the predicted value. This is a commonly used metric in regression tasks, but may also be relevant in some versions of musical sentiment analysis, such as emotion intensity prediction.

Some examples of performance indicators are shown in Fig. 3.

Figure 3.
Example of performance indicators.

This provides a comprehensive view of how different models perform on various performance evaluation measures. Such visual representations help research quickly evaluate and compare the performance of models.
6. Result analysis

6.1 Model performance comparison

To ensure that the best model was chosen for music sentiment analysis, the study compared several different models, including traditional machine learning models and deep learning models. Key metrics for comparison include accuracy, F1 scores, etc.

Exploration of advantages and limitations of the model:

Traditional machine learning methods, such as support vector machines and random forests, perform well on small or well-characterized music data, but may be limited by their feature extraction and processing capabilities when dealing with large or complex music data sets.

Deep Learning methods: CNNS and LSTMS excel at working with large-scale music data, especially at capturing the time-series properties and complex patterns of music. However, these models require large amounts of training data and computational resources, and may overfit on small data sets.

Discussion of practical application scenarios:

Application scenarios: In practical applications, such as music recommendation systems or sentiment analysis, deep learning methods (especially the combination of CNN and LSTM) may be better suited to large and diverse music datasets, as they can more effectively capture and analyze complex patterns in music. However, in scenarios where resources are limited or rapid prototyping is required, traditional machine learning methods may be more appropriate.

As shown in Fig. 4, the research compares the performance of the model:

As can be seen from the picture above:

(1)
Although traditional machine learning methods, such as support vector machines and random forests, perform quite well in music sentiment analysis, there are still some gaps compared with deep learning methods.
(2)
Good performance can be obtained using CNN or LSTM alone, which proves the powerful power of deep learning methods when processing music data.
(3)
By combining CNN and LSTM, the study further improves the performance of the model. This validates the previous hypothesis that combining multiple models can take advantage of their complementary strengths.

Overall, this comparison of model performance provides valuable insights into the strengths and weaknesses of different models for the task of music sentiment analysis, and provides guidance for future research directions.
6.2 Causes of incorrect classification

Although the best model in this study achieved 91% accuracy, some music fragments were misclassified. Exploring the causes of these errors will not only help research further optimize the model, but also provide insight into the difficulties of music sentiment analysis.

(1)
Complexity of music: Some musical works are emotionally complex; for example, a song may contain both happy and sad elements. This multi-layered nature of emotion makes it difficult for machine learning models to classify accurately. For example, a fast paced song with sad lyrics may be misclassified as “happy.”
(2)
Cultural background of music: Listeners in different cultures may interpret the same piece of music differently emotionally. For example, in some cultures, a particular melody may be interpreted as sad, while in others it may be seen as happy. To this end, taking cultural differences into account is crucial to improving the accuracy of the classification.
(3)
Limitations of feature extraction: Despite the use of advanced feature extraction techniques, musical features that have a decisive impact on emotion classification can still be missed. For example, certain subtle melodic changes or rhythmic patterns may not be fully captured.
(4)
Limitations of model structure: Even a combined deep learning model may not fully capture all relevant patterns in the music data, resulting in poor classification performance.

Some examples of misclassification and possible causes are shown in Table 7.

Table 7
Discusses the causes of wrong classification

True label Prediction tag Possible cause

Happy Sorrow Multi-level emotional structure

Sorrow Relax Cultural background difference of music

Relax Happy Limitations of feature extraction

Sorrow Happy Limitations of model structure

Figure 4.
Model performance comparison.

An in-depth analysis of a specific case of misclassification, such as analyzing the style of the misclassified music, the length of the music, the instruments used, etc. These analyses can reveal the specific causes of misclassification.

Consider using more sophisticated feature extraction techniques, such as audio signal processing combined with music theory analysis, to more accurately capture the emotional qualities of music. At the same time, annotated data under multi-cultural background can be considered to improve the adaptability of the model to different cultural interpretations.

Through these in-depth analysis and discussion, we can not only better understand the causes of misclassification, but also provide specific directions and solutions for future model optimization.
6.3 Advantages and disadvantages of the model are discussed

True label	Prediction tag	Possible cause
Happy	Sorrow	Multi-level emotional structure
Sorrow	Relax	Cultural background difference of music
Relax	Happy	Limitations of feature extraction
Sorrow	Happy	Limitations of model structure

In the task of music emotion analysis, this study uses a variety of models and finally chooses the mixed model of CNN $+$ LSTM. Here’s a discussion of the pros and cons of this hybrid model:

(1)
Advantages:

Feature extraction capability: By using CNN, models can automatically learn and extract meaningful features from raw audio data, reducing the need for manual feature engineering.

Processing sequence data: The introduction of LSTM enables the model to process time series data in music, capturing long-term musical dynamics and structure.

Flexibility: The design of the hybrid model can be easily adjusted and optimized. For example, you can adjust the number of convolutional layers or LSTM layers, or modify the loss function to suit different task requirements.

High generalization ability: Combining the local feature extraction of CNN and the global time dependence of LSTM, the hybrid model exhibits high generalization ability, even on complex and diverse music data.
(2)
Disadvantages:

Computational requirements: Hybrid models can be more expensive in parameters and calculations than single models, requiring more time and computational resources to train.

Model complexity: The complexity of the model can make debugging and optimization difficult. For example, you need to select and adjust hyperparameters for the CNN and LSTM sections, respectively.

Risk of overfitting: Although a hybrid model is highly flexible, it can also be more prone to overfitting, especially if the data volume is small. Strategies such as regularization and early stopping may need to be applied more carefully.

Interpretive challenges: Deep learning models, especially complex hybrid models, may not be as easy to interpret as traditional machine learning models. For some application scenarios, this may be a consideration.

In this study, music emotion analysis mainly starts from the perspective of automatic music analysis. This approach focuses on extracting emotional features directly from musical compositions, rather than based on reactions or feedback from users. Although this approach is technically feasible and performs well in many applications, it ignores the subjectivity and variety of musical emotional experiences.

Automated analysis of music emotion based on automated user response analysis is a valuable complementary perspective, although it is outside the scope of this study. Integrating user behavior, reactions, and feedback into sentiment analysis can provide a richer emotional dimension, which is worth exploring in future research.

To sum up, although the CNN $+$ LSTM hybrid model performs well in music sentiment analysis, there are still some challenges in its design and application. At the same time, these discussions also provide directions for further model improvement and research, especially in considering the diversity and subjectivity of musical emotional experience.
7. Conclusions

This research focuses on music emotion analysis, discusses the definition and classification of music emotion, and the importance of using artificial intelligence technology for emotion analysis. Music, as a borderless language, conveys emotions that can vary depending on culture, background, and the listener’s personal experience, which adds to the complexity of the analysis. Nevertheless, through reasonable data collection, preprocessing and feature extraction, the study established a robust data set as the research basis.

Academic contribution: In terms of model design, this study compares traditional machine learning methods with modern deep learning techniques. In particular, the hybrid model of CNN and LSTM shows excellent performance in the task of music sentiment analysis. This hybrid model successfully combines the advantages of CNN in processing local features with the capability of LSTM in processing time series data. In addition, this study introduces deep learning techniques in the field of music sentiment analysis, which is a significant improvement over existing techniques and provides new tools and methods for future research.

Implications for future research directions: Although the model performs well in most cases, it can still misclassify some complex or culturally ambiguous musical works. This finding points to directions for future research, including improving the model’s understanding of the cultural and emotional complexity of music, as well as exploring new feature extraction and classification techniques. In addition, future research could consider incorporating user response data to more fully understand and predict musical emotion.

Overall, music sentiment analysis is an area full of challenges and opportunities. Through this study, not only enhanced the understanding of music sentiment analysis technology, but also laid a solid foundation for future research in this field. With the advancement of technology, it is believed that the emotion contained in music will be more accurately analyzed, providing strong support for music recommendation, creation and other application scenarios.

References

Hsu

Zhen

Lin

Chiu

. Affective content analysis of music emotion through EEG. Multimedia Syst. 2018; 24(2): 195-210. doi: 10.1007/s00530-017-0542-0.

Koelsch

. A coordinate-based meta-analysis of music-evoked emotions. NeuroImage. 2020; 223: 117350. doi: 10.1016/j.neuroimage.2020.117350.

Yang

. Musi-ABC for predicting musical emotions. IEEE Access. 2023; 11: 79455-79465. doi: 10.1109/ACCESS.2023.3300042.

Yang

Shen

Zeng

Luo

Lin

. COSMIC: Music emotion recognition combining structure analysis and modal interaction. Multimed Tools Appl. 2023. doi: 10.1007/s11042-023-15376-z.

Miranda

Blais-Rochette

. Neuroticism and emotion regulation through music listening: A meta-analysis. Music Sci. 2020; 24(3): 342-355. doi: 10.1177/1029864918806341.

Talamini

Eller

Vigl

Zentner

. Musical emotions affect memory for emotional pictures. Sci Rep-UK. 2022; 12(1): 10636. doi: 10.1038/s41598-022-15032-w.

D’Onofrio

Caldwell

Limb

Smith

Kessler

Gifford

. Musical emotion perception in bimodal patients: relative weighting of musical mode and tempo cues. Front Neurosci-Switz. 2020; 14: 114. doi: 10.3389/fnins.2020.00114.

Pandeya

Bhattarai

Lee

. Deep-learning-based multimodal emotion classification for music videos. Sensors-Basel, 2021; 21(14): 4927. doi: 10.3390/s21144927.

Liu

. Detecting music-induced emotion based on acoustic analysis and physiological sensing: A multimodal approach. Appl Sci-Basel. 2022; 12(18): 9354. doi: 10.3390/app12189354.

10.

Ouyang

. Design of semantic matching model of folk music in occupational therapy based on audio emotion analysis. Occup Ther Int. 2022; 2022: 6841445. doi: 10.1155/2022/6841445.

11.

Qian

Liao

Tian

. EEG decoding for musical emotion with functional connectivity features. Biomed Signal Proces. 2024; 89: 105744. doi: 10.1016/j.bspc.2023.105744.

12.

Zheng

. EEG processing in emotion recognition: inspired from a musical staff. Multimed Tools Appl. 2023; 82(3): 4161-4180. doi: 10.1007/s11042-022-13405-x.

13.

Sun

Wen

Huan,

Chao

. Using machine learning analysis to interpret the relationship between music emotion and lyric features. PeerJ Comput Sci. 2021; 7: e785. doi: 10.7717/peerj-cs.785.

14.

Lee

Han

Lee

. Development of novel musical stimuli to investigate the perception of musical emotions in individuals with hearing loss. J Korean Med Sci. 2023; 38(12): e82. doi: 10.3346/jkms.2023.38.e82. PM: 36974396.

15.

Ahmed

Paquette

Zeitouni

Lehmann

. Neural processing of musical and vocal emotions through cochlear implants simulation. Clin EEG Neurosci. 2018; 49(3): 143-151. doi: 10.1177/1550059417733386.

16.

Seo

Huh

. Automatic emotion-based music classification for supporting intelligent IoT applications. Electronics-Switz. 2019; 8(2): 164. doi: 10.3390/electronics8020164.

17.

Wang

Guo

. Research on intelligent recognition and classification algorithm of music emotion in complex system of music performance. Complexity. 2021; 2021: 4251827. doi: 10.1155/2021/4251827.

18.

MacGregor

Müllensiefen

. The musical emotion discrimination task: A new measure for assessing the ability to discriminate emotions in music. Front Psychol. 2019; 10: 1955. doi: 10.3389/fpsyg.2019.01955.

19.

Harding

Gaudrain

Hrycyk

Harris

Tillmann

Maat

, et al. Musical emotion categorization with vocoders of varying temporal and spectral content. Trends Hear. 2023; 27: 23312165221141142. doi: 10.1177/23312165221141142.

20.

Lange

Frieler

. Challenges and opportunities of predicting musical emotions with perceptual and automatized features. Music Percept. 2018; 36(2): 217-242. doi: 10.1525/MP.2018.36.2.217.

21.

. Research on automatic classification method of ethnic music emotion based on machine learning. J Math-UK. 2022; 2022: 7554404. doi: 10.1155/2022/7554404.