Effects of individual factors on perceived emotion and felt emotion of music: Based on machine learning methods

Abstract

Music emotion information is widely used in music information retrieval, music recommendation, music therapy, and so forth. In the field of music emotion recognition (MER), computer scientists extract musical features to identify musical emotions, but this method ignores listeners’ individual differences. Applying machine learning methods, this study formed relations among audio features, individual factors, and music emotions. We used audio features and individual features as inputs to predict the perceived emotion and felt emotion of music, respectively. The results show that real-time individual features (e.g., preference for target music and mechanism indices) can significantly improve the model’s effect, and stable individual features (e.g., sex, music experience, and personality) have no effect. Compared with the recognition models of perceived emotions, the individual features have greater effects on the recognition models of felt emotions.

Keywords

music emotion recognition individual factor machine learning perceived emotion felt emotion

For the ability to convey or induce emotions, music plays an important role in human life. Almost all music pieces are created to convey feelings, composers create music to resonate with their listeners, and performers use the language of music to elicit the emotional responses of audiences (Feng, Zhuang, & Pan, 2003). Therefore, the emotion information of music has been widely used in, for example, music information retrieval (MIR; Downie, 2008; Wang, Yang, Wang, & Jeng, 2012; Y. H. Yang & Chen, 2010), music recommendation (Deng, Leung, Milani, & Chen, 2015; Park, Ihm, Jang, Nasridinov, & Park, 2015), and music therapy (Bernatzky, Presch, Anderson, & Panksepp, 2011; Dingle, Kelly, Flynn, & Baker, 2015). Considering the importance of emotion information of music, a technology for automatically recognizing music emotion is urgently needed (X. Yang, Dong, & Li, 2018). Because there is so much existing music, it is difficult for people to manually tag emotion information to all music. Thus, music emotion recognition (MER), a research area investigating computation models to detect the emotion of music (Chen, Lee, Hsieh, & Wang, 2015), has become an urgent issue, and computer scientists have expended lot of effort on it during the past decade.

MER constitutes a process of using computers to (a) extract and analyze music features from original music, (b) construct the relationship between music features and perceived emotions, and (c) recognize the emotion that the untagged music expresses (see Figure 1). Through the above process, a music database can be organized and managed according to emotion (Y. E. Kim et al., 2010). Therefore, MER has become an important part of the research on digital music applications (Dobashi, Ikemiya, Itoyama, & Yoshii, 2015).

Figure 1.

Music emotion recognition process (X. Yang, Dong, & Li, 2018).

Although the research on MER has made considerable progress, common problems remain. A critical problem is the strong subjectivity of the ground-truth data, which reflect the perceived emotions of human beings, because different people listening to the same music may produce different emotion perceptions (Gabrielsson, 2001). To solve this problem, each musical segment must be annotated by many subjects (Raykar et al., 2010) to obtain a relatively accurate emotional assessment. However, it raises another problem, that is, the emotions identified by MER models are not the emotions perceived by the target individual but the average perception of a group of individuals. Therefore, the importance of using personalized MER models has been advocated (Y. H. Yang, Lin, & Chen, 2009), despite the difficulty of collecting modeling data for everyone. As a compromise, we used individual features as inputs for our MER models, and the individual factors were classified according to the difficulty of acquisition. We suppose that the MER models’ performance will improve with the increase in individual features.

Notably, in the field of music cognition, the judgment of musical emotion has proven to be influenced by individual factors, for example, age (Dolgin & Adelson, 1990; Laukka, 2007), sex (Kratus, 1993), music education (Mualem & Lavidor, 2015), trait empathy (Kawakami & Katahira, 2015), and personality (Kallinen & Ravaja, 2006). However, these studies have often focused on a single or small number of factors and did not consider different types of individual factors together. Through machine learning (ML) methods, music emotion judgments were first considered from a computational perspective by Vempala and Russo (2018), and this method opened new lines of music cognition research. In this study, ML toolkits used in previous MER studies, such as multiple linear regression (MLR; Soleymani, Caro, Schmidt, Sha, & Yang, 2013), k-nearest neighbor regression (KNN; Dewi & Harjoko, 2010), support vector regression (SVR; Deng et al., 2015), and so forth (see below), were applied to investigate the relationship between individual factors and musical emotion.

To better select modeling features, we had to review the process of listening to music. And the data collected during this process was divided into three categories based on the difficulty of acquisition (Figure 2). The first type, easiest to obtain, is the music information that can be obtained directly from music without considering the listener and has been the only input in most MER studies. The personal information reflecting relatively stable individual characteristics is the second type of data, for example, sex, academic background, music experience, music preference, and personality (Y. H. Yang, Su, Lin, & Chen, 2007). The third type of data must be collected when listening or immediately after listening (real-time data) and includes subjective evaluation (e.g., mechanism indices; Juslin, Harmat, & Eerola, 2014), objective evaluation (Kallinen & Ravaja, 2006), context, physiological signals (e.g., skin conductance, electrocardiogram, pulse, and facial muscle activity; Gouizi, Bereksi Reguig, & Maaoui, 2011; J. Kim & André, 2008; Li & Chen, 2006; Vempala & Russo, 2018), and brain signals (e.g., electroencephalograph; Bhatti, Majid, Anwar, & Khan, 2016; Byun, Lee, & Han, 2017; Jenke, Peer, & Buss, 2013). In this study, all three types of data were considered, and we collected audio features, stable individual factors, subjective feelings, and perceived emotions for modeling.

Figure 2.

Three types of data that can be collected in different contexts.

The term “music emotion” in the field of MER has been used to represent the audience’s perceptual emotion of music (X. Yang et al., 2018), and felt emotions (emotions evoked by music) have seldom been considered. Although the relationship and differences between perceived emotions and felt emotions have been widely discussed by psychologists (e.g., Gabrielsson, 2001; Juslin & Zentner, 2001; Kallinen & Ravaja, 2006), using ML models to predict felt emotions of music is rare. Therefore, we also considered using ML methods to construct recognition models of felt emotion and comparing them with recognition models of perceived emotion. Gabrielsson (2001) concluded after conducting a review that perceived emotion and felt emotion are the “objective” and “subjective” aspects of music-elicited emotion. Each music may have a recognizable character (e.g., emotional meaning) that differs from the subjective meaning of different listeners (Kallinen & Ravaja, 2006). In the evaluation to determine if felt emotions are more subjective, we supposed that individual features would have greater effects on feeling models than perception models. In addition, researchers have found more complex models that account for music-elicited emotions better than the basic emotion and dimensional emotion models (e.g., a 9-factor model of music-induced emotions; Zentner, Grandjean, & Scherer, 2008). And whether musical and naturalistic emotions really map onto one another in a 1:1 fashion is still controversial (Allen, Walsh, & Zangwill, 2013). But in this study, basic emotions were used to conduct the evaluation. We considered four representative emotions—happy, relaxed, sad, and angry—because of their relation to basic emotions, and because they cover the four quadrants of the 2-dimensional model of emotion (Laurier, Grivolla, & Herrera, 2008). And these four emotions were also widely used in previous MER studies (e.g., Han, Rho, Dannenberg, & Hwang, 2009; Y. E. Kim et al., 2010; Laurier et al., 2008).

By combining ML techniques with research methods in the area of music psychology, this study contributes to the field of music cognition and MER. The contribution to music cognition is as follows: We analyzed the effect of individual factors on music emotional judgment from a computational modeling perspective; in this manner, we investigated the relationship between variables and constructed models that automatically recognize individuals’ perceived emotion or felt emotion of music. The contribution to the field of MER is as follows: We attempted to solve the subjectivity problem by introducing individual features; our MER models provided a reference for the creation of personalized MER systems. And three research questions arose. First, as mentioned above, traditional MER researches used music features to predict average perceived emotion of a group of individuals (also refer to as the general MER model), which was inaccurate for each individual. Therefore, we decided to build MER models that predict perceived emotions of each individual (the personal MER model). And we investigated whether using only music features can also effectively predict individuals’ perceived emotions by comparing the prediction effects of the general and personal MER models. We assumed that individuals’ perceived emotions will be more difficult to predict, that is, the personal MER models will perform worse than the general MER models (Hypothesis 1). Second, we needed to investigate whether adding individual features can improve the effect of personal MER models. And we believed that, when individual features are added, the personal MER models will perform better than using only music features as input (Hypothesis 2), because the recognition models may be able to obtain additional valid information from individual features. Third, felt emotion of music might be more susceptible to individual features than perceived emotion, so we assumed that individual features will have greater effects on feeling models than those on perception models (Hypothesis 3). In addition, individual features considered in Hypothesis 2 and Hypothesis 3 included sex, academic background, music experience, personality, mechanism indices, preference for music, familiarity with music, felt emotion (perception models), and perceived emotion (feeling models).

Methods

To build our MER models, we collected audio features, stable individual factors (demographic property, music experience, and personality), subjective feelings of excerpts (felt emotion, preference, familiarity and mechanism indices), and individual perception of music emotion. The ML methods used are introduced generally below. All features extracted for modeling are listed in Supplemental material A.

Stimuli, musical feature extraction, and reduction

Sixty famous popular songs were collected from Chinese albums, and the collected excerpts were trimmed to 25 s and converted to a uniform format: 22,050 Hz, 16 bits, and mono channel PCM WAV. Each excerpt was annotated by 20 participants in our experiment.

Then, we need to extract effective information from each excerpt as input to later models, and Figure 3 presents the complete flow of audio signal processing in this study. For musical feature extraction, we used librosa (McFee et al., 2015), a Python package for audio and music signal processing. Spectral features (root mean square [RMS], mel-frequency cepstrum coefficient [MFCC], spectral centroid, spectral contrast, spectral rolloff, zero-crossing rate, and tonnetz), which were widely used to represent the intensity, timbre, or pitch of music (see Table 1), and rhythmic features (by beat and tempo detection) were extracted from each excerpt, and all the features were used in other MER studies (Harte, Sandler, & Gasser, 2006; Jiang, Lu, Zhang, Tao, & Cai, 2002; Tzanetakis & Cook, 2002; Wieczorkowska, Synak, & Raś, 2006; Yang, Su, et al., 2007).

Figure 3.

Audio signal processing method in this study.

Table 1.

Illustration of the spectral features.

Feature	Description
Root mean square (RMS) value	One of the descriptive features which represent the intensity of music; extracted in a Frame (Lu, Liu, & Zhang, 2005).
Mel-frequency cepstrum coefficients (MFCCs)	Widely used to represent the timbre of music in MER studies (Fu, Lu, Ting, & Zhang, 2010); low-dimensional representation of the spectrum obtained from the Mel-scale distortion (Wang, Yang, Wang, & Jeng, 2012); reflect the nonlinear frequency sensitivity of the human auditory system.
Spectral centroid	One of the dimensions of statistical spectrum descriptors (SSDs), which are usually related to the structure of timbre (Soleymani, Caro, Schmidt, Sha, & Yang, 2013).
Spectral contrast	One of the dimensions of SSDs; high contrast values generally correspond to clear, narrow-band signals, while low contrast values correspond to broad-band noise (Jiang, Lu, Zhang, Tao, & Cai, 2002).
Spectral rolloff	One of the dimensions of SSDs, which are usually related to the structure of timbre (Soleymani et al., 2013).
Zero-crossing rate	The zero-crossing rate of an audio time series; extracted in the spectrogram of music; related to the timbre of music (Chang, Lo, Wang, & Chung, 2010).
Tonnetz	The tonal centroid features (tonnetz) are computed following the method of Harte, Sandler, and Gasser (2006); extracted in the chromagram, which is related to the pitch of music.

MER: music emotion recognition.

After feature extraction, each excerpt was represented in a subspace of high dimensionality (38,778 dimensions per excerpt). To manage the high-dimensional dataset, feature reduction is typically a critical step that reduces the storage and computational space (Vempala & Russo, 2018). Thus, principal components analysis (PCA) was applied to reduce the dimensionality of the data. After applying PCA, dimensions with little variation were eliminated, and 58 dimensions explaining 99% of the variation were retained. These 58-dimensional features reflecting music information would be one of the final inputs of our MER models.

Participants

Participants were recruited from the campus of Zhejiang University, and each was rewarded with ¥40. The experiment was designed such that each excerpt would be annotated 20 times by different participants, and we obtained a complete data set from 54 participants. On average, the final 54 participants (33 females, 21 males) were aged 22.56 years (SD = 1.99). Five participants annotated the entire dataset.

Stable individual factors

Using personal information for MIR has been advocated (Chai & Vercoe, 2000; Y. H. Yang, Su, et al., 2007); therefore, individual factors are critical inputs of our MER model. The following stable individual factors evaluated by Y. H. Yang, Su, et al. (2007) were collected before the experiment:

1) Demographics: We collected the age, sex, occupation, and academic background (defined as follows: the first academic category comprises colleges of liberal arts, social science, and management, and the second academic category comprises colleges of engineering, science, medicine, and computer science) of the participants, although the relationship between academic background and music emotion is rarely discussed. Only sex and academic background were applied in our models because the participants were mainly college students.

2) Music experience: We represented music experience in terms of their habit of listening to music (from rare to often) and ability to play an instrument (can or cannot) and whether they had received professional music training and liked to listen to music with the following prototypical emotions used by Kallinen and Ravaja (2006): alert, surprised, energetic, peppy, happy, satisfied, relaxed, calm, inactive, sleepy, bored, vegetated, sad, hopeless, fearful, and angry. Because none of the participants had received professional music training, this feature was removed.

3) Personality: We used the Big Five personality traits to describe personality. A Chinese-language 44-item Big Five Inventory personality scale, assessed to be valid (Carciofo, Yang, Song, Du, & Zhang, 2016) was applied to measure the personality of each participant.

Experimental design

The experiment was designed such that (a) each participant listened to 12 of the 60 excerpts and (b) each excerpt was heard by 20 unique participants. To collect the personal information of participants, after a brief description of the experiment, the participant was asked to fill out questionnaires described above. Next, participants received a listening order that was independently randomized to minimize the influence of presentation order. Each excerpt was preceded by 30 s of silence and followed by self-report questionnaires.

In the self-report phase, participants were first asked to evaluate the emotions that the music aroused in them while listening (i.e., emotion felt; “Did you feel happy when you listened to the music?”), the preference and familiarity of the excerpt, and the mechanism indices (i.e., “Did the music evoke a memory of an event from your life?”). After the subjective measures, participants were then asked to evaluate the perception of the emotions expressed by the music (i.e., emotion perceived; “Do you think the music expresses a happy emotion?”). The subjective measurements had to be performed first because it may dilute faster compared with the more objective evaluation of the perceived emotion and the felt emotion may have included physiological responses that decrease as a function of time (Kallinen & Ravaja, 2006). In addition, the participants were asked to concentrate on the music and listen to the music with their eyes closed.

In general, each participant performed the experiment according to the following sequence

S \to Ex \to Sm \to Om \to S \to Ex \to Sm \to Om \dots

In this sequence, S indicates silence, Ex indicates excerpt, Sm indicates subjective measurements, and Om indicates objective measurements. All the target emotions (happy, relaxed, sad, angry), felt and perceived, were evaluated on two scales from 1 (not at all) to 5 (very much). The mechanism indices (brain stem reflex, rhythmic entrainment, episodic memory, conditioning, visual imagery, contagion, musical expectancy, cognitive appraisal) were measured by the MecScale (Juslin et al., 2014). The preference for and familiarity with music were also evaluated on a scale from 1 (not at all) to 5 (very much). In addition, for subsequent modeling, all continuous inputs were scaled to a value between 0 and 1(e.g., the preference for music), and all binary inputs were labeled as 0 or 1 (e.g., the mechanism indices).

ML methods

In this study, we attempted to form the mapping relations among music features, individual factors, and emotion (i.e., perceived emotion and felt emotion). And the ML method is a bridge that links the aforementioned variables. A multitude of ML methods is available for both classification and regression. In this study, we formulated MER as a regression problem (Y. H. Yang, Lin, Su, & Chen, 2007) and adopted the random forest regression and the Bayesian ridge regression to train regressors. Other ML methods, such as MLR (Soleymani et al., 2013), KNN regression (Dewi & Harjoko, 2010; Pao et al., 2008), and SVR (Deng et al., 2015; Imbrasaitė, Baltrušaitis, & Robinson, 2013; Y. H. Yang, Su, et al., 2007), were also attempted and presented below for reference. In addition, scikit-learn (Pedregosa et al., 2011), a Python module integrating a wide range of state-of-the-art ML algorithms for medium-scale supervised and unsupervised problems, was applied for modeling. More details on the ML methods are presented along with the results below.

Results

Manipulation check

As the first step of the exploration of the data, we assessed whether the distribution of annotated emotions was appropriate for modeling. Table 2 shows the mean ratings for perceived and felt emotions. The mean rating for perceived anger was significantly lower than for the other emotions, χ² = 30.07, p < .01, and only 11 excerpts’ mean ratings for perceived anger were higher than 2, which means “a little angry.” In addition, anger was not well-aroused by our excerpts (felt anger; M = 1.24, SD = 0.27) compared to other emotions. Therefore, mean ratings for perceived and felt anger were retained for correlation analyses but excluded from modeling.

Table 2.

Mean ratings for perceived and felt emotions.

	Perceived emotions		Felt emotions
	M	SD	M	SD
Happy	2.13	0.98	2.08	0.71
Relaxed	2.36	0.76	2.55	0.61
Sad	2.25	0.91	1.84	0.63
Angry	1.45	0.51	1.24	0.27

SD: standard deviation.

Correlation analyses

Before building MER models, we analyzed how well the independent variables (stable individual features and subjective feelings) accounted for the dependent variables (perceived emotions and felt emotions). For example, we observed that individuals’ preferences for music were positively correlated with perceived happy ratings, r(1198) = 0.25, p < .01; perceived relaxation ratings, r(1198) = 0.19, p < .01; felt happiness ratings, r(1198) = 0.49, p < .01; and felt relaxation ratings, r(1198) = 0.34, p < .01. These results are presented in their entirety in Supplemental material B. In addition, among the four perceived emotions, emotions with same valence were positively correlated, i.e., perceived happy and perceived relaxed, r(1198) = 0.57, p < .01, and emotions with a different valence were negatively correlated (Table 3). Felt emotions had the same phenomenon as perceived emotions, although the average correlation coefficient (M|r| = 0.42, SD = 0.15) was smaller than perceived emotions (M|r| = 0.29, SD = 0.17). We also found that all the perceived emotions were positively correlated with the corresponding felt emotions. The aforementioned results illustrate the linear relationship between variables, but not all results can be explained by a linear relationship. Thus, we used ML to investigate their relationship.

Table 3.

Bivariate correlations between perceived emotions and felt emotions.

	1	2	3	4	5	6	7	8
1. Perceived happy	1
2. Perceived relaxed	.57**	1
3. Perceived sad	−.62**	−.42**	1
4. Perceived angry	−.23**	−.36**	.36**	1
5. Felt happy	.66**	.45**	−.48**	−.16**	1
6. Felt relaxed	.40**	.59**	−.26**	−.28**	.54**	1
7. Felt sad	−.45**	−.28**	.67**	.26**	−.45**	−.23**	1
8. Felt angry	−.10**	−.17**	.21**	.60**	−.09**	−.20**	.27**	1

Correlation is significant at the 0.01 level (2-tailed).

MER models using audio features only

In this section, we built the first type of MER model and attempted to verify Hypothesis 1. We used the same algorithm and input to construct the MER models and used the model to predict individual perceived emotions and average emotions, respectively.

To obtain relatively good models, we must select the algorithm first. Using audio features as inputs and average emotions as the ground truth, we used the following five algorithms for modeling: MLR, BR, SVR, RF, and KNN. The best performing algorithm would be used for subsequent analysis and modeling.

We built three models separately to predict the perception of happy, relaxed, and sad. The dataset in the current models is too small (60 samples), so the performances of our models were evaluated by the leave-one-out cross-validation technique, which uses one data instance of the dataset as the testing data and uses the remaining instances as training data to train the regressor (Y. H. Yang, Su, et al., 2007). And the prediction accuracy of a regressor was evaluated in terms of the root mean-squared error (RMSE)

R M S E = \sqrt{\frac{1}{N} \sum_{t = 1}^{N} {(o b s e r v e d_{t} - p r e d i c t e d_{t})}^{2}}

(1)

where $o b s e r v e d_{t}$ is the ground truth and $p r e d i c t e d_{t}$ is the predicted result of our model. RMSE values for the models applying different algorithms are shown in Table 4, and models applying the random forest regression performed best.

Table 4.

Mean RMSE values of five ML models (60 samples).

	MLR	BR	SVR	RF	KNN
Happy	1.89	1.27	0.85	0.83	0.83
Relaxed	1.93	1.21	0.66	0.65	0.69
Sad	1.84	1.24	0.82	0.79	0.82
Mean	1.89	1.24	0.77	0.76	0.78

RMSE: root mean-squared error; MLR: multiple linear regression; SVR: support vector regression; KNN: k-nearest neighbor regression; BR: Bayesian ridge; RF: Random forest.

Bold values are the RMSE values of the best performing algorithm.

To evaluate the effect of the MER models in predicting individual emotions, we trained a personalized regressor, by using random forest regression and using audio features as input, for each participant who had annotated all 60 songs. Leave-one-out was also used to evaluate the performance, and the mean RMSE values for the five participants are shown in Table 5. The general regressor achieved a mean RMSE value of 0.83 for happy, 0.65 for relaxed, and 0.79 for sad (Table 4); its performance decreased to 1.26 for happy, 1.16 for relaxed, and 1.00 for sad on average when predicting the five participants. The results indicate that the MER models using audio features as the only input are less effective in predicting individual emotions than the mean opinions. We supposed that the mean perception of music would be more responsive to the general part of music; thus, we could better predict the mean opinions using audio features. When predicting the individual’s perception of emotion, individual uniqueness must be considered.

Table 5.

Mean RMSE values for the five participants.

Emotion	Participant 1	Participant 2	Participant 3	Participant 4	Participant 5	M
Happy	0.94	1.24	1.13	1.36	1.63	1.26
Relaxed	1.06	0.98	0.76	1.39	1.62	1.16
Sad	0.89	1.32	0.82	0.85	1.11	1.00
Mean	0.96	1.18	0.90	1.20	1.45	1.14

RMSE: root mean-squared error.

Perception models of music emotion

Realizing the importance of individual uniqueness, we used individual features as the input for our MER models in this section. First, we used audio features as the input and individual perceived emotion as the ground truth, and the aforementioned five algorithms were compared for modeling. By contrast, 10-fold cross-validation was used to evaluate the performance here. In Table 6, models using BR performed best and achieved a mean RMSE value of 0.94 for perceived happy, 1.04 for perceived relaxed, and 0.98 for perceived sad.

Table 6.

Mean RMSE values of five ML models (1,200 samples).

	MLR	BR	SVR	RF	KNN
Happy	0.94	0.94	1.10	0.95	0.96
Relaxed	1.04	1.04	1.11	1.04	1.06
Sad	0.98	0.98	1.12	0.98	1.00
Mean	0.99	0.99	1.11	0.99	1.01

RMSE: root mean-squared error; MLR: multiple linear regression; SVR: support vector regression; KNN: k-nearest neighbor regression.

Bold values are the RMSE values of the best performing algorithm.

Next, the individual features were gradually added to the basic model as inputs. We divided individual features into two categories. The first category was stable individual features, and comprised demographics, music experience, and personality. The second category was dynamic features, including felt emotion, preference, familiarity and mechanism indices. We constructed four models with different inputs for each prototypical emotion. The first type of model, already built, was a base model that used only audio features. The second type of model added stable individual features as the input on the basis of the first type. Similarly, the third model added subjective feelings. The fourth type of model added all individual features as the input.

Figure 4 shows the prediction results of the four types of models for individual perceived emotions. When only stable individual features were added, the model effect for the three prototypical emotions did not improve significantly. However, when adding subjective feelings, the mean RMSE value of the perception of happy decreased from 0.94 to 0.82, relaxed decreased from 1.04 to 0.93, and sad decreased from 0.98 to 0.85. The results demonstrated that when predicting individual perceived emotions, the stable individual features have a relatively weak effect on the prediction results, and the dynamic individual features (subjective feelings in this study) improve the prediction results.

Figure 4.

Mean RMSE values of perception models with different inputs.

Comparison between feeling models and perception models

The next step was to build the third type of model to predict felt emotions and observe how individual features contribute to prediction. In addition, we also had to verify Hypothesis 3 by comparing the role of individual features in predicting perceived emotions and felt emotions.

First, we used audio features as the input and individual felt emotion as the ground truth and applied BR to construct our basic models. Ten-fold cross-validation was also used to evaluate the performance of our models. The basic feeling models achieved a mean RMSE value of 1.03 for felt happy, 1.09 for felt relaxed, and 0.88 for felt sad.

Next, we gradually added individual features as the input for the basic model. Stable individual features were still the first type of individual features. However, we divided the subjective feelings into two parts. Because the felt emotion in subjective feelings became the ground truth, the remainder of the subjective feelings (i.e., mechanism indices, familiarity, and preference) was considered a separate category of individual features. In addition, the perceived emotion was regarded as the third type of individual feature. We added the aforementioned three types of features to the basic model and compared the effects of the models.

Figure 5 shows the mean RMSE value of the models. Similar to the perception models, when only stable individual features were added, the model effect was slightly worse. Subjective feelings achieved good results when predicting happiness and relaxation, which improved the model’s effect by 13.11% and 3.42%, but the effect of model improvement remained small when predicting sadness.

Figure 5.

Mean RMSE values of feeling models with different inputs.

To compare the role of individual features in the perception models and feeling models, we also classified subjective feelings into two categories (a. mechanism indices, familiarity, and preference, and b. felt emotions), and then compared the effects of the three types of individual features on the perception model. Table 7 shows the mean RMSE value of the perception models and the improvement rate of the current model compared with the basic model. We found that, in general, individual features played a greater role in feeling models than in perception models.

Table 7.

RMSE values and improvement rate of perception models with different inputs.

Perceived emotions	AF	AF and SIF		AF and SF		AF and FE		AF, SIF, SF, and FE
Perceived emotions	RMSE	RMSE	IR	RMSE	IR	RMSE	IR	RMSE	IR
Happy	0.94	1.00	−6.67%	0.91	3.00%	0.82	12.61%	0.86	8.63%
Relaxed	1.04	1.08	−4.11%	1.04	−0.03%	0.92	10.99%	0.95	8.24%
Sad	0.98	1.05	−7.71%	0.98	0.11%	0.84	13.79%	0.89	8.92%

AF: audio features; SIF: stable individual factors; SF: subjective feelings (except for perceived emotions); FE: felt emotions; IR: the improvement rate of the current model compared with the model using audio features only; RMSE: root mean-squared error.

Model interpretability

In the last step of data exploration, we assessed if the models were able to differentiate between perceived emotions or felt emotions, by examining the information gain of features. The final models were constructed by BR; thus, they can be interpreted by the value of the coefficients. Because 10-fold cross-validation was used to evaluate the performance of our models, the coefficients of features might differ when predicting each test set; to show this, we plotted the distribution of the coefficients for each feature using boxplots (Figures 6 and 7). Boxplots were arranged in descending order of the absolute value of the median, and only the top 30 features were included for visibility, and the trend of the remaining features was approximately the same. All features applied to the modeling are listed in Supplemental material A.

Figure 6.

Distribution of coefficients of each feature for recognition models of perceived emotion.

Figure 7.

Distribution of coefficients of each feature for recognition models of felt emotion.

A compact boxplot indicates that the feature has a similar value when predicting different test sets. In Figure 6, for perceived emotions, the corresponding felt emotion was the most important feature. The remaining features had smoothly decreasing distributions. For felt emotions (Figure 7), the corresponding perceived emotion was also important. In addition, for felt happiness, the preference for target music became the second important feature. This feature was also important in predicting felt relaxation. We also observed that for the perception models, the value of the audio features accounted for 73.81% of happy, 64.88% of relaxed, and 68.54% of sad. For feeling models, the value of the audio features accounted for 51.52% of happy, 46.34% of relaxed, and 53.64% of sad. These findings demonstrated that individual features can provide more information for feeling models than for perception models.

Discussion

In this study, we revisited the influence of individual factors on the perception of music emotion and the emotion evoked by music. By using ML methods, we applied the individual variables verified in psychological research to the recognition model of individual music emotion perception and feelings. This process resulted in a re-understanding of the relationship between individual factors and music emotion judgment through a computational modeling perspective. In addition, our work is of practical value, because the models constructed in this study may improve the traditional MER system and may indirectly contribute to the personalized MIR and music recommendation systems.

We first compared the predictive effect of individual perceived emotions (personal MER model) and average opinions of perceived emotions (general MER model) to investigate whether we must consider individual differences. The results support the argument that the MER models, using audio features as the only input, are less effective in predicting individual emotions. This finding further proves that individuality plays a critical role in making an MER system effective in practice.

Next, we investigated the role of individual factors in perceived emotion recognition by gradually adding individual features to the basic MER models based on audio features. After adding stable individual features, all the models performed worse, and this showed that stable individual features were not effective information for our models. By contrast, the increase in redundant information reduced the effect. Similar to the conclusion of Y. H. Yang, Su, et al. (2007), that is, individuality is too subtle to be captured by each single individual factor, the combination of several stable individual factors considered in our experiment is also insufficient. It is necessary to exploit other individual factors, such as absorption (Herbert, 2013; Kreutz, Ott, Teichmann, Osawa, & Vaitl, 2008), rumination (Garrido & Schubert, 2013), and so forth, to adequately describe individuality. Compared with stable individual features, adding real-time features significantly improved the effectiveness of our models. This means an individual’s perception of music emotion is affected by the state of the individual after receiving music stimulation. However, from a practical point of view, real-time features, currently, are difficult to widely use in the personalized MER model because they are difficult to obtain.

We also constructed recognition models for felt emotions and assessed the role of individual factors. We found that the positive effect of individual features on the feeling models is higher than on the perception models. Because perceived emotion and felt emotion are the “objective” and “subjective” aspects of music-elicited emotion (Gabrielsson, 2001), from a theoretical perspective, we suspect that emotions evoked by music are more susceptible to the individual’s own state than the objective judgment of the emotions expressed by music. Some studies reported the relationship between individual variables and the gap across perceived and felt emotion (e.g., Kallinen & Ravaja, 2006), which indirectly support the above assumption, but we still need further research. From a computational modeling perspective, this result illustrates the close relationship between individual factors and the felt emotion of music. Compared to using only music features for recognition (e.g., Koelstra et al., 2011), this result is a reminder that we should consider individual factors to enhance model’s effect in the process of modeling the recognition model of music emotion.

In addition, we observed that the features that are more relevant to the prediction target in correlation analysis often contribute more to the prediction models. This reminds us that the statistical results of experimental studies can also provide a reference for the selection of input features in modeling applications. Finally, we used the value of the coefficients to interpret our final models, and this allowed us to observe the importance of each feature in the model. We observed that, both for perceived emotions and felt emotions, the corresponding felt emotion or perceived emotion was the most important feature. And individual features can provide more information for feeling models. This result further illustrates the importance of individual features in the prediction of felt emotion.

Our work is also of practical value. As mentioned earlier, music emotion information has been widely used in various fields (e.g., MIR; Downie, 2008; Wang et al., 2012). And MER, a research area investigating computation models to automatically recognize the emotion of music, is an important part of the research on digital music applications (Dobashi et al., 2015). In this study, we improved the effect of individual MER models by adding individual features as model inputs. Therefore, our work provided a reference for the creation of personalized MER systems, which can provide individuals with more accurate emotion information of music.

This study has several notable limitations. First, the audio features that considered input in our models were not studied in depth in the modeling process, and they may only be a subset of the full profile of features actually processed by listeners. In further modeling research, feature selection and processing methods must be consistently updated to achieve optimal results. Second, our modeling attempts were handicapped by the size of our dataset. However, participants have difficulty annotating a large number of audio recordings, and the quality of annotation may also decline because of fatigue or other reasons. Thus, a better means to collect the labeling results is necessary. Third, the combination of several stable individual factors was not of great help to MER, and we suggested the exploitation of other individual factors to adequately describe individuality. From the perspective of theoretical exploration, follow-up work is of great significance. However, from a practical point of view, discovering and collecting a large number of stable individual features is not necessarily easier than collecting individual real-time features. Therefore, in the modeling processes, trade-offs must continue to be made between the two aforementioned methods. Finally, when choosing the ML method, we only referred to the prediction effect of the models, without considering the interpretability. Fortunately, BR, which has good interpretability, performed best in this study. Notably, most of the models developed with flexible methods tend to be powerful in terms of fitting the training data, but the flexibility of methods is opposite to the interpretability of methods (Hastie, Tibshirani, Friedman, & Franklin, 2005). Therefore, when interpreting mechanism exploration, we suggest adopting more interpretable methods. And when pursuing model effects, more flexible methods, such as deep neural networks (Orjesek, Jarina, Chmulik, & Kuba, 2019; Zhou, Chen, & Yang, 2019), convolutional neural networks (Malik et al., 2017), recurrent neural networks (Liu, Fang, & Huang, 2019; Malik et al., 2017), and so forth (e.g., Dong, Yang, Zhao, & Li, 2019) can be used.

Conclusion

In this study, we used ML methods to form relations among audio features, individual factors, and music emotions. We observed that (a) the MER models, using audio features as input only, are less effective in predicting individual emotions than the mean opinions; (b) stable individual features have no positive effects on MER models, but real-time individual features can significantly improve the model’s effect; and (c) the individual features have greater effects on feeling models than those on perception models. Our work investigated the relationship between variables, constructed models with practical value, and provided a reference for the creation of personalized MER systems.

Supplemental Material

Supplemental_material – Supplemental material for Effects of individual factors on perceived emotion and felt emotion of music: Based on machine learning methods

Supplemental material, Supplemental_material for Effects of individual factors on perceived emotion and felt emotion of music: Based on machine learning methods by Liang Xu, Xin Wen, Jiaming Shi, Shutong Li, Yuhan Xiao, Qun Wan and Xiuying Qian in Psychology of Music

Footnotes

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Xiuying Qian

Supplemental material

Supplemental material for this article is available online.

References

Allen

Walsh

Zangwill

(2013). The same, only different: What can responses to music in autism tell us about the nature of musical emotions? Frontiers in Psychology, 4, Article 156.

Bernatzky

Presch

Anderson

Panksepp

(2011). Emotional foundations of music as a non-pharmacological pain management tool in modern medicine. Neuroscience & Biobehavioral Reviews, 35, 1989–1999.

Bhatti

A. M.

Majid

Anwar

S. M.

Khan

(2016). Human emotion recognition and analysis in response to audio music using brain signals. Computers in Human Behavior, 65, 267–275.

Byun

S. W.

Lee

S. P.

Han

H. S.

(2017). Feature selection and comparison for the emotion recognition according to music listening. In 2017 International Conference on Robotics and Automation Sciences (ICRAS) (pp. 172–176). New York, NY: IEEE.

Carciofo

Yang

Song

Zhang

(2016). Psychometric evaluation of Chinese-language 44-item and 10-item big five personality inventories, including correlations with chronotype, mindfulness and mind wandering. PLoS ONE, 11(2), Article e0149963.

Chai

Vercoe

(2000, October). Using user models in music information retrieval systems. In ISMIR. Retrieved from https://pdfs.semanticscholar.org/48dd/598b5b43181f430fd983a66eec689cb9ab9e.pdf

Chang

C. Y.

Wang

C. J.

Chung

P. C.

(2010). A music recommendation system with consideration of personal emotion. In 2010 International Computer Symposium ICS2010 (ICS2010) (pp. 18–23). New York, NY: IEEE.

Chen

S. H.

Lee

Y. S.

Hsieh

W. C.

Wang

J. C.

(2015). Music emotion recognition using deep Gaussian process. In 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) (pp. 495–498). New York, NY: IEEE.

Deng

J. J.

Leung

C. H.

Milani

Chen

(2015). Emotional states associated with music: Classification, prediction of changes, and consideration in recommendation. ACM Transactions on Interactive Intelligent Systems, 5(1), Article 4.

10.

Dewi

K. C.

Harjoko

(2010). Kid’s song classification based on mood parameters using K-Nearest Neighbor classification method and Self Organizing Map. In 2010 International Conference on Distributed Frameworks for Multimedia Applications (pp. 1–5). New York, NY: IEEE.

11.

Dingle

G. A.

Kelly

P. J.

Flynn

L. M.

Baker

F. A.

(2015). The influence of music on emotions and cravings in clients in addiction treatment: A study of two clinical samples. The Arts in Psychotherapy, 45, 18–25.

12.

Dobashi

Ikemiya

Itoyama

Yoshii

(2015). A music performance assistance system based on vocal, harmonic, and percussive source separation and content visualization for music audio signals. In Proceedings of the 12th Sound & Music Computing (SMC) Conference (pp. 99–104).

13.

Dolgin

K. G.

Adelson

E. H.

(1990). Age changes in the ability to interpret affect in sung and instrumentally-presented melodies. Psychology of Music, 18(1), 87–98.

14.

Dong

Yang

Zhao

(2019). Bidirectional Convolutional Recurrent Sparse Network (BCRSN): An efficient model for Music Emotion Recognition. IEEE Transactions on Multimedia, 21, 3150–3163.

15.

Downie

J. S.

(2008). The music information retrieval evaluation exchange (2005–2007): A window into music information retrieval research. Acoustical Science and Technology, 29(4), 247–255.

16.

Feng

Zhuang

Pan

(2003). Popular music retrieval by detecting mood. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 375–376). New York, NY: ACM.

17.

Ting

K. M.

Zhang

(2010). A survey of audio-based music classification and annotation. IEEE Transactions on Multimedia, 13(2), 303–319.

18.

Gabrielsson

(2001). Emotion perceived and emotion felt: Same or different? Musicae Scientiae, 5(1_suppl), 123–147.

19.

Garrido

Schubert

(2013). Adaptive and maladaptive attraction to negative emotions in music. Musicae Scientiae, 17(2), 147–166.

20.

Gouizi

Bereksi Reguig

Maaoui

(2011). Emotion recognition from physiological signals. Journal of Medical Engineering & Technology, 35(6–7), 300–307.

21.

Han

B. J.

Rho

Dannenberg

R. B.

Hwang

(2009). SMERS: Music Emotion Recognition using Support Vector Regression. In ISMIR (pp. 651–656). Retrieved from https://www.cs.cmu.edu/~rbd/papers/emotion-ismir-09.pdf

22.

Harte

Sandler

Gasser

(2006). Detecting harmonic change in musical audio. In Proceedings of the 1st ACM Workshop on Audio and Music Computing Multimedia (pp. 21–26). New York, NY: ACM.

23.

Hastie

Tibshirani

Friedman

Franklin

(2005). The elements of statistical learning: Data mining, inference and prediction. The Mathematical Intelligencer, 27(2), 83–85.

24.

Herbert

(2013). An empirical study of normative dissociation in musical and non-musical everyday life experiences. Psychology of Music, 41(3), 372–394.

25.

Imbrasaitė

Baltrušaitis

Robinson

(2013). Emotion tracking in music using continuous conditional random fields and relative feature representation. In 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW) (pp. 1–6). New York, NY: IEEE.

26.

Jenke

Peer

Buss

(2013, May). Effect-size-based electrode and feature selection for emotion recognition from EEG. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 1217–1221). New York, NY: IEEE.

27.

Jiang

D. N.

Zhang

H. J.

Tao

J. H.

Cai

L. H.

(2002). Music type classification by spectral contrast feature. In IEEE International Conference on Multimedia and Expo (Vol. 1, pp. 113–116). New York, NY: IEEE.

28.

Juslin

P. N.

Harmat

Eerola

(2014). What makes music emotionally significant? Exploring the underlying mechanisms. Psychology of Music, 42(4), 599–623.

29.

Juslin

P. N.

Zentner

M. R.

(2001). Current trends in the study of music and emotion: Overture. Musicae Scientiae, 5(1_suppl), 3–21.

30.

Kallinen

Ravaja

(2006). Emotion perceived and emotion felt: Same and different. Musicae Scientiae, 10(2), 191–213.

31.

Kawakami

Katahira

(2015). Influence of trait empathy on the emotion evoked by sad music and on the preference for it. Frontiers in Psychology, 6, Article 1541.

32.

Kim

André

(2008). Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(12), 2067–2083.

33.

Kim

Y. E.

Schmidt

E. M.

Migneco

Morton

B. G.

Richardson

Scott

. . . Turnbull

(2010). Music emotion recognition: A state of the art review. In Proceedings of the 11th annual conference of the International Society for Music Information Retrieval (ISMIR) (Vol. 86, pp. 937–952).

34.

Koelstra

Muhl

Soleymani

Lee

J. S.

Yazdani

Ebrahimi

. . .Patras

(2011). Deap: A database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing, 3(1), 18–31.

35.

Kratus

(1993). A developmental study of children’s interpretation of emotion in music. Psychology of Music, 21(1), 3–19.

36.

Kreutz

Ott

Teichmann

Osawa

Vaitl

(2008). Using music to induce emotions: Influences of musical preference and absorption. Psychology of Music, 36(1), 101–126.

37.

Laukka

(2007). Uses of music and psychological well-being among the elderly. Journal of Happiness Studies, 8(2), 215–241.

38.

Laurier

Grivolla

Herrera

(2008). Multimodal music mood classification using audio and lyrics. In 2008 Seventh International Conference on Machine Learning and Applications (pp. 688–693). New York, NY: IEEE.

39.

Chen

J. H.

(2006). Emotion recognition using physiological signals. In Pan

Cheok

Haller

Lau

R.W.H

Saito

Liang

(Eds.), International Conference on Artificial Reality and Telexistence (pp. 437–446). Berlin, Germany: Springer.

40.

Liu

Fang

Huang

(2019). Music emotion recognition using a variant of recurrent neural network. In 2018 International Conference on Mathematics, Modeling, Simulation and Statistics Application (MMSSA 2018). Paris, France: Atlantis Press.

41.

Liu

Zhang

H. J.

(2005). Automatic mood detection and tracking of music audio signals. IEEE Transactions on Audio, Speech, and Language Processing, 14(1), 5–18.

42.

Malik

Adavanne

Drossos

Virtanen

Ticha

Jarina

(2017). Stacked convolutional and recurrent neural networks for music emotion recognition. arXiv preprint arXiv:1706.02292.

43.

McFee

Raffel

Liang

Ellis

D. P.

McVicar

Battenberg

Nieto

(2015, July). librosa: Audio and music signal analysis in python. In Proceedings of the 14th Python in Science Conference (Vol. 8). Retrieved from https://conference.scipy.org/proceedings/scipy2015/pdfs/brian_mcfee.pdf

44.

Mualem

Lavidor

(2015). Music education intervention improves vocal emotion recognition. International Journal of Music Education, 33(4), 413–425.

45.

Orjesek

Jarina

Chmulik

Kuba

(2019). DNN-Based music emotion recognition from raw audio signal. In 2019 29th International Conference Radioelektronika (pp. 1–4). New York, NY: IEEE.

46.

Pao

T. L.

Cheng

Y. M.

Yeh

J. H.

Chen

Y. T.

Pai

C. Y.

Tsai

Y. W.

(2008). Comparison between weighted d-KNN and other classifiers for music emotion recognition. In 2008 3rd International Conference on Innovative Computing Information and Control (p. 530). New York, NY: IEEE.

47.

Park

S. H.

Ihm

S. Y.

Jang

W. I.

Nasridinov

Park

Y. H.

(2015). A music recommendation method with emotion recognition using ranked attributes. In Stojmenović

Jeong

H.-Y.

(Eds.), Computer science and its applications (pp. 1065–1070). Berlin, Germany: Springer.

48.

Pedregosa

Varoquaux

Gramfort

Michel

Thirion

Grisel

. . . Vanderplas

(2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.

49.

Raykar

V. C.

Zhao

L. H.

Valadez

G. H.

Florin

Bogoni

Moy

(2010). Learning from crowds. Journal of Machine Learning Research, 11, 1297–1322.

50.

Soleymani

Caro

M. N.

Schmidt

E. M.

Sha

C. Y.

Yang

Y. H.

(2013). 1000 songs for emotional analysis of music. In Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia (pp. 1–6). New York, NY: ACM.

51.

Tzanetakis

Cook

(2002). Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10(5), 293–302.

52.

Vempala

N. N.

Russo

F. A.

(2018). Modeling music emotion judgments using machine learning methods. Frontiers in Psychology, 8, Article 2239.

53.

Wang

J. C.

Yang

Y. H.

Wang

H. M.

Jeng

S. K.

(2012). The acoustic emotion Gaussians model for emotion-based music annotation and retrieval. In Proceedings of the 20th ACM International Conference on Multimedia (pp. 89–98). New York, NY: ACM.

54.

Wieczorkowska

Synak

Raś

Z. W.

(2006). Multi-label classification of emotions in music. In Kłopotek

M.A.

Wierzchoń

S.T.

Trojanowski

(Eds.), Intelligent information processing and web mining (pp. 307–315). Berlin, Germany: Springer.

55.

Yang

Dong

(2018). Review of data features-based music emotion recognition methods. Multimedia Systems, 24(4), 365–389.

56.

Yang

Y. H.

Chen

H. H.

(2010). Ranking-based emotion recognition for music organization and retrieval. IEEE Transactions on Audio, Speech, and Language Processing, 19(4), 762–774.

57.

Yang

Y. H.

Lin

Y. C.

Chen

(2009). Personalized music emotion recognition. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 748–749). New York, NY: ACM.

58.

Yang

Y. H.

Lin

Y. C.

Y. F.

Chen

H. H.

(2007). Music emotion classification: A regression approach. In 2007 IEEE International Conference on Multimedia and Expo (pp. 208–211). New York, NY: IEEE.

59.

Yang

Y. H.

Y. F.

Lin

Y. C.

Chen

H. H.

(2007). Music emotion recognition: The role of individuality. In Proceedings of the International Workshop on Human-Centered Multimedia (pp. 13–22). New York, NY: ACM.

60.

Zentner

Grandjean

Scherer

K. R.

(2008). Emotions evoked by the sound of music: Characterization, classification, and measurement. Emotion, 8(4), 494–521.

61.

Zhou

Chen

Yang

(2019). Multimodel music emotion recognition using unsupervised deep neural networks. In Li

Shao

(Eds.), Proceedings of the 6th Conference on Sound and Music Technology (CSMT) (pp. 27–39). Singapore: Springer.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB