Image emotion distribution learning based on enhanced fuzzy KNN algorithm with sparse learning

Abstract

With the trend of people expressing opinions and emotions via images online, increasing attention has been paid to affective analysis of visual content. Traditional image affective analysis mainly focuses on single-label classification, but an image usually evokes multiple emotions. To this end, emotion distribution learning is proposed to describe emotions more explicitly. However, most current studies ignore the ambiguity included in emotions and the elusive correlations with complex visual features. Considering that emotions evoked by images are delivered through various visual features, and each feature in the image may have multiple emotion attributes, this paper develops a novel model that extracts multiple features and proposes an enhanced fuzzy k-nearest neighbor (EFKNN) to calculate the fuzzy emotional memberships. Specifically, the multiple visual features are converted into fuzzy emotional memberships of each feature belonging to emotion classes, which can be regarded as an intermediate representation to bridge the affective gap. Then, the fuzzy emotional memberships are fed into a fully connected neural network to learn the relationships between the fuzzy memberships and image emotion distributions. To obtain the fuzzy memberships of test images, a novel sparse learning method is introduced by learning the combination coefficients of test images and training images. Extensive experimental results on several datasets verify the superiority of our proposed approach for emotion distribution learning of images.

Keywords

Image emotion recognition emotion distribution learning fuzzy emotional membership sparse learning

1 Introduction

With the prevalence of social networks, people prefer to record their daily lives and express their opinions via visual content, such as images and videos [1]. As images are one of the most intuitive and convenient mediums, image emotion recognition has attracted increasing research interests. The emotions people perceive from images may directly determine their decision and influence their visual preference. Therefore, practical analysis of images at an affective level can help understand the users’ behaviors and attitudes, which benefits a wide range of applications, such as advertising [2], opinion mining [3], depression detection [4].

However, compared to traditional visual tasks, image emotion recognition is inherently challenging because of the subjectivity perception problem. The same image may invoke different emotions for viewers with diverse social and cultural backgrounds [5]. Furthermore, even one viewer may have multiple feelings toward an image. Recent studies mainly paid attention to image emotion classification based on the supposition that emotions perceived by viewers can reach a consensus [6]. A certain label is unable to describe the affective content of the image accurately. Thus, emotion distribution [7, 8] has been proposed to capture the latent emotions of images rather than pure classification. By assigning a given number of labels, the emotion distribution denotes the degree to which each label describes the image emotions. Figure 1 shows two samples from different datasets and their distribution labels, which indicates that there exists ambiguity among emotions. Hence, this work considers image emotion recognition as an emotion distribution prediction task instead of a classification task.

Fig. 1

Affective images with their emotion distributions from different datasets.

Similar to other computer vision tasks, the semantic gap between low-level features and high-level semantics still exists in image emotion recognition. There is a major obstacle that how to express subjective emotion with objective visual information. For image emotion recognition, the semantic gap extends to the affective gap [9]. Some studies have investigated various approaches to address this challenge. In the early stages, hand-crafted features were designed based on the observation of people perceiving the emotion of an image and related to image aesthetics [6 , 11]. With the rise of deep convolutional neural networks (CNN), recent image emotion recognition methods have shifted to learn the discriminative features of an image [12], which shows the outstanding potential of robust features.

In fact, the deep features are not sufficient for emotion recognition because emotions and semantics may also be delivered by low-level and mid-level visual features [13]. Thus, several attempts have been made to probe more discriminative representations for image emotion recognition by jointly combing multiple features through the fusion strategies [14, 15]. But these studies focus on mapping visual features to emotions directly, which is difficult for people to comprehend how to make decisions. Others prefer to learn the affective-related visual concepts [16, 17] to bridge the affective gap. However, the visual concepts should be pre-defined, which may fail to handle complex scenes because one object may convey different emotions in different contexts. Additionally, they ignore that each feature in the image may have multiple emotional attributes. For example, the color red can not only evoke feelings of excitement but also stimulate violence. As the emotions are naturally contained with ambiguous and imprecision, there are elusive connections between visual features and emotions. The discriminative learning methods struggle to deal with such fuzziness. Based on this observation, this work introduces the fuzzy approach to bridge the affective gap for image emotion recognition by generating the fuzzy emotional membership of visual features to emotional terms. In this way, the relationships between various visual features and emotions of an image can be quantified. To achieve this, we propose an enhanced fuzzy k-nearest neighbor algorithm (EFKNN) to convert the visual features into fuzzy emotional memberships from the perspective of affective semantic, which is coming under a supervised learning approach and deterministic form of k-nearest neighbor with the benefit of fuzzy set theory.

In this paper, we develop a novel emotion distribution learning framework for image emotion recognition. Various types of visual features are extracted to explore the relations between the multiple visual features and emotions. The EFKNN algorithm is proposed to calculate the fuzzy emotional memberships of these visual features. The values of the fuzzy emotional memberships represent the correlations between each feature extracted from training images and each emotion class, which are regarded as the intermediate representations to narrow the affective gap. Then, the fuzzy emotional memberships are taken as the input of a fully connected neural network to learn the relationship between fuzzy emotional memberships and the labeled emotion distributions. For an unlabeled image, it cannot obtain the fuzzy emotional memberships directly because of the affective gap. Hence, sparse learning is utilized to obtain the fuzzy emotional memberships of test images, which constructs the visual features of test images with the visual features of the training images by learning the combination coefficients. It is based on the hypothesis that the fuzzy emotional memberships of an unlabeled image can be approximately modeled as a linear combination of the fuzzy emotional memberships of the training images. The final emotion distribution of a test image is predicted through the trained fully connected neural network.

The main contributions of this paper are summarized as follows:

We propose a new strategy for emotion distribution prediction. This method transforms the visual features into fuzzy emotional memberships to bridge the affective gap. To our best knowledge, it is the first time to introduce fuzzy memberships to explore the complex connections between visual features and emotions from an interpretable perspective.

In order to capture the ambiguity and uncertainty of visual features and emotion classes, an EFKNN algorithm based on semantics is proposed to calculate the fuzzy emotional memberships of multiple features to emotions.

The sparse learning algorithm based on fuzzy emotional memberships is proposed for emotion distribution learning, which learns the shared parameters by reconstructing the features of training images and testing images. And the fuzzy emotional memberships of the test image are calculated by linearly combining the fuzzy memberships of the training images.

Experiments performed on Emotion6 [5], Abstract [6] and IESN [18, 19] datasets verify the effectiveness of our proposed method.

The rest of this paper is organized as follows. Related works are reviewed in Section 2. The proposed approach for image emotion distribution learning is described in Section 3. Experimental settings, results and analysis are elaborated in Section 4. And the conclusion and future work are followed in Section 6.

2 Related works

This section briefly reviews existing works on image emotion recognition, the background of the fuzzy approach and sparse learning.

2.1 Image emotion recognition

With the development of computer vision, image emotion recognition has become a meaningful research topic in recent years. The solutions mainly fall into two types of emotion representation approaches: the dimensional model that projects emotions into a dimension emotion space [7, 20] and categorical model that classifies emotions into several basic categories [6, 21]. The categorical models are obvious for common people understanding and thus have been mainly employed by previous studies. As emotion is a highly subjective and complex variable, existing studies which take image emotion recognition as a classification task have neglected explicit difference on emotion intensity. Therefore, label distribution learning algorithms [22] were proposed to assign a descriptive degree value to each category. In such a case, it is more reasonable to predict the emotion distribution of images. Consequently, Peng et al. [5] constructed a CNN regression that utilized a deep CNN with the Euclidean loss to train the regressions for each emotion, and the regression results were transformed to emotion distribution through normalization. Based on the condition probability neural network (CPNN) [23], binary conditional probability neural network (BCPNN) and augmented conditional probability neural network (ACPNN) [24] were developed for emotion distribution prediction. Zhao et al. [8, 25] proposed shared sparse learning (SSL) to transfer the features to emotion distribution by learning the mapping factors. Considering the co-occurrence among emotion labels, graph convolutional networks [26] and structural learning framework [27] were both applied to model these correlations among labels for emotion distribution prediction.

Learning a discriminative feature representation is crucial to image emotion recognition. Early researchers on image emotion recognition concentrated on designing hand-crafted features, such as holistic features [11], color and texture [6], principles-of-art features [10], attributes [28] and Adjective Noun Pairs (ANPs) [16]. Recently, deep CNN-based features have been widely used in image emotion recognition [12, 17]. Because of the complexity of emotions, researchers gradually prefer to integrate multiple features for emotion recognition [14, 29]. However, the correlations between the features and the emotions have not been explored well. The existing methods stated above only map the features to labels directly, and people cannot comprehend why such features induce a particular emotion. To tackle this issue, we combine multiple features for image emotion recognition with the fuzzy approach to probe the relationships between multiple visual features and emotions, which can improve the interpretability of image emotion recognition and quantify the connections of multiple features and emotions.

2.2 Applications of fuzzy approach

Fuzzy approach, which is based on fuzzy logic [30], is aimed to deal with uncertain data. In this context, each instance is typically not crisp and thus belongs to different classes to different degrees. In recent years, fuzzy approaches have been preferred for a variety of applications, such as text classification [31], medical diagnosis [32] and decision-making [33]. As the affective analysis is permeated with uncertainty, the fuzzy approach is more appropriate to compute such fuzziness that includes in images and emotions. Li et al. [34] developed the 3D fuzzy visual features by fuzzy c-means (FCM) to construct the emotional feature space for analyzing the semantic of images. The fuzzy set theory also has been applied to learn the automatic semantic annotation [35] and emotion understanding [36]. But most of them just fuzzify the features or constructed the fuzzy classifiers for the final emotion classification. Besides, researchers attempted to exploit fuzzy clustering for visual content analysis [37, 38]. It is an unsupervised manner for fuzzy classification, hence, it cannot overcome the affective gap. The aforementioned researches only solved the ambiguity of emotion to a certain extent, which is unable to bridge the affective gap at the same time.

The fuzzy k-nearest neighbor algorithm (FKNN) [39] is one of the successful algorithms that performs the classification by adding fuzzy logic into the standard KNN algorithm [40]. It allocates the degree of fuzzy membership to each class while considering the distance of its k-nearest neighbors. It has been applied to image classification [41] and retrieval [42]. With the benefit of its computational simplicity and conceptual, as well as the supervised manner for fuzzy classification, this work proposes an EFKNN algorithm based on FKNN to tackle the fuzziness of visual features and emotions. The proposed EFKNN can not only reduce the affective gap by the fuzzy emotional memberships but also tackle the black box non-interpretability of previous methods.

2.3 Sparse learning

The main idea of sparse learning is that many important signals enable to be approximately represented as a linear combination of basic functions [43]. Sparsity has been exploited in computer vision to retrieve the sparse representation of data with regard to a dictionary or a series of bases [44, 45]. Wright et al. [46] have demonstrated that it is able to sparsely represent images from the perspective of the attributes of visual neurons. Zhao et al. integrated sparse learning to learn the continuous probability distribution [7] and the discrete probability distribution [47] of image emotions. In this paper, we also leverage the sparse representation method to acquire the fuzzy emotional memberships of unlabeled images. The novelty of our model lies in that the visual features are transferred to fuzzy membership vectors to be the intermediate representation instead of learning emotion distribution by sparse learning directly.

3 Methodology

In this section, we introduce the proposed model in detail. The goal of this work is to learn the emotion distribution of images. For an affective image, multiple features are extracted to probe the correlations between features and emotions. Suppose there are N training images I ={ I₁, I₂, …, I_N } and L emotion categories c₁, c₂, …, c_L. Given an affective image I_n (n = 1, 2, …, N), we extract J types of features and the multiple features representation of the image I_n is $I_{n} = {X_{n 1}, X_{n 2}, \dots, X_{nj}}_{j = 1}^{J}$ , where X _nj denotes the jth feature of image I_n, and the dimension of the feature is d_j. Assuming that p _n = [p_n1, p_n2, …, p_nl] ^T (l = 1, 2, …, L) denotes the emotion distribution of the image I_n. p_nl is the probability of image I_n delivers emotion c_l and satisfies the constrains $\sum_{l = 1}^{L} p_{nl} = 1$ . Let X _j ={ X _1j, X _2j, …, X _nj, …, X _Nj } represent the jth features of the training images. Suppose we have M test images, and the jth features of the test images are $Y_{j} = {Y_{1 j}, Y_{2 j}, \dots, Y_{mj}, \dots Y_{Mj}}_{j = 1}^{J}$ . The task of this paper is to learn the emotion distribution of the test image I_m, which is represented as p _m = [p_m1, p_m2, …, p_ml] ^T (m = 1, 2, …, M).

3.1 Overall framework

As illustrated in Fig. 2, our proposed framework consists of four parts: a) multiple features extraction, b) fuzzy emotional membership calculation by the proposed EFKNN, c) sparse learning based on fuzzy emotional memberships, and d) emotion distribution prediction. Firstly, in the training phase, various kinds of features are extracted from training images to strengthen the representation ability of emotions. The dimensions of the extracted features are reduced by the principal component analysis (PCA) algorithm [48]. Secondly, an enhanced fuzzy k-nearest neighbor (EFKNN) based on semantics is proposed to calculate the fuzzy emotional memberships, which represent the degrees of each feature belonging to each emotion category. Thirdly, the obtained fuzzy emotional memberships are fed into a fully connected neural network to learn the relationship between the fuzzy emotional memberships and the labeled emotion distribution. Then, in the test phase, the sample transfer method based sparse learning algorithm is proposed to learn the correlation coefficients (which called shared factors learning) to construct the visual features of the test image with the visual features of training images. The learned coefficients are employed to gain the fuzzy memberships of the test image by the linear combination of the fuzzy emotional memberships of training images. In the end, the emotion distributions of the test images are predicted by the trained fully connected neural network taking the acquired fuzzy emotional memberships as input.

Fig. 2

The overall framework of our proposed method for image emotion distribution learning. The solid lines and dash lines indicate the training phase and test phase, respectively.

3.2 Enhanced Fuzzy K-Nearest Neighbor (EFKNN) algorithm

In this section, an extended version of the FKNN algorithm is proposed, called Enhanced FKNN (EFKNN). The proposed EFKNN overcomes the drawback of the original FKNN, which is incapable to compute the fuzzy memberships of the visual features from the perspective of affective semantics. In image emotion recognition task, it often appears that the low-level features of the two images are very similar, but the evoked emotions are quite different. Even though the FKNN can manage the ambiguity by assigning the probabilities (which are called memberships) of instances to a class, it generates the membership degree by the location of the instance in feature space, which is determined by the low-level features. Therefore, it is unable to narrow the affective gap through the original FKNN.

To tackle this issue, we propose the EFKNN based on semantics to calculate the fuzzy emotional memberships of visual features to emotion categories. Like the conventional FKNN algorithm, the EFKNN will consider K nearest neighbors to the vectors in the feature space. In contrast to the FKNN, the EFKNN method measures similarity as a weight factor based on the emotion distribution of images to distinguish the images with similar features but with different emotions. An illustrative example is depicted in Fig. 3, showing the procedure of the EFKNN considering three emotion categories A, B and C in a two-dimensional feature space. As shown in Fig. 3(1), we first project all items to the feature space and randomly select a set of class centers for these items. Meanwhile, the emotion distributions of these K nearest neighbors within the same class are averaged as the emotion distribution of the class centers. As the fact that the emotion distribution of an image is delivered through diverse visual features from that image, we suppose that the emotion distribution of each feature is the same as the emotion distribution of that image. Then, we measure the similarity between the emotion distribution of the feature and that of each class center as the intensity value in Fig. 3(2). The intensity values are regarded as weight factors. The more the closeness of the emotion distributions between the features and the class centers, the more important the weights. After that, the new class centers are acquired by taking the average of K nearest neighbors as shown in Fig. 3(3). When calculating the final membership degree of each feature vector belongs to emotion classes, only the distance of features to class centers are taken into account. Finally, we get the approximate real emotion distribution of each feature according to the same type of features from different images in the space as shown in Fig. 3(4).

Fig. 3

The procedure of EFKNN considering three emotion categories A, B and C of one feature in a two-dimensional space.

Given a set of feature vectors of the jth features of the training images X _j ={ X _2j, X _2j, …, X _nj, …, X _Nj }. The new center of the feature belongs to each category is acquired by means of the K neighbors, which is considered to be the new central point of this kind of emotion category and represented as v^l. The centers of each class containing K neighbors in the feature space can be accomplished as follows: $\begin{matrix} V_{j} = {v_{j}^{1}, v_{j}^{2} \dots, v_{j}^{l}, \dots, v_{j}^{L}} \end{matrix}$ (1) $\begin{matrix} v_{j}^{l} = \frac{\sum_{k = 1}^{K} μ_{lk} x_{k}}{\sum_{k = 1}^{K} μ_{lk}} \end{matrix}$ (2) where V_j is the set of emotional class centers in the feature space, $v_{j}^{l}$ denotes the center of the l emotion category of the jth feature. x_k is the kth neighbor of l emotion category, K is the number of nearest neighbors within the class. And μ_lk represents the fuzzy membership, which indicates the degree that x_k belongs to the class center $v_{j}^{l}$ . $\begin{matrix} μ_{lk} = \frac{1}{{∥ x_{k} - v_{j}^{l} ∥}^{\frac{2}{α - 1}}} \end{matrix}$ (3) where $∥ x_{k} - v_{j}^{l} ∥$ computes the distance between x_k and $v_{j}^{l}$ . Various metrics can be selected, such as Euclidean distance, Hamming distance and so on. In this study, the Euclidean distance is chosen as the measure. The parameter α is the fuzzy strength parameter used to determine the weighting degree of the distance when calculating the contribution of each neighbor to the membership degree, and α ∈ (1, ∞) (it is usually selected as α = 2).

Let $μ_{nj} = [μ_{nj}^{l}] (l = 1, 2, \dots, L)$ denote the fuzzy membership vector of the jth feature of the image I_n, which contains L elements. $μ_{nj}^{l}$ is the membership degree of the feature that belongs to l emotion category and satisfies $\sum_{l = 1}^{L} μ_{nj}^{l} = 1, μ_{nj}^{l} \in [0, 1]$ . It is formulated as follows: $\begin{matrix} μ_{nj}^{l} = \frac{w_{l} μ_{l}}{\sum_{l = 1}^{L} w_{l} μ_{l}}, l \in [1, L] \end{matrix}$ (4) where μ_l takes the closeness of a sample to the class center as its degree of affiliation to the class. w_l is the strength of x_nj considering the proximity to the class center according to semantic annotation, which can be regarded as a weighting factor that adjusts the membership degree of the features from the perspective of affective semantic. The μ_l and w_l are calculated by Eqs. (6), respectively. $\begin{matrix} μ_{l} = \frac{1}{{∥ x_{nj} - v_{j}^{l} ∥}^{\frac{2}{α - 1}}} \end{matrix}$ (5) $\begin{matrix} w_{l} = \frac{1}{W (p (x_{nj}), p (v_{j}^{l}))} \end{matrix}$ (6)

We adjust the membership degree by the weighting factor w_l to gain a reasonable emotional membership degree. Considering the symmetry between different distributions, we adopt Wasserstein distance [49] to measure the distribution similarity between the emotion distribution of the feature vectors p (x_nj) and the emotion distribution of the class center $p (v_{j}^{l})$ , denoted as $W (p (x_{nj}), p (v_{j}^{l}))$ . We assume that the emotion distribution of each feature is the same as the emotion distribution of that image initially. The emotion distribution of the class center is the average of the emotion distribution of K nearest neighbors within the same class and is formulated as follows. $\begin{matrix} p (v_{j}^{l}) = \frac{\sum_{k = 1}^{K} p (x_{k})}{K} \end{matrix}$ (7)

As depicted in Eq. (4), the fuzzy memberships of jth features of the image I_n can be collected and denoted as $μ_{nj} = [μ_{nj}^{l}] (l = 1, 2, \dots, L)$ . In like manner, the fuzzy memberships of other features x_ni∣i≠j of the image I_n can also be calculated by Eq. (4). Finally, the fuzzy memberships of all types of features from training images can be collected and represented as { μ_nj}, which will be used as input to a fully connected neural network for emotion distribution prediction. The procedure of the proposed EFKNN algorithm is summarized as Algorithm 1.

Algorithm 1:

The proposed EFKNN algorithm.
Input: A set of features from training images
${X_{j}}_{j = 1}^{J}$ , the labeled emotion distribution of
training images ${p_{n}}_{n = 1}^{N}$ , K.
Output: Fuzzy memberships { μ_nj} of each
feature belonging to each emotion classes.
1 forj ← 1 to J do
2 a set of random feature points as initial class
centers for each type features ${v_{0}^{l}}_{l = 1}^{L}$ .
3 forl ← 1 to L do
4 compute the initial distance and weight
factor from x_nj to $v_{0}^{l}$ :
$μ_{l} \leftarrow \frac{1}{∥ x_{nj} - v_{0}^{l} ∥^{\frac{2}{α - 1}}}$ , $w_{l} \leftarrow \frac{1}{W (p (x_{nj}), p (v_{0}^{l}))}$ ;
5 get the new class centers and emotion
distributions of centers:
$v_{j}^{l} \leftarrow \frac{\sum_{k = 1}^{K} μ_{lk} x_{k}}{\sum_{k = 1}^{K} μ_{lk}}$ , $p (v_{j}^{l}) = \frac{\sum_{k = 1}^{K} p (x_{k})}{K}$ ;
6 calculate the fuzzy membership $μ_{nj}^{l}$ of
the jth feature belonging to l emotion
class using Eq. (4);
7 end
8 get the collections of the fuzzy membership
for all features { μ_nj}.
9 end

Figure 4 shows two sample images and their fuzzy emotional memberships computed by the proposed EFKNN. Multiple types of features extracted from an image can be associated with multiple emotions, each of which with a different degree of fuzzy membership.

Fig. 4

Sample images from Emotion6 dataset and their fuzzy emotional memberships calculated by EFKNN. Six types of visual features extracted from an image are associated with emotion categories, each of which with a degree of fuzzy membership.

3.3 Sparse learning based on fuzzy emotional memberships

Based on the proposed EFKNN algorithm, the fuzzy emotional memberships of multiple features are calculated from the labeled images. However, it fails to directly acquire the fuzzy emotional memberships for unlabeled test images because of the affective gap. In view of the fact that an image is composed of different visual features, we assume that each feature of the image takes on a certain emotional membership degree and the emotional membership degrees between the features from different images have high similarity. This similarity makes the expression of similar features sparse. Thus, we introduce a sparse learning method to generate the fuzzy emotional membership of a certain feature in a test image.

Different from prior work [47] that shifted the features to emotions directly, we adopt sparse learning to learn the correlation coefficients between the same features in different images. Then these correlation coefficients are used as joint sparsity constraints to linearly combine the fuzzy memberships of unlabeled images and training images. As illustrated in Fig. 2, the sparse learning based on fuzzy emotional memberships contains two procedures, shared factors learning and fuzzy emotional memberships mapping. First, the shared factors are learned through the visual features of training images. Then, we acquire the fuzzy memberships of the test image by linearly combining the fuzzy memberships of training images through the learned shared factors.

Given a test image I_m, the feature set is defined as I_m ={ Y _m1, Y _m2, …, Y _mj, …, Y _mJ }. In the previous step, our proposed EFKNN algorithm calculates the fuzzy emotional memberships of training images. And we represent the fuzzy emotional memberships of features set X _j as U_{X
_j} = [ μ_1j, …, μ_nj, …, μ_Nj], where μ_nj denotes the fuzzy membership vector of the jth feature of image I_n. Our goal is to get the fuzzy membership vector μ_mj of the feature Y _mj extracted from the test image I_m through the sparse learning method. In this paper, the task of proposed sparse learning aims to learn an appropriate mapping: f : X _j → μ_mj, and is based on the idea that Y _mj and μ_mj of image I_m can be represented in terms of the bases $X_{j} \in ℝ^{d_{j} \times N}$ and $U_{X_{j}} \in ℝ^{L \times N}$ respectively. It is defined in the following forms: $\begin{matrix} Y_{mj} = X_{j} θ_{j} and μ_{mj} = U_{X_{j}} θ_{j} \end{matrix}$ (8) where θ_j is shared sparse coefficients, $θ_{j} \in ℝ^{N}$ . It can be formulated as an optimization problem: $\begin{matrix} \begin{matrix} [θ^{*}] = \min ‖ Y_{mj} - X_{j} θ_{j} ‖_{2}^{2} + β ‖ θ_{j} ‖_{0} \\ subject to θ_{j} ⩾ 0, ‖ θ_{j} ‖_{1} = 1 \end{matrix} \end{matrix}$ (9) where ∥ · ∥ ₀, ∥ · ∥ ₁ and ∥ · ∥ ₂ are ℓ₀, ℓ₁ and ℓ₂ norm, respectively. They count the number of nonzero elements in a vector. The parameter β is a regularized normative coefficient. θ_j satisfies the constraints θ_j ⩾ 0 and || θ_j||₁ = 1. The constraint conditions are optimized to ensure the predicted result μ_mj in a certain range of μ_mj ⩾ 0 and ||μ_mj||₁ = 1. Note that unlike previous applications using sparse learning [45, 50], the ℓ₁ norm is replaced with ℓ₀ norm, and it is optimized with constraints to guarantee the property of fuzzy memberships. If the ℓ₁ norm is optimized directly without constraints, there is no guarantee that the sum of the results is 1.

The Eq. (9) is precisely an NP-hard problem that is unable to be solved directly. We replace ℓ₀ norm with ℓ_r norm and rewrite the objective function to: $\begin{matrix} J (θ_{j}) = ‖ Y_{mj} - X_{j} θ_{j} ‖_{2}^{2} + β ‖ θ_{j} ‖_{r}^{r} \end{matrix}$ (10) where 0 < r ⩽ 1. We optimize shared factors by iteratively reweighted least squares (IRLS) [51]. The IRLS for solving Eq. (10) were mostly considered for r ⩾ 1. The case of r < 1 was used by Rao and Kreutz-Delgado [52], which replaced a reweighted ℓ₂ norm with ℓ_r objective function. In this paper, we consider 0 < r ⩽ 1 following [51]: $\begin{matrix} \begin{matrix} J (θ_{j}) = ‖ Y_{mj} - X_{j} θ_{j} ‖_{2}^{2} + β \sum_{n = 1}^{N} {| θ_{nj} |}^{r} \\ = ‖ Y_{mj} - X_{j} θ_{j} ‖_{2}^{2} + β \sum_{n = 1}^{N} \frac{1}{δ_{n}} {| θ_{nj} |}^{2} \end{matrix} \end{matrix}$ (11) where δ_n = |θ_nj|^2-r + φ. φ > 0 is introduced to avoid dividing by zero. The solution of Eq. (11) can be converted into the following forms: $\begin{matrix} J (θ_{j}) = {(θ_{j})}^{T} ({(X_{j})}^{T} X_{j} + β Q_{n}) θ_{j} - 2 {(Y_{mj})}^{T} X_{j} θ_{j} \end{matrix}$ (12) where Q _n is a diagonal matrix with $Q_{n} = \frac{1}{δ_{n}}$ . Accordingly, Eq. (9) is solved by transforming into the following equation with constrained least square problems: $\begin{matrix} \begin{matrix} \min {{(θ_{j})}^{T} ({(X_{j})}^{T} X_{j} + β Q_{n}) θ_{j} - 2 {(Y_{mj})}^{T} X_{j} θ_{j}} \\ subject to θ_{j} ⩾ 0, ‖ θ_{j} ‖ = 1 \end{matrix} \end{matrix}$ (13)

In practice, the value of r approximates 0, r → 0. Finally, the fuzzy membership μ_mj is computed through Eq. (8). The whole learning procedure is summarized in Algorithm 2.

Algorithm 2:

The sparse learning based on fuzzy
memberships.
Input: The training feature set X_j and the collection
of fuzzy membership matrixes ${U_{X_{j}}}_{j = 1}^{J}$ , test
feature y_mj, regularization coefficient β, max-epochs
E, error threshold τ.
Output: The fuzzy membership μ_mj for y_mj.
1 Initialization: $θ_{j}^{(0)} = [θ_{1 j}^{(0)}, \dots, θ_{Nj}^{(0)}] \leftarrow 1 / N$ ,
φ ← 10^-9, r ← 0;
2 fore ← 1 to E do
3 $Q_{n}^{(e)} \leftarrow \frac{1}{{\| θ_{nj}^{(e - 1)} \|}^{2 - r} + φ}, n = 1, \dots, N$ ;
4 $θ_{j}^{(e)} \leftarrow$ Eq. (9)
5 if $‖ θ_{j}^{(e)} - θ_{j}^{(e - 1)} ‖ < τ$ then
6 break;
7 end
8 end
9 return $μ_{mj} = U_{X_{j}} θ_{j}^{(e)}$

3.4 Fully connected neural network for emotion distribution prediction

Through our proposed EFKNN, the fuzzy emotional membership of each feature extracted from training images is collected and represented as ${μ_{nj}}_{n = 1}^{N}$ , where N is the number of training images. In this section, a fully connected neural network is introduced to predict the distribution for image emotion recognition, which plays the role of mapping the obtained fuzzy memberships to the image emotion space. The fully connected neural network is composed of an input layer, output layer, and one or more hidden layers. This work adopts two hidden layers for the fully connected neural network considering the number of parameters and the experimental results. During the training phase, the value of the neurons of each layer and weights are trained. The collected fuzzy memberships of training images by EFKNN are used as the input to the fully connected neural network.

Suppose the input vector in layer h is $Z^{h} = [z_{t}^{h}, \dots, z_{T}^{h}]$ , and the pre-activation value z_i for neuron i of layer h + 1 is computed through the function f^h+1: $\begin{matrix} z_{i}^{h + 1} = f^{h + 1} (z^{h}) = \sum_{t = 1}^{T} W_{it}^{h + 1} z_{t}^{h} + b_{i}^{h + 1} \end{matrix}$ (14) where $W_{it}^{h + 1}$ represents the weights of the connection of $z_{t}^{h}$ in layer h to the neuron i in layer h + 1, $b_{i}^{h + 1}$ is the bias of neuron i in layer h + 1. And T is the number of neurons in layer h. Then, the output of layer h is obtained through a non-linear activation function g (·). In this work, the tanh function is used as the non-linear activation function throughout the network.

Finally, the output of the last fully connected layer is converted into the probability distribution of different emotions by a softmax function, which makes sure the output of the neural network p _n = [p_n1, p_n2, …, p_nl] satisfies p_nl ∈ [0, 1] and $\sum_{l = 1}^{L} p_{nl} = 1$ . Let {z_l∣ l = 1, 2, … L } be the activation values of the last fully connected layer, and the emotion distribution of image I_n is calculated by the softmax function as the following formulae: $\begin{matrix} {\hat{p}}_{nl} = \frac{exp (z_{l})}{\sum_{l = 1}^{L} exp (z_{l})} \end{matrix}$ (15)

There are various criteria to measure the similarity between two label distributions. Following the literature [53], this work utilizes the Kullbavk-Leibler (KL) divergence as the overall loss function to measure the similarity of the ground-truth p _n and the predicted emotion distribution ${\hat{p}}_{nl}$ : $\begin{matrix} L = - \frac{1}{N} \sum_{n = 1}^{N} \sum_{l = 1}^{L} p_{nl} ln {\hat{p}}_{nl} \end{matrix}$ (16)

4 Experiments

In this section, we evaluate the proposed method against the state-of-the-art emotion distribution prediction methods by extensive experiments to verify the effectiveness of our proposed method.

4.1 Experiment settings

4.1.1 Datasets

To validate the availability of our proposed model for image emotion distribution prediction, we conduct experiments on three image emotion distribution datasets:

Emotion6: this dataset consists of 1980 images with 330 images per category collected from Flickr by retrieving the six basic emotions keywords and synonyms. The responses of subjects were received by Amazon Mechanical Turk (AMT). Each image was voted via diving sum vote of six basic emotions categories (anger, disgust, fear, joy, sadness, surprise) and a neutral category.

Abstract [6]: this dataset includes 279 abstract paintings, which are without any discernible objects but with color and texture. These images were classified to 8 emotion categories (anger, amusement, awe, contentment, disgust, excitement, fear, sadness) by approximately 230 people.

IESN [18, 19]: this dataset was firstly established for predicting personalized emotion, which contains 1012901 images acquired from Flickr by searching keywords. The emotion labels are obtained from the corresponding metadata and comments. The images are classified into eight emotions like the Abstract dataset. Similar to the literature [25], we select 3792 images that are assigned with more than 15 emotion categorical labels.

For emotion distribution prediction, emotion distribution can be obtained through normalization. The number of votes for each emotion category is divided by the total votes of 7 or 8 categories. For example, assuming there are 15 viewers voting on 8 emotion categories {2, 5, 0, 8, 0, 1, 2, }, the emotion probability distribution is {0.1, 0.25, 0, 0.4, 0, 0.05, 0.1, 0.1}. Note that the same image can be voted with multiple emotions for one viewer.

4.1.2 Features

Based on the fact that the emotion evoked by an image is delivered by various types of visual features, we extract multiple features from different levels to enhance the visual feature representation and explore the relationships between multiple features and emotions.

Low-level features

GIST feature [28]: one of the most usually used low-level features describes the visual features in scene attributes.

Elements-of-art feature [6]: the features are derived from psychology and art theory consisting of eight different elements-of-art features. This work extracts the color and texture features among them.

Mid-level features

Principles-of-art feature (PAEF) [10]: the features based on the principle of art theory, including balance, contrast, harmony, variety, gradation, and movements. The PAEF features are more interpretable and semantic than the elements-of-art features.

Attribute-based features [28]: the attribute-based feature is often used to describe scenes, and each image is represented by a 102-dimensional attribute feature vector.

High-level features

SentiBank features [16]: the 1200 adjective noun pairs (ANPs) detected by 1200 detectors called SentiBank. The SentiBank outputs a 1200-dimensional binary vector for images.

Deep CNN features [54]: the deep CNN feature extracted from the last fully connected layer of the AlexNet pre-trained on ImageNet.

4.1.3 Baseline models

We compare our proposed method with the following state-of-art approaches for image emotion distribution learning: CNNR [6], ACPNN [24], DLDL [53], SSL [25], WMSSL [47] and WMCPNN [55]. The CNNR changes the last fully connected layer to 1 and replaces the softmax loss by a Euclidean loss. Referring to the literature [6], we pre-train the Caffe model [56] on ImageNet and fine-tune the CNN in our training set for CNNR. The ACPNN and BCPNN share similar network structures with CPNN [23]. But different from CPNN that takes singles integers as label representation, the BCPNN and ACPNN replace the integer labels by binary coding and augmented label distribution, respectively. However, the CPNN is specially designed for age estimation where the integral labels correspond to ages, but emotions have no relevance with numbers. It is meaningless to feed the emotion labels to CPNN, so this work does not conduct experiments on it. Besides, the ACPNN was proposed based on the BCPNN. As proved in the literature [24], the performance of ACPNN is better than BCPNN, so we only compare our method with the ACPNN. The DLDL method is a CNN-based algorithm that replaces the Euclidean loss with the KL divergence loss. The SSL predicts the emotion distribution of the test image by learning the combination coefficients and linearly combining the coefficients with the emotion distribution of training images. The WMSSL is an expanding version of SSL, which learns the optimal weights for multiple features and utilizes the joint sparsity constraints across different features. The WMCPNN extends the CPNN into multiple features setting and associates the visual features with emotion distribution by learning the combination coefficients of multiple features and exploring the complementarity.

4.1.4 Evaluation metrics

Since the evaluation metrics for the single-label classification method are not applicable in distribution learning, we use five evaluation metrics introduced in [22, 47] to assess our model, including the squared difference (SSD), the Bhattacharyya coefficient (BC), the Kullback-Leibler (KL) divergence, the Chebyshev distance (Che) and the cosine coefficient (Cos). We estimate the performance of emotion distribution by measuring the similarity or distance between the predicted emotion distribution and the ground-true. In these metrics, the SSD measures in terms of regression. The BC and KL measure the distance between two distributions, and the Cos measures the similarity. For SSD, KL and Che metrics, the lower is better. For BC and Cos metrics, the higher is better.

4.1.5 Implementation details

We execute our experiments on three datasets. The Abstract dataset randomly divides into 80%for training and 20%for testing. And the Emotion6 and IESN datasets randomly divide into 70%for training and 30%for testing. For our proposed EFKNN, the fuzzy parameter α is set to 2 following the recommendation in the literature [39]. We also investigate the optimal value of K for the EFKNN in section 4.2.4 to ensure that the model has superior performance. For the sparse learning, the parameter β is set to 0.0001 empirically. Then, we set up a fully connected neural network with two hidden layers to learning the relationship between the fuzzy memberships of multiple features and the emotion terms. As the datasets for our experiments contain seven or eight emotion categories, and we employ six types of features for emotion distribution prediction, the dimension of the input fuzzy membership vectors is 48. For the Emotion6 dataset containing seven categories, the remaining nodes of the input layer are filled with 0. The output is an emotion label distribution with emotion categories. During the training phase, we set the learning rate to 0.001. The network is trained through the stochastic gradient descent (SGD) optimization method. Our experiments are all carried out on NVIDIA GTX 1080Ti GPU with 32 GB onboard memory.

4.2 Results and analysis

4.2.1 The effectiveness of multiple features

Previous studies have proved that combining multiple features can significantly improve the performance of image emotion recognition [15]. However, the influence of various types of features in our proposed fuzzy membership generation still needs to be explored. In order to verify the necessity of the features required in this model, we conduct an ablation study on three datasets to compare the impact of different visual features. We remove one feature each time, including each type of visual feature mentioned in section 4.1.2 to execute experiments for our proposed model. Meanwhile, the results of using all features are also reported. The results are illustrated in Fig. 5. The “-GIST”, “-Elem”, “-PAEF”, “-Attribute”, “-SentiBank”, and “-CNN” represent the proposed model without GIST, Elem, PAEF, Attribute, SentiBank, and CNN, respectively. And “Our” represent our model with all features. Please note that comparing to our method with all features, the worse the performance, the larger gain the feature impacts for our method to predict emotion distribution. Among them, the lower the value of SSD, KL, and Che, the better the performance, and the higher the value of BC and Cosine, the better the performance.

Fig. 5

The effectiveness of different features in the proposed model on three datasets evaluated by SSD, KL, Che, BC, and Cos, respectively.

From Fig. 5, we can observe that: (1) The high-level features have the greatest impact on the performance of the model. When the SentiBank feature or the CNN feature are removed, the performance of the model has been significantly reduced. In addition, the CNN feature is superior to other hand-crafted features. (2) Among the low-level and mid-level features, the Elem and PAEF features have more influence on the results of emotion distribution prediction. A possible explanation for this might be that these features are artistic features, which can describe emotions more comprehensively. (3) Our model still achieves satisfying results when the attribute feature on the Abstract dataset and the GIST feature on the IESN dataset is removed. It may be explained by the reason that the images on the Abstract dataset are abstract paintings without apparent objects and scenes, so the attribute feature has little impact on the Abstract dataset. In contrast with the IESN dataset, the images are collected from the social network. These images have obvious objects and expressions that are directly related to emotions, which results in the GIST feature has almost no effect on the IESN dataset. (4) Across different datasets, the prediction accuracy on Emotion6 fluctuates significantly when the corresponding emotion features are removed. But the fluctuations of accuracy on the Abstract and IESN datasets are relatively small. (5) Our method with all visual features performs best compared to the model with any features removed, which indicates that the multiple features selected in this model are valid.

4.2.2 The effectiveness of EFKNN

To assess the effectiveness of the proposed EFKNN in fuzzy emotional membership calculation, we compare our framework with fuzzy emotional memberships calculated using traditional triangle function, fuzzy C-means clustering (FCM), original FKNN and the proposed EFKNN, respectively. The experiments were performed on the Emotion6 and Abstract datasets. And the results on two datasets estimated by KL and BC are listed in Table 1. As shown in Table 1, the prediction metrics of our framework with EFKNN exceed that of our framework with other methods, which demonstrates the effectiveness of our proposed EFKNN. Moreover, our framework with the triangle function outperforms than using the FCM. The reason is that the triangle function manually selects the centers of the initial emotion categories when calculating the fuzzy emotional memberships. It is artificially intervened with the prior knowledge. Therefore, the performance is superior to the FCM, which is unsupervised clustering based on the similarity of features. Although our framework with the original FKNN improves the accuracy compared to our framework with triangle function and FCM, the prediction results of using the original FKNN are even lower than some baseline methods in Table 2. It is because employing the original FKNN to calculate fuzzy emotional memberships is based on the similarity of features rather than the affective perspective, which further demonstrates the availability of our proposed EFKNN for improving performance.

Table 1
Performance comparison between our framework with fuzzy emotional memberships calculated by the triangle function, FCM, the FKNN and the proposed EFKNN, respectively

Method Emotion6 Abstract

KL↓ BC↑ KL↓ BC↑

With triangle function 0.5052 0.8436 0.5186 0.8198

With FCM 0.5103 0.8408 0.5203 0.8195

With FKNN 0.4987 0.8523 0.5136 0.8208

With EFKNN 0.4612 0.8635 0.4552 0.8535

Method	Emotion6	Abstract
With triangle function	0.5052	0.8436	0.5186	0.8198
With FCM	0.5103	0.8408	0.5203	0.8195
With FKNN	0.4987	0.8523	0.5136	0.8208
With EFKNN	0.4612	0.8635	0.4552	0.8535

Table 2

Performance comparison with state-of-art methods for emotion distribution learning on the Emotion6, Abstract and IESN datasets

Datasets		CNNR	ACPNN [24]	DLDL	SSL [8]		WMSSL [47]	WMCPNN [55]	Ours
		Uni-feature			Early	Late	Multi-level features
Emotion6	SSD↓	0.1384	–	0.1281	0.1344	0.1402	0.1268	0.1231	0.1212
	BC↑	0.8432	–	0.8513	0.8484	0.8411	0.8529	0.8631	0.8635
	KL↓	0.5071	0.5057	0.4855	0.4825	0.5064	0.4793	0.4642	0.4612
	Che↓	0.2685	0.2799	0.2612	–	–	–	–	0.2521
	Cos↑	0.7314	0.7658	0.7691	–	–	–	–	0.7721
Abstract	SSD↓	0.1262	–	0.1253	0.1271	0.1241	0.1191	0.1088	0.1062
	BC↑	0.8194	–	0.8212	0.8142	0.8210	0.8319	0.8442	0.8535
	KL↓	0.5103	0.5134	0.5096	0.5126	0.5034	0.4820	0.4611	0.4552
	Che↓	0.2606	0.2344	0.2323	–	–	–	–	0.2235
	Cos↑	0.7368	0.7628	0.7685	–	–	–	–	0.7762
IESN	SSD↓	0.1701	–	0.1698	0.1676	0.1706	0.1569	0.1532	0.1528
	BC↑	0.8462	–	0.8507	0.8542	0.8524	0.8583	0.8704	0.8689
	KL↓	0.4856	–	0.4812	0.4802	0.4837	0.4777	0.4696	0.4691
	Che↓	0.2682	–	0.2563	–	–	–	–	0.2528
	Cos↑	0.7415	–	0.7552	–	–	–	–	0.7563

4.2.3 Comparison with the state-of-the-arts

We compare our model with the state-of-art approaches mentioned above and followed by a detailed discussion. First, our method compares with the uni-feature based models for emotion distribution prediction, including CNNR [5], ACPNN [24], and DLDL [53]. Then, the early fusion and late fusion strategies are employed to process multiple features for the SSL [8] models, including the emotion features mentioned in section 4.1.2. Further, the WMSSL [47] and MCPNN [55] can make use of multi-features, so they feed the multiple features directly into the models. Table 2 presents the experimental results on three datasets evaluated by five metrics. The results highlighted in bold indicate the best value of each metric.

From the results shown in Table 2, we can conclude that: (1) The DLDL performs better than the other uni-feature based methods, which validates its effectiveness for emotion distribution prediction. (2) The accuracy of the SSL for fusing multiple features is related to datasets. The early fusion achieves better results on the Emotion6 and IESN, while late fusion performs better on the Abstract. (3) The DLDL method is superior to SSL on Emotion6 and IESN datasets. But some metrics are worse than the SSL on the Abstract dataset. It may be explained by the reason that the DLDL is a deep-CNN based model built for object recognition. But the images in the Abstract dataset are without recognizable objects, which makes the DLDL unable to achieve better performance on Abstract than the SSL that also uses low-level and mid-level features. (4) In some cases, the CNNR outperforms the late fusion of SSL on Emotion6 and IESN, and early fusion on Abstract. It may because the training procedure of CNNR is under supervised, and the optimal emotion distribution is achieved by minimizing the objective function of each emotion category. This result shows its comparable performance for emotion distribution prediction. (5) The WMSSL and WMCPNN, which jointly fusion multiple features, exceed all uni-features based models.

As shown in Table 2, compared with the state-of-the-art approaches on three different scale and emotion categories datasets, our proposed method makes an obvious improvement in all datasets. Our model shows an advantage over uni-feature based methods. The reasons lie in two aspects. First, our proposed method combines multiple visual features. Second, our method calculates the fuzzy membership of each feature and emotion category by the proposed EFKNN, which verifies the feasibility of the EFKNN. As for the multi-feature approaches, our model also outperforms the WMSSL and the WMCPNN. Unlike the WMSSL and WMCPNN, which weight multi-features, our method computes the fuzzy membership of each feature to emotion categories based on the proposed EFKNN. Therefore, the emotion distribution of an image is indirectly deconstructed to fuzzy memberships for intermediate representation, which indicates the fuzzy emotional membership is capable of narrowing the affective gap for image emotion distribution prediction. Compared with the best results of the SSL, WMSLL, and WMCPNN in different datasets, our method makes a more obvious improvement of performance on Emotion6 and Abstract datasets than that of IESN dataset, especially on the Abstract dataset. The BC evaluation metric of our approach on the IESN dataset is even lower than that of WMCPNN. In view of the fact that images in the IESN dataset are from the social networks, and the labeled emotions are easily impacted by others, the labeled emotions in the IESN dataset are less subjective than those in the Emotion6 and Abstract datasets. This result suggests that our method shows advantages in dealing with ambiguous data.

To give intuitive predicted results of our method, Fig. 6 visualizes the ground truth and the predicted emotion distribution for several image examples from different datasets. The predicted emotion distributions of our proposed method are more similar to the ground truth than the other approaches, which further demonstrates the contribution of our model to improving the prediction accuracy of image emotion distribution.

Fig. 6

Visualization of the predicted emotion distribution results of several examples using the proposed model and comparable approaches.

4.2.4 Choice of K value

As described above, our proposed model calculates the fuzzy emotional memberships of visual features by EFKNN. In terms of the FKNN algorithm, the value of K is the number of considering nearest neighbors for assigning the items into emotion terms. The selection of the K value has an important influence on the prediction of the EFKNN algorithm. A meager K value may produce unreasonable predicted results, while a high K value may lead to outliers influence the results. Therefore, we study the optimal value for the parameter K by cross-validation over the possible choices {1, 3, 5, 7, 9, 11, 13, 15, 17, 19}. The experimental results for each feature versus the parameter K on the Emotion6, Abstract and IESN datasets are measured by KL divergence. As illustrated in Figs. 7 , 8 and 9, the best performance is obtained with K = 7 on the Emotion6 dataset, K = 5 on the Abstract dataset and K = 13 on the IESN dataset, respectively. We chose them in all our experiments.

Fig. 7

Impact of different values of K on the Emotion6.

Fig. 8

Impact of different values of K on the Abstract.

Fig. 9

Impact of different values of K on the IESN.

5 Conclusion

In this paper, we propose a novel framework for emotion distribution learning, which can solve the subjectivity problem of emotion perceptions. We employ the fuzzy approach to explore the correlations between visual features and human subjective emotion. The EFKNN algorithm is developed to transform visual features into fuzzy emotional memberships as the intermediate representation to bridge the affective gap. It provides the capability to capture the ambiguity and uncertainty between visual features and emotions. Moreover, our method also shows an advantage in interpretability for image emotion recognition. And we implement a sparse learning algorithm based on fuzzy memberships to acquire the fuzzy emotional memberships of the unlabeled images. The experimental results on three datasets verify that our proposed method outperforms the state-of-the-art methods for emotion distribution learning.

However, the computational efficiency of the proposed approach costs more time because of its complexity of optimization. In the future, we will improve the efficiency of our method and extend the model to tackle larger-scale datasets. Furthermore, we also plan to implement real-world applications based on emotion distributions, such as artworks emotion understanding and use the learned principles in painting education or intelligent creation.

Footnotes

Acknowledgments

This work is supported by the National Key Research and Development Plan of China (No. 2017YFD0400101).

References

Zhao

, Gao

, Ding

and Chua

T.S.

, Real-Time Multimedia Social Event Detection in Microblog, IEEE Transations on Cybernetics 48 (2018), pp. 3218–3231.

Gauba

, Kumar

, Roy

P.P.

, Singh

, Dogra

D.P.

and Raman

, Prediction of advertisement preference by fusing EEG response and sentiment analysis, Neural Networks 92 (2017), 77–88.

Qian

, Zhang

and Xu

, Multi-modal multi-viewtopic-opinion mining for social event analysis, in: Proceedings of the 24th ACM international conference on Multimedia, 2016, pp. 2–11.

Shen

, Jia

, Nie

, Feng

, Zhang

, Hu

, Chua

T.-S.

and Zhu

, Depression Detection via Harvesting Social Media: A Multimodal Dictionary Learning Solution, in: IJCAI, 2017, pp. 3838–3844.

Peng

K.-C.

, Chen

, Sadovnik

, Gallagher

, and Ieee, A Mixed Bag of Emotions: Model, Predict, and Transfer Emotion Distributions, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 860–868.

Machajdik

and Hanbury

, Affective image classification using features inspired by psychology and art theory, in: Proceedings of the 18th ACM international conference on Multimedia, 2010, pp. 83–92.

Zhao

, Yao

H.X.

, Gao

, Ji

R.R.

and Ding

G.G.

, Continuous Probability Distribution Prediction of Image Emotions via Multitask Shared Sparse Regression, Ieee Transactions on Multimedia 19 (2017), 632–645.

Zhao

, Ding

, Gao

and Han

J.J.T.

, Approximating discrete probability distribution of image emotions by multi-modal features fusion, 1000 (2017), 4669–4675.

Hanjalic

, Extracting moods from pictures and sounds, IEEE Signal Processing Magazine 23 (2006), 90–100.

10.

Zhao

, Gao

, Jiang

, Yao

, Chua

T.-S.

, Sun

, and Acm, Exploring Principles-of-Art Features For Image Emotion Recognition, in: Proceedings of the 2014 Acm Conference on Multimedia, 2014, pp. 47–56.

11.

Yanulevskaya

, van Gemert

J.C.

, Roth

, Herbold

A.K.

, Sebe

, Geusebroek

J.M.

, and Ieee, Emotional valence categorization using holistic image features, in: 2008 15th IEEE International Conference on Image Processing, 2008, pp. 101–104.

12.

You

, Luo

, Jin

, Yang

, and Aaai, Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and the Benchmark, in: Thirtieth AAAI Conference on Artificial Intelligence, 2016, pp. 308–314.

13.

Alameda-Pineda

, Ricci

, Yan

, Sebe

, and Ieee, Recognizing Emotions from Abstract Paintings using Non-Linear Matrix Completion, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 5240–5248.

14.

Zhang

, He

X.Y.

and Lu

W.Z.

, Exploring Discriminative Representations for Image Emotion Recognition With CNNs, IEEE Transactions on Multimedia 22 (2020), 515–523.

15.

Rao

T.R.

, Li

X.X.

and Xu

, Learning Multi-level Deep Representations for Image Emotion Classification, Neural Processing Letters 51 (2020), 2043–2061.

16.

Borth

, Ji

, Chen

, Breuel

and Chang

S.-F.

, Large-scale visual sentiment ontology and detectors using adjective noun pairs, in: Proceedings of the 21st ACM International Conference on Multimedia, 2013, pp. 223–232.

17.

Chen

, Borth

, Darrell

and Chang

S.-F.J.a.p.a.

, Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks, (2014), arXiv:1410.8586.

18.

Zhao

, Yao

, Gao

, Ji

, Xie

, Jiang

and Chua

T.-S.

, Predicting personalized emotion perceptions of social images, in: Proceedings of the 24th ACMInternational Conference on Multimedia, 2016, pp. 1385–1394.

19.

Zhao

S.C.

, Yao

H.X.

, Gao

, Ding

G.G.

and Chua

T.S.

, Predicting Personalized Image Emotion Perceptions in Social Networks, IEEE Transactions on Affective Computing 9 (2018), 526–540.

20.

, Xu

C.S.

, He

X.J.

, Jin

J.S.

, Luo

S.H.

and Rui

, Hierarchical affective content analysis in arousal and valence dimensions, Signal Processing 93 (2013), 2140–2150.

21.

Zhou

, Cao

, Zhong

and Geng

, Learning meta-knowledge for few-shot image emotion recognition, Expert Systems with Applications 168 (2021).

22.

Xin

, Label distribution learning, IEEE Transactions on Knowledge Data Engineering 28 (2016), 1734–1748.

23.

Geng

, Yin

and Zhou

Z.H.

, Facial age estimation by learning from label distributions, IEEE Transations Pattern Analysis and Machine Intelligence 35 (2013), 2401–2412.

24.

Yang

, Sun

, and Aaai, Learning Visual Sentiment Distributions via Augmented Conditional Probability Neural Network, in: 31st AAAI Conference on Artificial Intelligence, 2017, pp. 224–230.

25.

Zhao

, Yao

, Jiang

, Sun

, and Ieee, Predicting discrete probabillity distribution of image emotions, in: IEEE International Conference on Image Processing, 2015, pp. 2459–2463.

26.

, Jin

and Assoc Comp

, Image Emotion Distribution Learning with Graph Convolutional Networks, in: Proceedings of the 2019 ACM International Conference on Multimedia Retrieval, 2019, pp. 382–390.

27.

Fan

Y.Y.

, Yang

H.S.

, Li

Z.H.

and Liu

, Predicting Image Emotion Distribution by Learning Labels’ Correlation, IEEE Access 7 (2019), 129997–130007.

28.

Patterson

, Hays

, and Ieee, SUN Attribute Database: Discovering, Annotating, and Recognizing Scene Attributes, in: IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 2751–2758.

29.

Liu

, Li

and Xia

, Affective image classification by jointly using interpretable art features and semantic annotations, Journal of Visual Communication and Image Representation 58 (2019), 576–588.

30.

Zadeh

L.A.

, Fuzzy logic-a personal perspective, Fuzzy Sets and Systems 281 (2015), 4–20.

31.

Liu

, Burnap

, Alorainy

and Williams

M.L.

, A Fuzzy Approach to Text Classification With Two-Stage Training for Ambiguous Instances, IEEE Transactions on Computational Social Systems 6 (2019), 227–240.

32.

Antonio Sanz

, Galar

, Jurio

, Brugos

, Pagola

and Bustince

, Medical diagnosis of cardiovascular diseases using an interval-valued fuzzy rule-based classification system, Applied Soft Computing 20 (2014), 103–111.

33.

Akram

, Adeel

, Al-Kenani

A.N.

and Alcantud

J.C.R.

, Hesitant fuzzy N-soft ELECTRE-II model: a new framework for decision-making, Neural Computing and Applications, (2020). https://doi.org/10.1007/s00521-020-05498-y.

34.

Lee

, Kwon

, Sri

S.K.

and Lee

, Emotion recognition based on 3D fuzzy visual and EEG features in movie clips, Neurocomputing 144 (2014), 560–568.

35.

Cao

and Chen

, Fuzzy Emotional Semantic Analysis and Automated Annotation of Scene Images, Computational Intelligence and Neuroscience 2015 (2015).

36.

Nguyen

T.L.

, Kavuri

and Lee

, A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips, Neural Networks 118 (2019), 208–219.

37.

Pedrycz

, Amato

, Di Lecce

and Piuri

, Fuzzy Clustering With Partial Supervision in Organization and Classification of Digital Images, IEEE Transactions on Fuzzy Systems 16 (2008), 1008–1026.

38.

Bhat

A.J.I.J.o.C.

, and Indormatics, Possibility Fuzzy C-Means Clustering For Expression Invariant Face Recognization, International Journal on Cybernetics & Infomational (IJCI) 3 (2014), 35–45.

39.

Keller

J.M.

, Gray

M.R.

and Givens

J.A.J.I.t.o.s.

, man, A fuzzyk-nearest neighbor algorithm, IEEE Transactions on Systems Man & Cybernetics SMC-15 (1985), 580–585.

40.

Cover

and Hart

, Nearest neighbor pattern classification, IEEE Transactions on Information Theory 13 (1967), 21–27.

41.

Kaur

, Saini

B.S.

and Gupta

, An adaptive fuzzy K-nearest neighbor approach for MR brain tumor image classification using parameter free bat optimization algorithm, Multimedia Tools and Applications 78 (2019), 21853–21890.

42.

Natarajan

and Sathiamoorthy

, Content based medical image retrieval using multi-trend structure descriptor and fuzzy k-NN classifier, in: 2019 International Conference on Communication and Electronics Systems (ICCES), IEEE, 2019, pp. 2097–2102.

43.

Sun

, Shao

, Li

, Liu

, Yan

, Qiu

and Huang

, Learning sparse sharing architectures for multiple tasks, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2020, pp. 8936–8943.

44.

Guha

and Ward

R.K.

, Learning sparse representations for human action recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (2012), 1576–1588.

45.

, Xiong

, Hu

and Ding

, Context-aware affective images classification based on bilayer sparse representation, in: Proceedings of the 20th ACM International Conference on Multimedia, 2012, pp. 721–724.

46.

Wright

, Ganesh

, Zhou

Z.H.

, Wagner

and Ma

, Robust Face Recognition via Sparse Representation, IEEE Transactions on Pattern Analysis and Machine Intelligence 31 (2008), 942–943.

47.

Zhao

S.C.

, Ding

G.G.

, Gao

, Zhao

, Tang

Y.B.

, Han

J.G.

, Yao

H.X.

and Huang

Q.M.

, Discrete Probability Distribution Prediction of Image Emotions with Shared Sparse Learning, IEEE Transactions on Affective Computing 11 (2020), 574–587.

48.

Wold

, Esbensen

and Geladi

, Principal component analysis, Chemometrics Intelligent Laboratory Systems 2 (1987), 37–52.

49.

Rubner

, Tomasi

and Guibas

L.J.

, The Earth Mover’s Distance as a metric for image retrieval, International Journal of Computer Vision 40 (2000), 99–121.

50.

Yuan

X.T.

, Liu

and Yan

, Visual classification with multitask joint sparse representation, IEEE Transations Image Processing 21 (2012), 4349–4360.

51.

Chen

, Huang

, He

, Li

, and Ieee, Preconditioning for Accelerated Iteratively Reweighted Least Squares in Structured Sparsity Reconstruction, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2713–2720.

52.

Rao

B.D.

and Kreutz-Delgado

, An affine scaling methodology for best basis selection, IEEE Transactions on Signal Processing 47 (1999), 187–200.

53.

Gao

B.B.

, Xing

, Xie

C.W.

, Wu

and Geng

, Deep Label Distribution Learning With Label Ambiguity, IEEE Transations Image Processing 26 (2017), 2825–2838.

54.

Krizhevsky

, Sutskever

and Hinton

G.E.

, ImageNet Classification with Deep Convolutional Neural Networks, Communications of the ACM 60 (2017), 84–90.

55.

Zhao

, Ding

, Gao

, Han

, and Acm, Learning Visual Emotion Distributions via Multi-Modal Features Fusion, in: Proceedings of the 2017 ACM Multimedia Conference, 2017, pp. 369–377.

56.

Jia

, Shelhamer

, Donahue

, Karayev

, Long

, Girshick

, Guadarrama

, Darrell

, and Acm, Caffe: Convolutional Architecture for Fast Feature Embedding, in: Proceedings of the 2014 ACM Conference on Multimedia, 2014, pp. 675–678.