Speech Emotion Recognition Using an Enhanced Kernel Isomap for Human-Robot Interaction

4.1 System Structure

Figure 2 provides the system structure of speech emotion recognition using EKIsomap. As shown in Figure 2, it contains three principal parts: acoustic feature extraction, feature dimensionality reduction and emotion recognition. In the acoustic feature extraction stage, the original emotional speech samples from the emotional speech database are divided into two parts: training samples and testing samples. The corresponding acoustic features for training samples and testing samples, such as prosody and voice quality features, are extracted. The result of this stage is a speech data set represented by a set of high-dimensional speech features. The second stage aims at reducing the size of speech features and generating the new low-dimensional discriminating features with EKIsomap. The last stage in this system is that in the reduced low-dimensional feature space the trained SVM classifier is used to identify the accurate emotion categories, and provide recognition results.

OPEN IN VIEWER

Figure 2.

Speech emotion recognition system using EKIsomap

4.2 The Proposed EKIsomap Algorithm

The proposed EKIsomap is designed by integrating the kernel discriminant information extraction in a reproducing kernel Hilbert space (RKHS) with the existing kernel Isomap (KIsomap) [16] method. KIsomap effectively combines the kernel idea and Isomap and has a good generalization property on new data points, since it uses a kernel trick as is the case in kernel principal component analysis (KPCA) [17]. However, this kind of KIsomap still has two shortcomings. First, KIsomap, as an unsupervised learning method, fails to extract the discriminating embedded data representations since KIsomap does not take into account the known class label information of input data. This will heavily decrease the performance of KIsomap on classification tasks. Second, the kernel idea of KIsomap is that the geodesic distance matrix with a constant-shifting technique is referred to as a Mercer kernel matrix [16]. Therefore, as a kernel-based method, KIsomap cannot employ the characteristic of a kernel-based learning – i.e., a nonlinear kernel mapping – to explore higher-order information of input data sets, as KIsomap has no kernel function to perform a nonlinear kernel mapping. This is not a good property for KIsomap when adopted as a feature extraction method. To tackle the drawbacks of KIsomap, in this paper an enhanced variant of KIsomap, called ‘enhanced KIsomap’ (EKIsomap), is proposed and applied for speech emotion recognition.

The Fisher's criterion – namely, that the inter-class scatter should be maximized while the intra-class scatter should be simultaneously minimized – has become one of the most important selection criteria for projection techniques, since it endows the projected data vectors with good discriminating power. Motivated by Fisher's criterion, when using KIsomap to extract the low-dimensional embedded data representations, the inter-class dissimilarity could be maximized while the intra-class dissimilarity could be minimized in order to have improved tightness among similar patterns and better separability for dissimilar patterns. To develop the EKIsomap algorithm, a kernel matrix is first constructed by performing a nonlinear kernel mapping with a kernel function, and then a kernel discriminant distance – in which the inter class dissimilarity is maximized while the intra-class dissimilarity is simultaneously minimized – is designed to extract the discriminant information in a RKHS.

The detailed steps of EKIsomap are presented as follows:

Suppose the input data point (x_i,L_i), i = 1,2,3,…,N, where x_i ∊ R^D and L_i is the class label of x_i and the output data point is y_i ∊ R^d.

Step 1: Kernel nonlinear mapping for each x_i.

A nonlinear mapping φ is defined as:

φ : R D → F , x ↦ φ ( x )

(1)

By using the nonlinear mapping φ, the input data point x_i ∊ R^D is mapped into some potentially high dimensional feature space F. For a properly chosen φ, an inner product 〈,〉 can be defined on F, which makes a so-called reproducing kernel Hilbert space (RKHS). In a RKHS, a kernel function κ(x_i,x_j) can be defined as:

κ ( x i , x j ) = 〈 φ ( x i ) , φ ( x j ) 〉 = φ ( x i ) T φ ( x j )

(2)

where κ is known as a kernel.

Step 2: Find the nearest neighbours for each φ(x_i) by using the following kernel discriminant distance.

The kernel discriminant distance used in EKIsomap is defined as follows:

D κ ( x i , x j ) = { 1 − e − [ d κ ( x i , x j ) ] 2 β ​ ​ L i = L j e [ d κ ( x i , x j ) ] 2 β − α ​ ​ L i ≠ L j

(3)

where d_κ (x_i,x_j) is the kernel Euclidean distance matrix without class label information, whereas D_κ(x_i,x_j) is the kernel discriminant distance matrix integrating class label information. β is a smoothing parameter related to the data ‘density’, and it is usually feasible to set β to be the average kernel Euclidean distance between all pairs of data points. α is a constant factor (0 ≤ α ≤ 1) and gives the intra-class dissimilarity a certain probability of exceeding the inter-class dissimilarity. As shown in Eq. (3), we can make two observations. First, each dissimilarity function in D_κ(x_i,x_j) – i.e., inter-class dissimilarity and intra-class dissimilarity – is monotone, increasing with respect to the Euclidean distance. This ensures that the main geometric structure of the original data sets can be preserved well when using EKIsomap to produce low-dimensional embedded data representations. Second, the inter-class dissimilarity in D_κ(x_i,x_j) can always be definitely larger than the intra-class dissimilarity, conferring a high discriminating power of EKIsomap's projected data vectors. This is a good property for emotion classification.

Step 3: Compute the geodesic distances, d_ij, containing shortest paths for all pairs of data points by Dijkstra's algorithm and define D² = [d_ij²].

Step 4: Construct a matrix K(D²) based on the approximate geodesic distance matrix:

K ( D 2 ) = - 1 2 H D 2 H

(4)

where H = I-(1/N)ee^T,e = [1,···,1]^T ∊ R^N.

Step 5: Compute the top d eigenvectors and give an embedding of input points by:

Y = Λ 1 2 V T

(5)

where Λ^d×d is the eigenvalue matrix and V ∊ R^N×d is the eigenvector matrix.

5.1 Experiment Setup

We used the LIBSVM package, available at http://www.csie.ntu.edu.tw/∼cjlin/libsvm, to implement the SVM algorithm with a radial basis function (RBF) kernel, kernel parameter optimization and a one-versus-one strategy for the multi-class emotion classification problem.

As was done in [5], a tenfold cross-validation scheme was employed over the emotional speech data sets for all the emotion classification experiments and the recognition results were averaged. Following [12], the number of nearest neighbours was set to 12 for manifold learning methods, including LLE, Isomap, KIsomap and EKIsomap. For kernel-based methods, including KPCA and EKIsomap, the typical Gaussian kernel κ(x_i,x_j) = exp(-‖x_i -x_j‖² / 2σ²) was adopted for its better performance when compared with linear and polynomial kernels. Moreover, the parameter σ for the Gaussian kernel was set to 1 for its satisfying performance.

5.2 Experiment 1: Emotional Data Visualization

In this section, we compare the results of two dimensional embedded mappings of both KIsomap and EKIsomap on the extracted 48-dimensional acoustic features. For simplicity, we split the original acoustic feature data into two parts: one half for training and the other half for testing. Figure 3 shows the split training data and testing data from the Berlin speech corpus and the corresponding two-dimensional embedded mappings obtained by KIsomap and EKIsomap (α =0.2) on the training data and testing data. According to the distributed different colours and signs (the seven signs represent the seven emotions) of the embedding results in Figure 3, we can observe that, compared with KIsomap, EKIsomap projects training data and testing data into two embedded mappings with better tightness for similar patterns and better separability for dissimilar patterns. This is a good property for classification. In contrast, KIsomap produces the two dimensional embedded mappings with a visible overlapping of each other. The embedded results in Figure 3 clearly reveal that EKIsomap can nonlinearly extract the low-dimensional embedded data representations with a higher discriminating power in comparison with KIsomap. Therefore, EKIsomap is more suitable than KIsomap for speech emotion classification.

OPEN IN VIEWER

Figure 3.

Comparisons of the two-dimensional embedded results obtained by KIsomap and EKIsomap. (a) Training data from the Berlin speech corpus; (b) embedded points with KIsomap on the training data; (c) embedded points with EKIsomap on the training data; (d) testing data from the Berlin speech corpus; (e) projection of the testing data with KIsomap; (f) projection of the testing data with EKIsomap.

5.3 Experiment 2: Emotion Recognition Results and Analysis

The performance of PCA, LLE, Isomap, KIsomap and KPCA as well as EKIsomap on the speech emotion recognition tasks, and the corresponding embedded feature dimension, are presented in Figure 4. In each embedded feature dimension, the constant factor α (0 ≤ α ≤ 1) for EKIsomap can be optimized using a simple exhaustive search within a scope (α = 0,0.1,0.2,…,1). The optimal α corresponds to the best performance of EKIsomap. The best results of all the used methods along with the corresponding embedded feature dimensions are listed in Table 1. Note that the “Baseline” method denotes that the recognition result was directly obtained on the original 48-dimensional acoustic features.

OPEN IN VIEWER

Table 1.

The best results of different methods with the corresponding embedded feature dimension

Methods	Feature Dimension	Recognition Accuracy
Baseline	48	76.42%
PCA	15	72.35%
LLE	12	64.10%
Isomap	12	65.48%
KIsomap	17	71.32%
KPCA	16	76.21%
EKIsomap	9	80.85%

OPEN IN VIEWER

Figure 4.

Emotion recognition accuracy vs. the embedded feature dimension

From the results in Figure 4 and Table 1, we can see that EKIsomap obtains the best accuracy, of 80.85% with 9 embedded features, outperforming the other methods used. This indicates that EKIsomap is capable of extracting the low-dimensional embedded data representations with the highest discriminating power among all the used methods. This is attributed to the fact that EKIsomap aims to make the inter-class dissimilarity definitely larger than the intra-class dissimilarity with the aid of the kernel discriminant distance used in RKHS.

To further investigate the recognition results of different emotions when EKIsomap performs best with 9 embedded features, Table 2 gives the confusion matrix of the recognition results. As can be seen from Table 2, three emotions – i.e., anger, sadness and disgust – are identified relatively well (in detail, 90.83% for anger, 88.74% for sadness and 84.78% for disgust). Meanwhile, the other four emotions are classified with lower recognition rates (in detail, 65.76% for joy, 75.89% for neutral, 78.82% for boredom and 81.13% for fear).

OPEN IN VIEWER

Table 2.

The confusion matrix of recognition results (%) with EKIsomap (*Ang-anger, Joy-joy, Sad-sadness, Neu-neutral, Feafear, Bor-boredom, Dis-disgust)

	Ang	Joy	Sad	Neu	Fea	Bor	Dis
Ang	90.83	7.07	0	0	2.10	0	0
Joy	19.55	65.76	0.13	7.52	4.22	1.41	1.41
Sad	0	0	88.74	4.82	1.62	4.82	0
Neu	1.23	2.50	1.26	75.89	2.50	15.39	1.23
Fea	1.44	7.28	4.35	2.90	81.13	2.90	0
Bor	0.12	3.43	1.21	14.16	0.09	78.82	2.17
Dis	4.35	0	2.18	0	6.51	2.18	84.78