Analyzing mixed-type data by using word embedding for handling categorical features

Abstract

Most of real-world datasets are of mixed type including both numeric and categorical attributes. Unlike numbers, operations on categorical values are limited, and the degree of similarity between distinct values cannot be measured directly. In order to properly analyze mixed-type data, dedicated methods to handle categorical values in the datasets are needed. The limitation of most existing methods is lack of appropriate numeric representations of categorical values. Consequently, some of analysis algorithms cannot be applied. In this paper, we address this deficiency by transforming categorical values to their numeric representation so as to facilitate various analyses of mixed-type data. In particular, the proposed transformation method preserves semantics of categorical values with respect to the other values in the dataset, resulting in better performance on data analyses including classification and clustering. The proposed method is verified and compared with other methods on extensive real-world datasets.

Keywords

Word embedding categorical data mixed-type data visualized data analysis

1. Introduction

Due to popularity of information systems and decreased cost of storage devices, huge amount of data has been collected in many organizations. In those data, patterns or information which is valuable to the organization in decision making may be hidden. Nowadays, most of the collected data is of mixed-type, namely, containing numeric and non-numeric (or categorical) data. As shown in Table 1, in addition to the class attribute, feature attributes Hours_per_week (Hrs), Capital_gain, and Capital_loss are numeric while attributes Education, Marital_status and Relationship are categorical. The ability of analyzing mixed-type data to uncover patterns is essential in order to exploit the hidden information for benefit of the organizations.

Table 1
Portion of the Adult mixed-type dataset

Hrs	Education	Relationship	Marital_status	Capital_gain	Capital_loss	Class
31	Masters	Not-in-family	Windowed	0	0	$<=$ 50K
42	Assoc-acdm	Own-child	Married-AF-spouse	0	0	$<=$ 50K
58	Assoc-voc	Husband	Married-civ-spouse	0	1887	$>$ 50K
19	Some-college	Own-child	Divorced	0	0	$<=$ 50K
29	Assoc-voc	Husband	Separated	0	2415	$>$ 50K
27	Some-college	Other-relative	Never-married	0	0	$<=$ 50K
43	Some-college	Not-in-family	Divorced	0	0	$<=$ 50K
31	Bachelors	Not-in-family	Separated	0	1741	$<=$ 50K
22	7th-8th	Husband	Married-civ-spouse	2580	0	$<=$ 50K
37	Assoc-voc	Not-in-family	Windowed	4650	0	$<=$ 50K
…	…	…	…	…	…	…

Analyzing mixed-type data is not a trivial task. Most of data analysis algorithms take only one type of the data, either numeric or categorical. Transformation from one type to the other is usually performed before application of the algorithms. For instance, artificial neural networks take only numeric inputs while association-rule mining takes categorical (or discrete) values.

Many studies on measuring dissimilarity or similarity between multi-dimensional mixed-type data points have been conducted in the past. Most of the methods first measures the pairwise dissimilarities of individual feature attributes including numeric and categorical ones. Then, some sort of aggregation of the individual pairwise values is taken to measure the dissimilarity between two mixed-type data points.

The drawback of the previous type of approaches is that no numeric representation of individual categorical values but only the pairwise dissimilarity between the categorical values. Consequently, some analyses or applications are not applicable. For instance, the dimensionality reduction techniques, principal component analysis (PCA) [1] and linear discriminant analysis (LDA) [2], are applicable to only the datasets of which each attribute is represented by numeric values rather than the pairwise dissimilarities between categorical values.

The objective of this study is to tackle the problem with analyzing mixed-type datasets. In particular, we propose approaches to measure similarity between categorical values of a categorical attribute in the dataset. Unlike the existing methods, not only is the pairwise similarity measured between categorical values but also distributed, continuous representation of categorical values is obtained. The proposed approach is evaluated against existing methods in classification and clustering, which are considered commonly performed tasks in data analysis.

The contributions of the study are summarized as follows.

An approach to handling mixed-type data is proposed which not only transforms categorical values to their respective numeric representations but also preserves semantic relationship among categorical values. The approach makes more data analysis algorithms applicable to the transformed datasets.

Extensive experiments are conducted to verify the feasibility and superiority of the proposed approach against other schemes for handling mixed-type datasets.

The issue of hyperparameters setting of the proposed model is investigated, and the proper settings are identified.

The analysis on a real-world mixed-type dataset is presented to demonstrate the application and feasibility of the proposed approach.

The structure of this article is organized as follows. Section 2 reviews related work including traditional schemes for measuring similarity between categorical values and similarity between two mixed-type data instances. Section 3 presents the proposed approach to handling categorical values with preservation of semantics of the values. Extensive experiments are conducted, and the results are presented in Section 4. Conclusions are given in Section 5.

2. Literature review

2.1 Dissimilarity between categorical values

A simple method is to compare two categorical values and yields a dissimilarity zero if the two are the same and otherwise one. Another popular scheme called 1-of- $k$ coding is to convert a categorical attribute with a $k$ -size domain to a set of $k$ binary attributes in which each domain value becomes a binary attribute. Consequently, 1-of- $k$ coding converts each categorical value to a list of $k$ binary values with a value one corresponding the categorical value and the others zero, as shown in Fig. 1. Assume the domain of attribute Drink is {Coke, Pepsi, Espresso, Latte}. In this example, Coke is then encoded by the list [1 0 0 0].

Figure 1.

1-of- $k$ coding converts a categorical value to a binary-value vector.

The drawback of the two methods above is that they do not consider the semantics embedding in the categorical values. Specifically, two distinct values always yield the same dissimilarity, which is counter intuitive. For the example in Fig. 1, ideally, T1 and T2 which both have carbonated drink are more similar to each other than to T3 or to T4 which both have coffee drink. So are T3 and T4. In the first method, the dissimilarity is one while in the 1-of- $k$ coding the dissimilarity is two. Additionally, the 1-of- $k$ coding has a problem with dramatically increased dimensionality when the domain size of the categorical attribute is large or there are many categorical attributes in the dataset.

To address the drawbacks of the aforementioned methods, many measures have been proposed to calculate the degree of similarity between categorical values in a dataset [3, 4]. Note that a similarity value $s$ can be easily converted to a dissimilarity value $d$ . For instance, one common approach is by ( $d=1-s$ ). Boriah et al. [3] categorized similarity measures to three. The measures of the first category gave the minimum value 0 to two mismatch values and different weights to matches, such as Goodall [5] and Gambaryan [6]. The second gave the maximum values 1 to matches and different weights to mismatches, such as Eskin [7], Occurrence Frequency (OF), and Inverse Occurrence Frequency (IOF) [8]. The third gave different weights to both matches and mismatches, such as LIN [9] and SMIRNOV [10]. The authors [3] conducted an exhaustive comparison of 14 similarity measures for outlier detection task on 18 categorical datasets. The experimental results indicated that there is no single best similarity measure suitable for all kinds of datasets or applications.

Ahmad and Dey [11, 12] proposed a dissimilarity measure between two distinct values of a categorical attribute based on how two values co-occur with the values in other attributes. Desai et al. [13] proposed a similarity measure DISC and compared DISC with 14 similarity measures used by Boriah et al. [3] on 24 real-world datasets. Twelve of the datasets were used for classification task while the others were for regression task. DISC performed better than all the compared measures on almost all datasets. Ienco et al. [14] proposed DILCA and compared DILCA with three measures LIN, OF, and GOODALL3 on application to three clustering algorithms, DELTA [12], LIMBO [15], and ROCK [16]. Sixteen datasets including 4 synthetic and 12 real-world datasets were used in the experiments. DILCA was superior to the compared measures on most of the datasets.

Many researchers studied analysis of mixed-type data under the context of data clustering, in which similarity between categorical values is an essential ingredient in the clustering algorithms [17]. Wangchamhan et al. [18] proposed to use Gower distance, based on Gower similarity coefficient [19], and the mechanism for handling categorical attributes. Ding et al. [20, 21] proposed a similarity measure based on entropy for either categorical or numeric attributes so as to have a uniform criterion. Their measure avoided feature transformation and parameter adjustment between categorical and numeric values. Sangam and Om [22] proposed a framework for clustering time evolving mixed data, in which dissimilarity with respect to categorical attribute is measured based on observed frequencies of categorical values. In [23], the authors proposed that clustering of a mixed-type data point to a cluster is based on the object-cluster similarity for mixed data. For comparing the part of categorical attributes to the cluster, the idea is based on the count of the categorical value appearing in the cluster. The more the count, the more similar to the cluster. The same idea also appeared in the work by Huang [24, 25] proposing the $k$ -prototypes algorithm to extend the $k$ -means algorithm [26] for mixed-type datasets.

2.2 Mixed-type data analysis in SOM and dimension reduction

Mixed-type data analysis has also been studied under the context of extending self-organizing map (SOM) [27, 28, 29] and dimensionality reduction [30, 31]. In [27, 28], the authors proposed to use a data structure distance hierarchy for representing dissimilarity between categorical values. Each categorical value is mapped to a node in the hierarchy and then the dissimilarity is measured by the path length between the two nodes representing the categorical values. As shown in Fig. 2, the dissimilarity between Coke and Pepsi is two while four between Coke and Latte, assuming unit length between two nodes. The data structure is then used to generalizing the algorithms of Self-Organizing Maps for handling mixed datasets involving categorical attributes.

Figure 2.

Distance hierarchy for categorical values.

The distance hierarchy can be constructed manually according to knowledge of domain experts [27]. When domain knowledge is not obtainable, distance hierarchy can be constructed according to pairwise dissimilarity between the values learned in a supervised or unsupervised manner with a hierarchical agglomerative clustering [28, 29].

A supervised learning method for pairwise dissimilarity was proposed in [28]. The idea behind the method is that the similarity between two values is dependent on how each of the values co-occurs with the labels in the class attribute [32]. The more two categorical values of an attribute co-occur with the same class value, the more similar they are.

Several unsupervised methods were proposed in [29]. The idea of those methods is mainly that similarity between two values A and B of a categorical attribute is dependent on how each of the values co-occurs with the values of the other feature attributes, referred to as context values. Specifically, for categorical context values, if A and B co-occur with the same context values often, A and B are deemed more similar. As for numeric context values, if the average of the numeric values which A is associated with is close to that which B is associated with, A and B are deemed more similar. The similarities of A and B with respect to each of the feature attributes are measured and then averaged to a single similarity value. According to the idea mentioned above, two implementations UDL1 and UDL2 were proposed and compared [29]. The difference is how the degree of co-occurrence or the conditional probability between the target and a context value is calculated.

MDISC [29] was adapted from DISC [13] which measures similarity between two categorical values via cosine. Based on the idea of DISC, MDISC was devised for measuring dissimilarity between two categorical values of an attribute with respect to the other feature attributes, including numeric and categorical ones. For a numeric context attribute, the dissimilarity between A and B is dependent on the difference between the averages of the values which co-occur with A and with B, respectively. For a categorical context attribute, cosine similarity between A and B with respect to the context attribute is calculated and then transformed to a dissimilarity.

2.3 The research gap

Except for the 1-of- $k$ coding, most of the existing approaches on handling categorical values is to calculate pairwise dissimilarity between the values, like those in Eqs (1)–(3), by comparing the values directly or comparing their relationship with the other attributes of the dataset. A shortcoming of these schemes is that there is no numeric representation or encoding of the categorical values, unlike the 1-of- $k$ coding which represents a categorical value by a binary-value vector. Consequently, some of the data analysis algorithms, such as principal component analysis (PCA) or linear discriminant analysis (LDA), are not applicable to analyze the mixed-type datasets. PCA is based on covariance matrix of the dataset and thus numeric representation of categorical values is needed in order to calculate the covariance between two variables. LDA requires the average and variance of the values of each attribute.

The advantage of having numeric encoding of categorical values is that various quantities such as the average, variance, and covariance of categorical values, which are often required in many data analysis algorithms, can be easily estimated via their numeric representations. The estimation is infeasible or difficult when we have only pairwise dissimilarities between categorical values.

3. Method

3.1 Idea

The idea to tackle the problem is to adopt the techniques from word embedding which can represent a word by a multi-dimensional vector consisting of continuous values as shown in Fig. 3. What is special about the representation is that the semantic similarity between words can be reflected in the word vector space [33]. In other words, the vector representing Coke will be more similar to that representing Pepsi than that representing Latte. Consequently, the distance calculated by using the two representing vectors of Coke and Pepsi will be less than that of Coke and Latte. Therefore, we consider the word embedding techniques can be adapted to address the problem of mixed-type data analysis.

Figure 3.

Word embedding allows a word represented by a multi-dimensional vector.

Most of word embedding techniques are based on the idea that “a word is characterized by the company it keeps” popularized by Firth [34]. Mikolov et al. [33] proposed Word2Vec techniques including two word embedding models, the Continuous Bag of Words (CBOW) and the Skip-gram, for computing vector representation of words from a large size of corpus, which can contain several million up to several billion words. By training a neural network, the CBOW predicts the target word by its context words which are the words around the target in the corpus while the skip-gram model tries to predict the context words of a given word, as shown in Fig. 4. In Fig. 4a for the CBOW model, the word vector of the target word $y$ or “scores” is calculated according to the two context words, as the inputs to the neural network, preceding and following the target word $y$ as the output, respectively. In contrast, in Fig. 4b for the skip-gram model, the word vector of word $x$ or “scores” is calculated according to $x$ as the input and the four context words $y_{i}$ as the outputs of the neural network.

Figure 4.

a. The CBOW model predicts the target word $y$ according to the context words $x_{i}$ . b. The Skip-gram model predicts the context words $y_{i}$ according to the given word $x$ .

Although several pretrained embeddings available as open source APIs which provides access to precalculated vectors of many words, they may not be appropriate to the use of analyzing the mixed-type dataset at hand. The reasons have two. First, the word vectors are precomputed based on a large corpus from a domain which is not specific to the domain of the mixed-type dataset to be analyzed. For example, Gensim word2vec [35], with an interactive web app trained on GoogleNews, includes algorithms on the skip-gram and the CBOW model, using either hierarchical softmax or negative sampling [33, 36]. Wikipedia2Vec [37] learns embeddings of words and entities by iterating over entire Wikipedia pages based on the skip-gram model. Pretrained embeddings for 12 languages, including English, Chinese, French, German, and Japanese, in binary and text format are provided. Second, in many cases, categorical values recorded in mixed-type datasets cannot be found in the pretrained models due to abbreviation, compound words, or customized labels made up by the recording organization. Some of the examples can be seen in Table 1 such as “7th-8th” and “Married-AF-spouse”. We therefore argue that the word vector of a categorical value shall be calculated based on the data in the mixed-type dataset itself.

Inspired by the idea in the paper [33], we can measure the semantic similarity of categorical values in a mixed-type dataset analogously. To adapt the idea of word embedding algorithms in this study, we consider the mixed-type dataset as a corpus, one datapoint in the dataset as a document and then apply word-embedding algorithms for continuous vector representation of categorical values in the mixed-type dataset.

Take the Adult dataset from the UCI machine learning repository [38] as an example. The dataset has 14 feature attributes including 6 continuous attributes,

{age, fnlwgt, education-num, capital-gain, capital-loss, hours-per-week}

and 8 categorical attributes,

{workclass, education, marital-status, occupation, relationship, race, sex, native-country}

and one class attribute indicating whether a person makes over 50K a year.

The Adult dataset consists of 48,842 records and is considered as a corpus, in which each document has the same size, namely, containing 14 words.

3.2 The proposed procedure

Figure 5 shows the proposed procedure which transforms categorical values of a mixed-type dataset to their numeric representations and preserves the semantics of categorical values with respect to their context attributes. Note that the procedure does not involve domain expert or use of other domain data. The transformation of categorical values to their numeric representations by Word2Vec is mainly based on the other attributes in the same dataset.

Figure 5.

A procedure for handling mixed-type datasets in which categorical attribute values are transformed to numeric values via Word2Vec model.

3.2.1 Attribute discretization

To take into consideration of numeric values as context words of the target word, we discretize continuous attributes since occurrence of same numeric values may be limited in a dataset of a small or median size. Without discretization, the co-occurrence of a specific numeric value with the target word might be very low. Moreover, considering close numeric values such as 80 and 81 semantically similar is reasonable and even beneficial for most of the applications. For instance, each of ages 80 $\sim$ 85 may appear only once in the dataset. After discretization, the values can belong to the same discretized value, say, AgeOld, with an increased frequency six, instead of one for each of the values.

Several techniques can be applied for the discretization such as by equal width or equal frequency in individual bins, or clustering-based approaches [39]. The techniques based on equal width or equal frequency have one disadvantage that a group of frequent, close values may be split to different bins, for instance, to discretize two highly frequent values, say, age 84 and 85, into different bins. In this study, we apply a clustering-based approach, kernel density estimation (KDE) [40] to minimize such impact as much as possible.

Figure 6 shows a discretization example by KDE and the result of attribute fnlwgt in the Adult dataset. Specifically, the attribute was first normalized to the range between 0 and 1. Next, the histogram was constructed with the bin size of 0.01. After Gaussian smoothing, it is apparent that there are two local minimums, which are at fnlwgt $=$ 0.0364 and 0.0831, respectively. The attribute was therefore discretized to three groups, namely, fnlwgt0 $=$ [0 0.0364), fnlwgt1 $=$ [0.0364 0.0831), and fnlwgt2 $=$ [0.0831 1]. According to the range of each group, there are 4630, 9772, and 30820 data points belonging to the individual groups, respectively, in dataset Adult. Table 2 demonstrates a discretization sample of numeric attributes of Table 1.

Table 2
Discretization sample of numeric attributes of Table 1

Education	Relationship	Marital-status	Work-Hours_D	Capital-Gain_D	Capital-Loss_D
HS-grad	Unmarried	Never-married	Work-Hours1	Capital-Gain1	Capital-Loss1
Masters	Husband	Married-civ-spouse	Work-Hours2	Capital-Gain4	Capital-Loss3
Doctorate	Wife	Married-AF-spouse	Work-Hours3	Capital-Gain2	Capital-Loss1
Assoc-voc	Own-child	Divorced	Work-Hours1	Capital-Gain3	Capital-Loss2
Masters	Wife	Separated	Work-Hours2	Capital-Gain2	Capital-Loss1
HS-grad	Own-child	Never-married	Work-Hours2	Capital-Gain1	Capital-Loss2
Assoc-voc	Own-child	Divorced	Work-Hours1	Capital-Gain2	Capital-Loss1
HS-grad	Husband	Married-civ-spouse	Work-Hours2	Capital-Gain2	Capital-Loss2
4 ${}^{\text{th}}$ -grad	Wife	Married-civ-spouse	Work-Hours2	Capital-Gain1	Capital-Loss2
Assoc-voc	Own-child	Married-AF-spouse	Work-Hours2	Capital-Gain2	Capital-Loss1
…	…	…	…	…	…

Figure 6.

Discretization of attribute fnlwgt by using kernel density estimation.

3.2.2 Identify relevant attributes for context values

In the applications of natural language processing, the words around the target word are considered context words and used to train neural-network models. The number of context words considered is determined by a user-specified parameter referred to as the window size $w$ . $w$ words before and after the target word are taken as the context words.

In a mixed-type dataset, the attributes are considered order-less, i.e., no specific order among the attributes. The values of the first attribute in the dataset may be related to the values of the last, distant attribute. To identify relevant attributes for context values, correlation analysis can be performed. In this study, we apply Cramer’s V [41] as a measure of association between the target and each of the other attributes. Cramer’s V can be used with variables having two or more levels, namely, distinct values and is defined as follows.

$\displaystyle V\left({A,B}\right)=\sqrt{\frac{X^{2}({A,B})/n}{\min({k-1,r-1})}% }\mbox{ where }X^{2}({A,B})=\mathop{\sum}\limits_{i,j}\frac{\left({n_{ij}-% \frac{n_{i.}n_{.j}}{n}}\right)^{2}}{\frac{n_{i.}n_{.j}}{n}}$ (1)

In Eq. (1), $k$ and $r$ denote the number of distinct values in attributes $A$ and $B$ , respectively. $n$ is the number of data instances. $n_{i}$ is the frequency of value $i$ in $A$ . $n_{j}$ is the frequency of value $j$ in $B$ . $n_{ij}$ is the co-occurrence count of value $i$ and $j$ in $A$ and $B$ , respectively. The association exceeds the user-specified threshold $\theta$ is considered a context attribute.

The pairwise values of the target attribute along with each of the context attributes form the training set of the model. The set of context attributes of attribute $X_{A}$ , denoted by ${CA}(X_{A})$ , and the set of the training samples of $X_{A}$ for the CBOW model, denoted by ${TS}$ , are expressed as shown in Eqs (2) and (3).

$\displaystyle CA({X_{A}})=\{{X_{B}{|}V({X_{A},X_{B}})\geqslant\theta,X_{A}\in X% ,X_{B}\in X}\}$ (2) $\displaystyle TS=\{{({x_{B},x_{A}}){|}x\in X,x_{A}\in X_{A},x_{B}\in X_{B},X_{% B}\in CA({X_{A}})}\}$ (3)

where $x$ is a data point of $X$ and $x_{A}$ is the value of attribute $X_{A}$ of $x$ .

3.2.3 Train Word2Vec model

Figure 7 depicts the CBOW training model. The vocabulary size $V$ denotes the number of distinct values in the transformed dataset. The user-specified parameter $N$ indicates the number of neurons in the hidden layer and also represents the dimension of the word vectors. Each of the neurons in the input and the output layer corresponds to one word in the vocabulary. Given an input ( $x_{i}$ , $y_{j}$ ), say, (Unmarried, HS-grad), from the training set $T S$ , the neuron in the input layer corresponds to $x_{i}$ (i.e., Unmarried) is set to 1 and the others zero. The expected output of the neuron corresponding to word $y_{j}$ (i.e., HS-grad) is 1. During the training process, the difference (or error) between the current and the expected output is used to adjust the weights $W_{V\times N}$ and $W_{N\times V}$ via the backpropagation algorithm. At the end of the training, given the training instance ( $x_{i}$ , $y_{j}$ ) the neuron corresponding to $y_{j}$ is expected to output the largest value, although not 1, compared to the other neurons. After the training, the word vector for word $x_{i}$ in the vocabulary can be retrieved from the weight matrix $W_{V\times N}$ ., specifically, $x_{i}=[w_{i1}\ldots w_{iN}]$ .

Figure 7.

Word embedding with the CBOW model.

3.2.4 Transform categorical values to numeric values

The length of a word vector is determined by the number of neurons in the hidden layer. The length of word vectors in a large corpus is often several hundred. For example, one of the GloVe pre-trained models of word vectors has a vector dimension of 300. The model was trained with 840 billion tokens and a 2.2-million vocabulary. For a mixed-type dataset, the domain size of categorical attributes is often relatively small. For instance, the Adult dataset consist s of 48,842 records and each record contains 14 features. In other words, there are 683,788 values in total in the dataset, which is considered small compared to several billion tokens in the corpus used by GloVe. Therefore, it seems unnecessary to have a large vector length in our application. How length impacts the result of applications such as classification and data clustering is worth investigating.

On the other hand, using a multidimensional word vector to replace the categorical value in the mixed-type data, not only increases the dimension of the dataset but also the weight of the original categorical attribute. To have an equal weight with the other numeric attributes, we shall transform a categorical value to a single numeric value rather than a multidimensional vector. To do so, the simplest way is to set the dimension of the word vector to one. An alternative is to set a multiple dimensionality and then reduce the dimension to one by using dimensionality reduction techniques such as PCA. The idea is that the latter approach might be able to preserve more semantic information than the other one. We will compare the two approaches experimentally.

3.3 Evaluation

For quantitative evaluation, accuracy is used in the classification task. Adjusted mutual information (AMI) and adjusted rand index (ARI) are used to measure quality of clustering. Accuracy is defined as follows.

$\displaystyle\textit{Accuracy}=\frac{1}{|T|}\mathop{\sum}\limits_{i=1}^{|T|}I(% {y_{i}=\hat{y}_{i}})$

where $T$ is the test dataset, $y_{i}$ and $\hat{y}_{i}$ is the true and the predicted class label of data point $i$ , respectively, and $I(\cdot)$ is an indicator function which returns 1 if the argument is true and otherwise 0.

Adjusted mutual information (AMI) and adjusted rand index (ARI) [42] are measured as follows.

$\displaystyle\textit{AMI}({U,V})=\frac{MI({U,V})-E\{{MI({U,V})}\}}{\textit{avg% }\{{H(U),H(V)}\}-E\{{MI({U,V})}\}},$ $\displaystyle MI({U,V})=\mathop{\sum}\limits_{i=1}^{|U|}\mathop{\sum}\limits_{% j=1}^{|V|}P_{UV}({i,j})\log\left({\frac{P_{UV}({i,j})}{P_{U}(i)P_{U}(j)}}% \right),$ $\displaystyle H(U)=\mathop{\sum}\limits_{i=1}^{|U|}P_{U}(i)\log({P_{U}(i)})$

where $MI({U,V})$ , $E\{{MI({U,V})}\}$ and $H(U)$ represent mutual information of distributions $U$ and $V$ , expectation of $MI({U,V})$ , and entropy of $U$ , respectively.

$\displaystyle\textit{ARI}({U,V})=\frac{\mathop{\sum}\nolimits_{ij}\left({{% \begin{array}[]{*{20}c}{n_{ij}}\\ 2\\ \end{array}}}\right)-\left[{\mathop{\sum}\nolimits_{i}\left({{\begin{array}[]{% *{20}c}{a_{i}}\\ 2\\ \end{array}}}\right)\mathop{\sum}\nolimits_{j}\left({{\begin{array}[]{*{20}c}{% b_{j}}\\ 2\\ \end{array}}}\right)}\right]\left/\left({{\begin{array}[]{*{20}c}n\\ 2\\ \end{array}}}\right)\right.}{\frac{1}{2}\left[{\mathop{\sum}\nolimits_{i}\left% ({{\begin{array}[]{*{20}c}{a_{i}}\\ 2\\ \end{array}}}\right)\mathop{\sum}\nolimits_{j}\left({{\begin{array}[]{*{20}c}{% b_{j}}\\ 2\\ \end{array}}}\right)}\right]-\left[{\mathop{\sum}\nolimits_{i}\left({{\begin{% array}[]{*{20}c}{a_{i}}\\ 2\\ \end{array}}}\right)\mathop{\sum}\nolimits_{j}\left({{\begin{array}[]{*{20}c}{% b_{j}}\\ 2\\ \end{array}}}\right)}\right]\left/\left({{\begin{array}[]{*{20}c}n\\ 2\\ \end{array}}}\right)\right.},$

where $n_{ij}$ represents the number of data points in cluster $U_{i}$ as well as $V_{j}$ , and $a_{i}$ and $b_{j}$ denote the numbers of data points of $U_{i}$ and $V_{j}$ , respectively.

4. Experimental results

The proposed approach is compared with other schemes for handling mixed-type datasets on the tasks of classification and clustering on eleven datasets. The results are presented in Section 4.2. We discuss the issues of the hyperparameter setting of the proposed model in the followed section. An example of mixed-type data analysis is given in Section 4.4 to demonstrate the application and feasibility of the proposed approach.

4.1 Datasets

Eleven mixed-type datasets retrieved from the UCI machine learning repository [38] were used for experiments. The datasets include Australian Credit Approval (ACA and pACA), Adult, pAdult, Bank Marketing (Bank and bankAdditional), Contraceptive Method Choice (CMC), Credit Approval (CA) and German Credit Data (GCD), Mushroom, Teaching Assistant Evaluation (TAE). pACA and pAdult, another versions of ACA and Adult, retained only the attributes correlated to their class attribute [27, 28]. The data points with missing values are dropped. Table 3 shows the statistics of the datasets.

Table 3
Statistics of experimental datasets

Dataset	Data points	# of classes	% of major classes	# of categorical	# of numeric
ACA	690	2	55.51	4	10
pACA	690	2	55.51	3	8
Adult	45222	2	75.22	7	6
pAdult	48842	2	76.07	3	4
Bank	45211	2	88.30	7	9
bankAdditional	41188	2	88.73	9	11
CMC	1473	3	42.70	4	5
CA	690	2	55.51	5	10
GCD	1000	2	70.00	11	9
Mushroom	5644	2	61.80	15	6
TAE	151	3	34.44	2	3

4.2 Results

4.2.1 Classification

We apply $k$ -nearest neighbors (KNN) to classify the transformed datasets by word embedding. The proposed approach was compared with five other schemes of handling mixed-type data, including 1-of- $k$ , unsupervised feature learning with fuzzy ART (UFLA) [43], UDL1, UDL2, and MDISC. The last three proposed in [29] are able to measure the pairwise dissimilarity between two mixed-type data points.

We set $k$ to the odd values in between 1 and 19 and reported the best result. Each dataset was randomly divided to the training and the test dataset with the ratio of 80% and 20%, respectively. KNN was run five times for each $k$ and the average of accuracies on the test dataset was calculated.

Table 4 shows classification performance on the datasets and the scheme of Word2Vec outperformed the others. The coding method 1-of- $k$ performed worst among all the schemes. It is interesting that all schemes achieved 100% accuracy on dataset Mushroom which is an easy dataset.

Table 4
Comparison of classification accuracy

Dataset	Word2Vec	1-of- $k$	UFLA	UDL1	UDL2	MDISC
ACA	0.8913	0.8507	0.8783	0.8696	0.8638	0.8638
pACA	0.8870	0.8609	0.8841	0.8594	0.8797	0.8797
Adult	0.8485	0.8326	0.8315	0.8240	0.8326	0.8374
pAdult	0.8477	0.8392	0.8432	0.8370	0.8359	0.8359
Bank	0.8966	0.8945	0.8958	0.8881	0.8936	0.8959
BankAdditional	0.9092	0.8996	0.8997	0.9009	0.9024	0.9001
CMC	0.5444	0.4644	0.4895	0.5078	0.4956	0.5044
CA	0.8899	0.8638	0.8681	0.8710	0.8812	0.8710
GCD	0.7610	0.7420	0.7490	0.7230	0.7470	0.7290
Mushroom	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
TAE	0.7333	0.6333	0.6600	0.7000	0.5667	0.6067
Average	0.8372	0.8074	0.8181	0.8164	0.8090	0.8113

4.2.2 Clustering

We use $k$ -means++ [44] to perform clustering on the data transformed with those various schemes for handling categorical attributes and compare the results. Moreover, $k$ -prototypes [24] and COBWEB/3 [45] which can cluster mixed-type data are also compared. With random initialization of $k$ -cluster centers, the result can be different. We therefore run $k$ -means++ five times and then take the average of the performances. The number of clusters are set to 2 to 20. The best result of each scheme is taken and compared. The attribute values or the pairwise dissimilarities are normalized to the range between zero and one. The Euclidean distance is used to measure the distance between data points.

Tables 5 and 6 show the performance measured by adjusted mutual information (AMI) and adjusted rand index (ARI), respectively. It can be observed that the word embedding scheme outperformed the others on most of the datasets. Similar to the classification results, the coding method 1-of- $k$ performed worst among all the schemes in the data clustering task.

4.3 Hyperparameters

In the following, we analyze how the setting of hyperparameters of the Word2Vec models impact the applications to classification and clustering. The investigation issues include the length of word vectors, comparison of CBOW and skip-gram models, comparison of negative sampling and hierarchical selection, and relevance of context words.

Table 5
Clustering performance with measure AMI

Dataset	Word2Vec	1-of- $k$	UFLA	UDL1	UDL2	MDISC	$k$ -prototypes	Cobweb/3
ACA	0.4304	0.1250	0.2983	0.4273	0.4273	0.4273	0.4273	0.1741
pACA	0.4273	0.1613	0.2877	0.4273	0.4273	0.4273	0.4273	0.1744
Adult	0.1938	0.1705	0.1731	0.1010	0.1721	0.1717	0.0991	0.1091
pAdult	0.1801	0.1710	0.1930	0.1339	0.1732	0.1732	0.1317	0.1392
Bank	0.0524	0.0286	0.0829	0.0237	0.0211	0.0196	0.0184	0.0165
BankAdditional	0.1462	0.0518	0.1180	0.0745	0.0801	0.0801	0.0803	0.0141
CMC	0.0473	0.0392	0.0444	0.0352	0.0400	0.0356	0.0275	0.0180
CA	0.4273	0.1257	0.2598	0.4273	0.4273	0.4273	0.3733	0.3726
GCD	0.0443	0.0324	0.0363	0.0153	0.0238	0.0118	0.0232	$-$ 0.0008
Mushroom	0.4826	0.4819	0.4683	0.4505	0.4485	0.4644	0.3939	0.4224
TAE	0.1371	0.0555	0.0901	0.0768	0.0768	0.0948	0.0699	0.0563
Average	0.2335	0.1312	0.1865	0.1993	0.2107	0.2121	0.1884	0.1360

Table 6

Clustering performance with measure ARI

Dataset	Word2Vec	1-of- $k$	UFLA	UDL1	UDL2	MDISC	$k$ -prototypes	Cobweb/3
ACA	0.5077	0.1225	0.3661	0.5036	0.5036	0.5036	0.5036	0.2360
pACA	0.5036	0.1440	0.3729	0.5036	0.5036	0.5036	0.5036	0.2360
Adult	0.1989	0.1768	0.1872	0.0557	0.182	0.1798	0.0641	0.1634
pAdult	0.2454	0.1771	0.2180	0.1842	0.1851	0.1849	0.0487	0.1669
Bank	0.1544	0.0125	0.2322	0.0233	0.0233	0.0233	0.0233	0.0013
BankAdditional	0.3249	0.0164	0.2576	0.1167	0.1309	0.1309	0.1315	0.0044
CMC	0.0460	0.0429	0.0447	0.0221	0.0377	0.0244	0.0225	0.0193
CA	0.5036	0.1212	0.2970	0.5036	0.5036	0.5036	0.4462	0.4514
GCD	0.0921	0.0355	0.0490	0.0174	0.0146	0.014	0.0084	$-$ 0.0030
Mushroom	0.5233	0.4748	0.4767	0.4836	0.5172	0.4895	0.4310	0.3034
TAE	0.1078	0.0265	0.0725	0.0602	0.0602	0.0602	0.0477	0.0505
Average	0.2916	0.1227	0.2340	0.2249	0.2420	0.2380	0.2028	0.1481

4.3.1 Length of word vectors

A categorical attribute in the mixed-type dataset shall be replaced by a single numeric attribute rather than a multi-dimensional vector. To do so, we compared two approaches. The first approach is to set the number of neurons in the hidden layer to ten, obtain word vectors with the length of ten, and then applydimension reduction technique PCA on the word vectors to obtain one-dimensional values. The second is to set the number of neurons in the hidden layer to one and obtain one-dimensional word vectors directly.

Table 7 presents the result. The last row shows the count that one approach outperformed the other among the experimental datasets. The result indicates that the approach of training a multi-dimensional vector first and then applying dimension reduction yielded better outcomes on most of the datasets.

Table 7
Performance comparison between different sizes of word vectors. The last row indicates the count of the parameter setting outperforming the other

Dataset	Classification		Clustering
	Accuracy		AMI		ARI
	10 $\rightarrow$ 1	1	10 $\rightarrow$ 1	1	10 $\rightarrow$ 1	1
ACA	0.8797	0.8667	0.4298	0.4273	0.5069	0.5036
pACA	0.8725	0.8667	0.4273	0.4273	0.5036	0.5036
Adult	0.8377	0.8278	0.1725	0.0929	0.1823	0.0215
pAdult	0.8408	0.8346	0.1732	0.0880	0.1835	0.0474
Bank	0.8938	0.8898	0.0279	0.0184	0.0233	0.0233
BankAdditional	0.9021	0.8987	0.0801	0.0803	0.1312	0.1312
CMC	0.5058	0.5112	0.0427	0.0304	0.0361	0.0242
CA	0.8855	0.8493	0.2053	0.4273	0.1813	0.5036
GCD	0.7260	0.7140	0.0191	0.0155	0.0139	0.0153
Mushroom	1.0000	1.0000	0.3774	0.3782	0.3261	0.2655
TAE	0.6133	0.7000	0.1159	0.0734	0.0872	0.0449
No. of outperforming	8	2	7	3	6	2

Figure 8 and Table 8 demonstrate the projection to one-dimensional space of the three categorical attributes of dataset pAdult. It is clear that the projection of the first approach (i.e., 10 $\rightarrow$ 1) matches the intuition of people. For the example in Fig. 8a referring to the Education attribute in Table 8 as well, the values representing high degrees of education are projected in the right region of the one-dimensional space. This order of the values is not seen in the projection by the second approach, denoted by “1”. For the projection of attribute Marital_status shown in Fig. 8b, by the first approach, two points in the right are away from the others. The two are Married-AF-spouse and Married-civ-spouse. Similarly, in Fig. 8c, by the first approach, the two values, Wife and Husband, are in the far right away from the other four, namely, Unmarried, Own-child, Not-in-family, and Other-relative. Again, such orders did not present in the projections by the second approach.

Table 8

Attribute values corresponding to the projection results in Fig. 8 from the left to the right

Attribute	Method	Attribute values from left to right in Fig. 8
Education	10 $\rightarrow$ 1	11th, 12th, 10th, Preschool, 9th, 1st-4th, 5th-6th, 7th-8th, Some-college, HS-grad, Assoc-acdm, Assoc-voc, Bachelors, Masters, Doctorate, Prof-school
	1	Prof-school, Doctorate, 1st-4th, Preschool, 11th, Masters, 12th, 9th, 5th-6th, Bachelors, 10th, Some-college, 7th-8th, Assoc-acdm, HS-grad, Assoc-voc
Marital_status	10 $\rightarrow$ 1	Widowed, Separated, Divorced, Never-married, Married-spouse-absent, Married-AF-spouse, Married-civ-spouse
	1	Married-AF-spouse, Widowed, Married-spouse-absent, Never-married, Married-civ-spouse, Separated, Divorced
Relationship	10 $\rightarrow$ 1	Unmarried, Own-child, Not-in-family, Other-relative, Wife, Husband
	1	Unmarried, Other-relative, Not-in-family, Wife, Own-child, Husband

A Word2Vec model can be trained with hierarchical softmax or negative sampling. We compared the performance between the two settings. Table 9 shows that for classification task, negative sampling and hierarchical softmax did not have significant difference in term of superiority while for clustering task negative sampling outperformed hierarchical selection.

Table 9

Performance comparison between negative sampling (NS) and hierarchical softmax (HS). The last row indicates the count of the parameter setting outperforming the other

Dataset	Classification		Clustering
	Accuracy		AMI		ARI
	NS	HS	NS	HS	NS	HS
ACA	0.8797	0.8580	0.4298	0.4273	0.5069	0.5036
pACA	0.8725	0.8739	0.4273	0.4273	0.5036	0.5036
Adult	0.8377	0.8362	0.1725	0.1724	0.1823	0.1823
pAdult	0.8406	0.8408	0.1799	0.1732	0.2449	0.1835
Bank	0.8938	0.8891	0.0279	0.0279	0.0233	0.0233
BankAdditional	0.9037	0.9021	0.0802	0.0801	0.1314	0.1312
CMC	0.5058	0.4902	0.0427	0.0405	0.0361	0.0271
CA	0.8783	0.8855	0.2846	0.2053	0.3196	0.1813
GCD	0.7260	0.7060	0.0191	0.0126	0.0139	0.0087
Mushroom	1.0000	1.0000	0.4464	0.3774	0.2892	0.3261
TAE	0.6133	0.6667	0.1159	0.1091	0.0872	0.0966
No. of outperforming	6	4	9	0	6	2

Figure 8.

One-dimensional projection results of attributes (a) Education, (b) Marital_status, and (c) Relationship of dataset pAdult after word embedding.

4.3.2 CBOW and skip-gram models

There are several word embedding models such as Word2Vec, fastText [46], GloVe [47], etc. Do different models matter? Word2Vec estimates continuous representations of words by training neural networks to predict a word from its context words (i.e., the CBOW model) or to predict the context words of a given word (i.e., the skip-gram model) [33]. In this study, we compared the two models of Word2Vec, namely, CBOW and skip-gram.

Table 10 shows that for classification task, CBOW achieved superior performance than the skip-gram model while for clustering task the difference is not significant.

Table 10
Performance comparison between CBOW and skip-gram (SG) models

Dataset	Classification		Clustering
	Accuracy		AMI		ARI
	CBOW	SG	CBOW	SG	CBOW	SG
ACA	0.8797	0.8797	0.4298	0.4273	0.5069	0.5036
pACA	0.8725	0.8667	0.4273	0.4273	0.5036	0.5036
Adult	0.8377	0.8350	0.1725	0.1178	0.1823	0.1015
pAdult	0.8408	0.8417	0.1732	0.1794	0.1835	0.2407
Bank	0.8938	0.8874	0.0279	0.0276	0.0233	0.0233
BankAdditional	0.9021	0.8998	0.0801	0.0741	0.1312	0.1158
CMC	0.5058	0.5003	0.0427	0.0445	0.0361	0.0419
CA	0.8855	0.8536	0.2053	0.4273	0.1813	0.5036
GCD	0.7260	0.7230	0.0191	0.0185	0.0139	0.0096
Mushroom	1.0000	1.0000	0.3774	0.3908	0.3261	0.4114
TAE	0.6133	0.6800	0.1159	0.1027	0.0872	0.0716
No. of outperforming	7	2	6	4	5	4

4.3.3 Context words

In the applications of natural language processing, the CBOW or skip-gram models use a window size to determine the size of context words with respect to the target word. In mixed-type datasets, the attributes are considered order-less. Adjacent attributes do not imply semantically more related to the target attribute. In this study, we used Cramer’s V to measure the degree of association between attributes. If the association degree of a feature attribute to the target attribute exceeds a pre-determined threshold, the attribute is considered a context attribute.

Table 11 shows the experimental results with the threshold set to 0 and 0.05, respectively. It can be seen that in most of the cases, using a threshold to select context attributes could improve both in classification and clustering tasks.

4.4 Visualized analysis of mixed-type data

We present an example of visualized analysis of mixed-type data using dataset pAdult. In Fig. 9, categorical attributes of the dataset were handled with the Word2Vec scheme. Then, the dimension of the data was reduced to 2 by the supervised dimensionality reduction technique linear discriminant analysis (LDA) and plotted on a two-dimensional plane. In Fig. 10, categorical attributes were handled with the conventional method 1-of- $k$ and then LDA was applied to the transformed dataset. We like to note that if categorical attributes are handled with the schemes UDL1, UDL2, or MDISC [29], LDA cannot be applied due to the lack of numeric representations of categorical values with those schemes. However, SOM (self-organizing map) can be another option to consider in the clustering process.

Table 11
Performance comparison between the threshold of attribute relevance

Dataset	Classification		Clustering
	Accuracy		AMI		ARI
	0	0.05	0	0.05	0	0.05
ACA	0.8710	0.8797	0.4273	0.4298	0.5036	0.5069
pACA	0.8638	0.8725	0.4273	0.4273	0.5036	0.5036
Adult	0.8360	0.8377	0.1724	0.1725	0.1823	0.1823
pAdult	0.8412	0.8408	0.1793	0.1732	0.2421	0.1835
Bank	0.8883	0.8938	0.0285	0.0279	0.0233	0.0233
BankAdditional	0.9012	0.9021	0.0803	0.0801	0.1315	0.1312
CMC	0.5247	0.5058	0.0371	0.0427	0.0239	0.0361
CA	0.8768	0.8855	0.1916	0.2053	0.1707	0.1813
GCD	0.7170	0.7260	0.0138	0.0191	0.0137	0.0139
Mushroom	0.9996	1.0000	0.3722	0.3774	0.3055	0.3261
TAE	0.6533	0.6133	0.1162	0.1159	0.0860	0.0872
No. of outperforming	3	8	4	6	2	6

We used linking and brushing technique [48] and developed a function to allow an analyst to perform interactive clustering and analysis on the projection results. As shown in Figs 9 and 10, the projection results were clustered to 5 and 2 clusters, respectively, and the data points in each cluster were colored according to whether its class label is “ $>$ 50K” or “ $<=$ 50K”.

Figure 9.

Projection and clustering of dataset pAdult with Word2Vec by LDA.

Referring to Figs 9 and 10, it is apparent that handling categorical attributes with Word2Vec yielded better projection result than that with 1-of- $k$ . The clusters are better separated. Moreover, referring to Table 12, the data points in C1 and C3 all have class label “ $>$ 50K” and the largest cluster C4 has a very small ratio (i.e., 0.06) of data points having class label “ $>$ 50K”.

Table 12

The average of the numeric attributes and the mode of the categorical attributes in each cluster with respect to the projection shown in Fig. 9

C	Age	Gain	Loss	Hrs	Education	Marital_status	Relationship	Class	Ratio	Count
1	48	99999	0	51	Prof-school	Married-civ-spouse	Husband	$>$ 50K	1.00	196
3	42	99999	0	49	Bachelors	Never-married	Not-in-family	$>$ 50K	1.00	48
2	44	1713	199	45	Bachelors	Married-civ-spouse	Husband	$>$ 50K	0.45	9743
	43	212	61	42	HS-grad	Married-civ-spouse	Husband	$<=$ 50K	0.55	12108
5	36	1739	177	42	Some-college	Married-civ-spouse	Other-relative	$>$ 50K	0.16	59
	33	150	40	38	HS-grad	Married-civ-spouse	Other-relative	$<=$ 50K	0.84	308
4	43	3687	188	47	Bachelors	Never-married	Not-in-family	$>$ 50K	0.06	1641
	34	115	51	37	HS-grad	Never-married	Not-in-family	$<=$ 50K	0.94	24739
All	44	4042	193	45	Bachelors	Married-civ-spouse	Husband	$>$ 50K	0.24	11687
	37	147	54	39	HS-grad	Never-married	Not-in-family	$<=$ 50K	0.76	37155

Table 13

The average of the numeric attributes and the mode of the categorical attributes in each cluster with respect to the projection shown in Fig. 10

C	Age	Gain	Loss	Hrs	Education	Marital_status	Relationship	Class	Ratio	Count
1	41	3155	192	38	Bachelors	Married-civ-spouse	Wife	$>$ 50K	0.46	1083
	40	193	40	35	HS-grad	Married-civ-spouse	Wife	$<=$ 50K	0.54	1225
2	45	4210	190	46	Bachelors	Married-civ-spouse	Husband	$>$ 50K	0.23	10604
	36	142	57	39	HS-grad	Never-married	Not-in-family	$<=$ 50K	0.77	35930
All	44	4042	193	45	Bachelors	Married-civ-spouse	Husband	$>$ 50K	0.24	11687
	37	147	54	39	HS-grad	Never-married	Not-in-family	$<=$ 50K	0.76	37155

Figure 10.

Projection and clustering of dataset pAdult with 1-of- $k$ by LDA.

Although C1 and C3 all have class label “ $>$ 50K” and similar characteristics in some attributes, they have significant differences in the others. Specifically, both clusters have very large Capital_gain, zero Capital_loss, and large Hrs (Work_hours_per_week). However, the average Age in C1 is much larger than that in C3. The largest portion of Education in C1 is Prof-school (professional school). The largest portions of Marital_status and Relationship in C3 are Never-married and Not-in-family, respectively, in contrast to Married-civ-spouse and Husband in C1.

Unlike Fig. 9, not many clusters can be identified in Fig. 10. As shown in Table 13, the largest cluster C2 has a ratio of 0.23 with class label “ $>$ 50K”. That ratio is nearly the same with that of the original dataset, i.e., 0.24. On the contrary, Table 12 shows that each cluster in Fig. 9 has a ratio of class label “ $>$ 50K” considerably different from that of the original dataset. In other words, data points are better clustered with respect to the class label when the categorical values are handled with word embedding Word2Vec.

5. Conclusions

To facilitate data analysis on mixed-type datasets involving categorical attributes, categorical values in the datasets must be handled properly. The traditional method 1-of- $k$ coding transforming a categorical value to a list of binary values does not preserve semantic relationship between categorical values. Other methods measured pairwise (dis)similarity degree between categorical values. Using such methods does not generate numeric representation of categorical values for the datasets. As a result, some data analysis algorithms cannot be applied such as principal component analysis (PCA) and linear discriminant analysis (LDA). In this paper, we proposed an approach to transforming individual categorical values to numeric values via word embedding techniques. The advantage of the proposed approach is two-fold. First, having numeric representations of categorical values allows more data analysis techniques applicable to the transformed datasets. Second, the numeric representations preserve semantic relationship between the categorical values based on co-occurrence with the values in the other attributes, leading to more meaningful results of data analysis. Experimental results demonstrate the proposed approach is feasible and yields superior performance compared to other approaches. In the future, other word embedding techniques such as fastText and GloVe will be investigated and compared with the CBOW and the skip-gram models of Word2Vec.

Footnotes

Acknowledgments

This work is partially supported by Ministry of Science and Technology, Taiwan under grant MOST 109-2410-H-224-019, and by the “Intelligent Recognition Industry Service Center” from The Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan.

References

Pearson

, On lines and planes of closest fit to systems of points in space, Philosophical Magazine 2(11) (1901), 559–572.

Fisher

R.A.

, The use of multiple measurements in taxonomic problems, Annals of Eugenics 7(2) (1936), 179-188.

Boriah

Chandola

and Kumar

, Similarity Measures for Categorical Data: A Comparative Evaluation, Paper presented at the Proceedings of SDM ’08, Atlanta, Georgia, USA, 2008.

Ahmad

and Khan

S.S.

, Survey of state-of-the-art mixed data clustering algorithms, IEEE Access, 2019. doi: 10.1109/ACCESS.2019.2903568.

Goodall

D.W.

, A new similarity index based on probability, Biometrics 22(4) (1966), 882–907.

Gambaryan

, A mathematical model of taxonomy, Izvest. Akad. Nauk Armen. SSR 17(12) (1964), 47–53.

Eskin

Arnold

Prerau

Portnoy

and Stolfo

, A geometric framework for unsupervised anomaly detection, Paper presented at the Applications of Data Mining in Computer Security, 2002.

Jones

K.S.

, A statistical interpretation of term specificity and its application in retrieval, in: Document Retrieval Systems, Volume 3 of Taylor Graham Series In Foundations Of Information Science, Taylor Graham Publishing, London, UK, 1988, pp. 132–142.

Lin

, An information-theoretic denition of similarity, in: Paper Presented at the 15th International Conference on Machine Learning, San Francisco, CA, USA, 1998.

10.

Smirnov

E.S.

, On exact methods in systematics, Systematic Zoology 17(1) (1968), 1–13.

11.

Ahmad

and Dey

, A k-mean clustering algorithm for mixed numeric and categorical data, Data & Knowledge Engineering 63(2) (2007a), 503–527.

12.

Ahmad

and Dey

, A method to compute distance between two categorical values of same attribute in unsupervised learning for categorical data set, Pattern Recognition Letters 28(1) (2007b), 110–118.

13.

Desai

Singh

and Pudi

, Disc: Data-intensive similarity measure for categorical data, in: Paper Presented at the Pacific-Asia Conference on Knowledge Discovery and Data Mining, 2011.

14.

Ienco

Pensa

R.G.

and Meo

, From context to distance: Learning dissimilarity for categorical data clustering, ACM Transactions on Knowledge Discovery from Data (TKDD) 6(1) (2012), 1.

15.

Andritsos

Tsaparas

Miller

R.J.

and Sevcik

K.C.

, LIMBO: Scalable clustering of categorical data, in: Paper Presented at the International Conference on Extending Database Technology, 2004.

16.

Guha

Rastogi

and Shim

, ROCK: A robust clustering algorithm for categorical attributes, Information Systems 25(5) (2000), 345–366.

17.

Foss

A.H.

Markatou

and Ray

, Distance metrics and clustering methods for mixed-type data, International Statistical Review, 2018.

18.

Wangchamhan

Chiewchanwattana

and Sunat

, Efficient algorithms based on the k-means and chaotic league championship algorithm for numeric, categorical, and mixed-type data clustering, Expert Systems with Applications 90 (2017), 146–167.

19.

Gower

J.C.

, A general coefficient of similarity and some of its properties, Biometrics 27(4) (1971), 857–871.

20.

Ding

Sun

and Xue

, An entropy-based density peaks clustering algorithm for mixed type data employing fuzzy neighborhood, Knowledge-Based Systems 133 (2017), 294–313.

21.

Ding

and Xue

, A novel density peaks clustering algorithm for mixed data, Pattern Recognition Letters 97 (2017), 46–53.

22.

Sangam

R.S.

and Om

, Equi-clustream: A framework for clustering time evolving mixed data, Advances in Data Analysis and Classification 12(4) (Dec 2018), 973–995.

23.

Jia

and Cheung

Y.M.

, Subspace clustering of categorical and numerical data with an unknown number of clusters, IEEE Transactions on Neural Networks and Learning Systems 99 (2017), 1–18.

24.

Huang

, Clustering large data sets with mixed numeric and categorical values, in: Paper Presented at the Proceedings of the 1st Pacific-Asia Conference on Knowledge Discovery and Data Mining, 1997, (PAKDD).

25.

Huang

, Extensions to the k-means algorithm for clustering large data sets with categorical values, Data Mining and Knowledge Discovery 2(3) (1998), 283–304.

26.

MacQueen

, Some methods for classification and analysis of multivariate observations, in: Paper Presented at the Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 1967.

27.

Hsu

C.C.

, Generalizing self-organizing map for categorical data, IEEE Transactions on Neural Networks 17(2) (2006), 294–304. (SCI)

28.

Hsu

C.C.

and Lin

S.H.

, Visualized analysis of mixed numeric and categorical data via extended self-organizing map, IEEE Transactions on Neural Networks and Learning Systems 23(1) (2012), 72–86.

29.

Hsu

C.C.

Kung

C.H.

Jheng

J.J.

and Chang

C.Y.

, Unsupervised distance learning for extended self-organizing map and visualization of mixed-type data, Intelligent Data Analysis 23(4) (2019), 799–823.

30.

Hsu

C.C.

and Huang

W.H.

, Integrated dimensionality reduction technique for mixed-type data involving categorical values, Applied Soft Computing 43 (June 2016), 199–209.

31.

Hsu

C.C.

and Wu

J.W.

, Visualized mixed-type data manalysis via dimensionality reduction, Intelligent Data Analysis 22 (2018), 981–1007.

32.

Das

Mannila

and Ronkainen

, Similarity of attributes by external probes, Knowledge Discovery and Data Mining (1998), 23–29.

33.

Mikolov

Chen

Corrado

and Dean

, Efficient estimation of word representations in vector space, 2013a, arXiv preprint arXiv:1301.3781.

34.

Firth

J.R.

, A synopsis of linguistic theory 1930–1955, Studies in Linguistic Analysis: 1–32, Reprinted in Palmer

F.R.

, ed. (1968), Selected Papers of J.R. Firth 1952–1959, London: Longman, 1957.

35.

Gensim, https://radimrehurek.com/gensim/models/word2vec.html, retrieved in Dec. 22, 2019.

36.

Mikolov

Chen

Corrado

and Dean

, Distributed Representations of Words and Phrases and their Compositionality, 2013b, arXiv:1310.4546v1.

37.

Wikipedia2vec, https://wikipedia2vec.github.io/wikipedia2vec/pretrained/, retrieved in Dec. 22, 2019.

38.

Dua

and Graff

, UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science, 2019.

39.

Tan

P.N.

Steinbach

Karpatne

and Kumar

, Introduction to Data Mining, 2

{}^{\text{nd}}

ed., Pearson, New York, 2019.

40.

Parzen

, On estimation of a probability density function and mode, The Annals of Mathematical Statistics 33(3) (1962), 1065–1076.

41.

Cramér

, Mathematical Methods of Statistics, Princeton University Press, 1946.

42.

Vinh

N.X.

Epps

and Bailey

, Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance, The Journal of Machine Learning Research 11 (2010), 2837–2854.

43.

Lam

Wei

and Wunsch

, Clustering data of mixed categorical and numerical type with unsupervised feature learning, IEEE Access 3 (2015), 1605–1613.

44.

Arthur

and Vassilvitskii

, k-means++: The advantages of careful seeding, Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, 2007.

45.

McKusick

and Thompson

, COBWEB/3: a portable implementation, Technical Report FIA–90–6–18–2, NASA Ames Research Center, Moffett Field, 1990.

46.

Bojanowski

Grave

Joulin

and Mikolov

, Enriching word vectors with subword information, Transactions of the Association for Computational Linguistics 5 (2017), 135–146.

47.

Pennington

Socher

and Manning

C.D.

, GloVe: Global Vectors forWord Representation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, (EMNLP), Oct. 2014, pp. 1532–1543.

48.

Buja

McDonald

Michalak

and Stuetzle

, Interactive data visualization using focusing and linking, in: Proceedings IEEE Visualization ’91, 1991, pp. 156–163.

Analyzing mixed-type data by using word embedding for handling categorical features

Abstract

Keywords

1. Introduction

Table 1 Portion of the Adult mixed-type dataset

2.1 Dissimilarity between categorical values

3. Method

3.1 Idea

Table 2 Discretization sample of numeric attributes of Table 1

3.3 Evaluation

4. Experimental results

4.1 Datasets

Table 3 Statistics of experimental datasets

4.2.1 Classification

Table 4 Comparison of classification accuracy

4.3 Hyperparameters

Table 5 Clustering performance with measure AMI

Table 7 Performance comparison between different sizes of word vectors. The last row indicates the count of the parameter setting outperforming the other

Table 10 Performance comparison between CBOW and skip-gram (SG) models

4.4 Visualized analysis of mixed-type data

Table 11 Performance comparison between the threshold of attribute relevance

Footnotes

Acknowledgments

References

Table 1
Portion of the Adult mixed-type dataset

Table 2
Discretization sample of numeric attributes of Table 1

Table 3
Statistics of experimental datasets

Table 4
Comparison of classification accuracy

Table 5
Clustering performance with measure AMI

Table 7
Performance comparison between different sizes of word vectors. The last row indicates the count of the parameter setting outperforming the other

Table 10
Performance comparison between CBOW and skip-gram (SG) models

Table 11
Performance comparison between the threshold of attribute relevance