Dynamic sparsity control in Deep Belief Networks

Abstract

A Deep Belief Network (DBN) is a generative probabilistic graphical model that contains many layers of hidden variables and has excelled among deep learning approaches. DBN can extract suitable features, but improving these networks for obtaining features with more discrimination ability is an important issue. One of the important improvements is sparsity in hidden units. In sparse representation, we have the property that learned features can be interpreted, i.e., correspond to meaningful aspects of input, and are more efficient. One of the main problems in sparsity techniques is to find the best hyper-parameters values which need dozens of experiments to obtain them. In this paper, a dynamic hyper-parameter value setting is proposed for resolving this problem. This proposed method does not need to set parameters manually. According to the results, our new dynamic method achieves acceptable recognition accuracy on test sets in different applications, including image, speech and text. According to these experiments, the proposed method can find hyper-parameters dynamically without losing much accuracy.

Keywords

Deep Belief Network Restricted Boltzmann Machine normal sparse rbm dynamic sparsity control sparse feature

1. Introduction

Nowadays, the use of deep networks has been highly considered by researchers as one of the most powerful tools in machine learning in different applications [3]. Although using neural networks has been common from many years ago, the increase in computing power and new techniques in machine learning, have provided the possibility of using these networks with more layers.

Among different methods of deep learning techniques [3], Deep Belief Network (DBN) [7] is one of the most famous and most efficient techniques. The DBN uses a greedy layer-wise unsupervised learning algorithm [2], where each layer is a Restricted Boltzmann Machine (RBM). Although each layer in a DBN can extract useful features, new researchers use sparsity methods in hidden layers for extracting more beneficial features with more generalization [1].

Sparsity methods are used to decrease activation values of hidden units [6]. Since different methods are proposed as sparsity methods to be used in an RBM, these methods are highly dependent on the specified hyper parameters [11], and finding the best hyper parameters is very time consuming. In this paper, we propose a new method for dynamic adjustment of hyper parameters. This new method resolves the issue of finding the best hyper parameters. The proposed method uses the normal sparse RBM method and finds its hyper-parameters dynamically, while the other sparse methods need dozens of experiments to find the best hyper-parameters.

The rest of this paper is organized as follows: in Section 2, RBM and DBN are described. The related works about sparse RBM methods are in Section 3 and our proposed method is presented in Section 4. In Section 5, some experiments are conducted and the proposed method is compared with some other methods in the tasks of digit recognition on the MNIST dataset, spoken letter recognition on the ISOLET dataset and document topic classification on the 20 Newsgroups dataset. Finally, Section 6 concludes the paper.

2. Background

DBNs are composed of multiple layers of RBMs. Outputs of hidden layers in each RBM can be considered as input for the next RBM layer. As shown in Fig. 1, with this method, DBN will be trained layer by layer [7].

Figure 1.

Stack of RBM’s in which the samples from the lower-level RBM are used as the data for training the next RBM [18].

An RBM is a Markov random field with two groups of hidden and visible units (see Fig. 2). In RBMs, each neuron is connected to all the neurons in the other layer. However, there are no connections between neurons in the same layer [5]. This restriction in RBMs causes conditional independence between visible and hidden units.

Figure 2.

An example of Restricted Boltzmann Machines [19].

The joint probability distribution under the model uses an energy function of $\{\mathrm{v,h\}}$ .

$\displaystyle E\left(\bm{v},\bm{h}\right)=-\bm{v}^{T}\bm{Wh}-\bm{a}^{T}\bm{v}-% \bm{b}^{T}\bm{h}=-\sum_{i=1}^{g_{v}}\sum\limits_{j=1}^{g_{h}}w_{ij}v_{i}h_{j}-% \sum\limits_{i=1}^{g_{v}}a_{i}v_{i}-\sum\limits_{j=1}^{g_{h}}b_{j}h_{j}$ (1)

where $w_{ij}$ represents the symmetric interaction term between visible unit $i$ and hidden unit $j$ , while $b_{i}$ and $a_{j}$ are bias terms for hidden units and visible units, respectively.

According to the energy function, the joint probability distribution of the RBM model with visible and hidden units is defined as follows:

$\displaystyle P\left(\bm{v}\right)=\sum\limits_{\bm{h}}{P\left(\bm{v},\bm{h}% \right)}=\frac{1}{Z}\sum\limits_{\bm{h}}\exp\left(-E\left(\bm{v},\bm{h}\right)\right)$ (2)

where $Z$ is the partition function or normalization constant. Since there are no connections between neurons in the same layer, conditional distributions are factorial and are given by:

$\displaystyle P\left(h_{j}=1|{\bm{v}}\right)=g\left(b_{j}{+}\sum\limits_{i}{v_% {i}w_{ij}}\right)$ (3) $\displaystyle P\left(v_{i}=1|{\bm{h}}\right)=g\left(a_{i}\mathrm{+}\sum\limits% _{j}{h_{j}w_{ij}}\right)$ (4)

where ${g(.)}$ is the logistic sigmoid function [15].

Training in RBMs maximizes probability distribution in training data with respect to the model parameters:

$\displaystyle\textit{maximize}_{\left\{w_{ij},a_{i}\mathrm{,}b_{j}\right\}}% \frac{{1}}{m}\sum\limits_{l{=1}}^{m}\log\left(P\left(\bm{v}^{\left(l\right)}% \right)\right)$ (5)

where the parameter $m$ is the number of training data samples and $\bm{W}$ , $\bm{a}$ and $\bm{b}$ are RBM parameters. To maximize probability distribution, these RBM parameters need to be learnt. Due to the presence of $Z$ , gradient calculation is not possible. Therefore, sampling methods are used in gradient calculation.

$\displaystyle{-}\frac{\partial\log{P\left(\bm{v}\right)}}{\partial w_{ij}}{=<}% v_{i}h_{j}{>}_{data}{-<}v_{i}h_{j}{>}_{model}$ (6)

The expectation ${<}v_{i}h_{j}{>}_{data}$ is the expectation observed in the training set but ${<}v_{i}h_{j}{>}_{model}$ is the expectation under the distribution defined by the model. Since computing ${<}v_{i}h_{j}{>}_{model}$ is extremely time-consuming to be computed exactly, the Contrastive Divergence (CD) approximation is used instead [7].

3. Related work

Different methods are proposed to build sparse RBMs [8, 11, 13, 17]. Essentially, RBMs learn non-sparse distributed representations. In all the proposed methods, the learning algorithm in RBM has been changed to enforce RBM to learn sparse representation. For this purpose, a regularization term ( $L_{\textit{sparsity}}$ ) is defined to reduce the average activation probability on the training data. This regularization ensures that the “firing rate” of the model neurons (corresponding to the latent random variables $h_{j}$ ) are kept at a fairly low level, so that the activations of the model neurons are sparse [13]. Thus, given a training set $\bm{v}^{(1)},\mathellipsis,\bm{v}^{(m)}$ comprising $m$ examples, the optimization problem in Eq. (5) will be changed by adding the regularization term:

$\displaystyle\textit{maximize}_{\left\{w_{ij},a_{i},b_{j}\right\}}\frac{1}{m}% \sum\limits_{l=1}^{m}\log\left(P\left(\bm{v}^{\left(l\right)}\right)\right)+% \lambda L_{\textit{sparsity}}$ (7)

where $\lambda$ is a regularization constant (we call it “sparsity cost”). To achieve the maximum of Eq. (7), gradient computation for the gradient ascent method is necessary. As a result of issues in gradient computation in the log likelihood term, an efficient approximation to the gradient of the log likelihood such as CD, PCD or FEPCD can be used. Afterwards, the gradient in $L_{\textit{sparsity}}$ can be computed according to its equation independently. Therefore, in each iteration we can apply the approximation to the gradient of the log likelihood update rule, followed by one step of gradient descent using the gradient of the regularization term. Below is a pseudocode for description of the sparse RBM learning algorithm.

Update the parameters (W, a, b) using approximation to the gradient of the log likelihood like CD, PCD or FEPCD.

Update the parameters (W, a, b) using the gradient of the regularization term.

Repeat steps 1 and 2 until convergence or reach the max epoch.

Different researchers have used different regularization terms. One of them penalizes quadratically as a deviation of the expected activation of the hidden units from a (low) fixed level $p$ named “sparsity target” [14]. In this paper we call this method quadratic sparse RBM or qsRBM. This regularization term is presented as follows:

$\displaystyle L_{\textit{sparsity}}=-\sum\limits_{j=1}^{n}\left|p-\frac{1}{m}% \sum\limits_{l=1}^{m}\mathbb{E}[h^{(l)}_{j}|\bm{v}^{(l)}]\right|^{2}$ (8)

where $\mathbb{E}\left[.\right]$ is the conditional expectation on the $j$ th hidden unit given the data. $p$ is a constant controlling the sparseness of the $n$ hidden units $h_{j}$ .

In another state of the art paper based on rate distortion theory, the penalty factor is the activation probability of hidden units [8]. We call the method using this penalty factor the rate distortion sparse RBM or rdsRBM method. This regularization term is presented as follows:

$\displaystyle L_{\textit{sparsity}}=-\sum\limits_{l=1}^{m}\left\|P\left(\bm{h}% ^{\left(l\right)}|\bm{v}^{\left(l\right)}\right)\right\|_{1}$ (9)

where $\left\|.\right\|_{1}$ is the norm of its input.

4. Dynamic sparse RBM

Our proposed method uses normal probability density function as the regularization term [11]. The normal regularization term has different behavior according to deviation of the activation of the hidden units from a (low) fixed level $p$ . We call this method the normal sparse RBM or nsRBM. The normal regularization term is presented as follows:

$\displaystyle L_{\textit{sparsity}}=\sum\limits_{j=1}^{n}{f(q_{j},p,\sigma^{2})}$ (10)

where $f$ is a normal probability density function, $q_{j}$ is the average of conditional expectation on the $j$ th hidden unit given the data, $p$ is a constant (low) fixed level that controls the sparseness and $\sigma^{2}$ is variance.

$\displaystyle f(q_{j},p,\sigma^{2})=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}% \left(\frac{q_{j}-p}{\sigma}\right)^{2}}$ (11)

The normal regularization term has a parameter that can control the force degree of sparseness. According to Fig. 3 by different variance values, we can control the penalty value or degree of sparseness.

Figure 3.

Normal pdf with ${p=0.1}$ and different variance values [11].

Therefore, the RBM parameters are updated with the regularization term based on the following equations by computing the gradient of our regularization term [11]:

$\displaystyle\frac{\partial}{\partial w_{ij}}L_{\textit{sparsity}}=\frac{% \partial\left(f\left(q_{j},p,\sigma^{2}\right)\right)}{\partial w_{ij}}=\frac{% \partial\left(f\left(q_{j},p,\sigma^{2}\right)\right)}{\partial q_{j}}\times% \frac{\partial\left(q_{j}\right)}{\partial I}\times\frac{\partial\left(I\right% )}{\partial w_{ij}}$ (12)

where $I_{j}{=}b_{j}{+}\sum\limits_{i}{v_{i}w_{ij}}$ is equal to the sum of inputs to hidden unit $j$ .

$\displaystyle\frac{\partial\left(f\left(q_{j},p,\sigma^{2}\right)\right)}{% \partial q_{j}^{\left(l\right)}}=\frac{\left(p-q_{j}\right)}{\sigma^{2}}f\left% (q_{j},p,\sigma^{2}\right)$ (13) $\displaystyle\frac{\partial\left(q_{j}\right)}{\partial I}\times\frac{\partial% \left(I\right)}{\partial w_{ij}}=\frac{1}{m}\sum\limits_{l=1}^{m}{q_{j}^{\left% (l\right)}\left(1-q_{j}^{(l)}\right)v_{i}^{\left(l\right)}}$ (14)

where $q_{j}^{(l)}={g}\left(I_{j}\right)$ is equal to activation probability of hidden unit $h_{j}$ given $\bm{v}$ and ${g}$ as the logistic function. Therefore, the RBM parameters are updated with the regularization term based on the following equations:

$\displaystyle\frac{\partial}{\partial w_{ij}}L_{\textit{sparsity}}\propto\frac% {1}{m}\left(p-\frac{1}{m}\sum\limits_{l=1}^{m}q_{j}^{\left(l\right)}\right)f% \left(q_{j},p,\sigma^{2}\right)\sum\limits_{l=1}^{m}{q_{j}^{\left(l\right)}% \left(1-q_{j}^{\left(l\right)}\right)v_{i}^{\left(l\right)}}$ (15)

Similarly, for hidden bias we have the following equation:

$\displaystyle\frac{\partial}{\partial b_{j}}L_{\textit{sparsity}}\propto\frac{% 1}{m}\left(p-\frac{1}{m}\sum\limits_{l=1}^{m}q_{j}^{\left(l\right)}\right)f% \left(q_{j},p,\sigma^{2}\right)\sum\limits_{l=1}^{m}{q_{j}^{\left(l\right)}% \left(1-q_{j}^{\left(l\right)}\right)}$ (16)

The main problem in all of the described sparsity methods is to find the appropriate values for hyper-parameters in the sparsity regularization term. Based on experiments, all these methods are very sensitive to hyper-parameters and finding appropriate values for them is very time consuming [11]. For example, in normal sparse RBM, three hyper-parameters (sparsity cost, sparsity target and variance) must be used and finding appropriate values for them requires about 100 different runs of experiments.

In this paper, based on features of nsRBM we proposed a new dynamic method for adjusting hyper-parameters. This method does not need different experiments to find appropriate values of hyper-parameters. We call this method the dynamic normal sparse RBM or dynNsRBM.

The first hyper-parameter in nsRBM is sparsity target ( $p$ ). The sparsity target indicates that the expected activation of the hidden units should reach this desired value. In the proposed method, sparsity target value is set to a proportional amount of average of hidden unit activations in each batch dynamically. Thus, the sparsity target is always less than the average of the activation of the hidden units.

$\displaystyle p=\eta\times\frac{1}{m\times n}\sum\limits_{l=1}^{m}\sum\limits_% {j=1}^{n}q_{j}^{\left(l\right)}$ (17)

where $m$ and $n$ are the number of batch data samples and hidden units respectively, $q_{j}^{\left(l\right)}$ is the activation probability of a hidden unit $h_{j}$ given $\bm{v}^{(l)}$ and $\eta$ is a constant value in an interval [0,1]. In this dynamic parameter adjustment, the sparsity method is not sensitive to an exact value of $\eta$ and must only be set to a value less than 1. In all experiments and in all different datasets we have used the value of 0.75 for $\eta$ . Indeed, this parameter guarantees the sparsity target to be always less than the average of the activation of the hidden units and therefore, the gradient of $L_{\textit{sparsity}}$ can reduce the activation of the hidden units.

The other hyper-parameter in the nsRBM method is the variance of normal function ( $\sigma^{2})$ in $L_{\textit{sparsity}}$ . In the proposed method, the variance of hidden unit activations in each batch has been used dynamically. This dynamicity in determination of the parameter causes the sparsity regularization term to cover all possible values of hidden unit activations well.

$\displaystyle\sigma^{2}=\frac{1}{m\times n}\sum\limits_{l=1}^{m}\sum\limits_{j% =1}^{n}{(q_{j}^{\left(l\right)}-\mu)}$ (18)

where $\mu$ is the expected value, i.e.

$\displaystyle\mu=\frac{1}{m\times n}\sum\limits_{l=1}^{m}\sum\limits_{j=1}^{n}% q_{j}^{\left(l\right)}$ (19)

The last hyper-parameter in the nsRBM method is sparsity cost ( $\lambda$ ) that we set it to a constant value proportional to the learning rate. This value is proportional to the learning rate because the learning rate in Gaussian RBM is different from the binary RBM [6]. These different values of learning rates in different RBMs affect the amount of gradient in RBMs and we must use the gradient of $L_{\textit{sparsity}}$ proportional to it. In addition, according to the features of nsRBM, the effect of $L_{\textit{sparsity}}$ can be changed dynamically by changing the sparsity target and variance and therefore we do not require changing the sparsity cost dynamically.

Below is a pseudocode for the description of the dynamic sparse RBM method. For simplicity, computation of some parameters such as momentum and weight decay have been excluded.

According to the pseudocode, in the first stage some parameters of RBM learning will be initialized. In this initialization, $\bm{W}$ represents the weights between visible units and hidden units, while $\bm{b}$ and $\bm{a}$ are bias terms for hidden units and visible units, respectively. Also, according to [9] 30 Markov Chains are used to generate negative samples.

Then in the main loop for training dynamic sparse RBM, positive and negative samples are obtained. These samples are used for computing the gradient of log likelihood term in Eq. (7). Afterwards, the gradient of $L_{\textit{sparsity}}$ in Eq. (7) is computed using the normal sparse RBM method to decrease the activation of hidden units. This stage uses Eqs (17) and (18) in each batch to compute hyper parameters. Indeed, in each batch, the hyper parameters are obtained in such a way that average activation in hidden units tends to a small value. Finally, the computed gradient modifies the RBM parameters.

Program parameters:

•

Setting learning rate and sparsity cost ( $\lambda$ ) according to type of RBMs.

Initialization:

•

Initialize $\bm{W}$ to small random values and initialize $\bm{a}$ and $\bm{b}$ to zero.

•

Initialize the 30 Markov Chains $v^{-}$ to all random states for FEPCD sampling method.

Then repeat:

Get the next batch of training data, $v^{+}$ .

Calculate $h^{+}=P(h|v^{+},Parameters_{\textit{RBM}}$ ). Calculate the positive gradient $g^{+}=v^{+T}h^{+}$ .

Calculate $h^{-}$ and $v^{-}$ from Markov Chains according to FEPCD sampling method [9]. Calculate the negative gradient $g^{-}=v^{-^{T}}h^{-}$ .

Update the parameters ( $\bm{W}$ , $\bm{a}$ , $\bm{b})$ using FEPCD approximation to the gradient of the log likelihood. ( $g_{\log{\textit{likelihood}}}=g^{+}-g^{-})$

Calculate the dynamic sparse parameters ( $m$ and $n$ are the number of batch data samples and hidden units respectively).

$\displaystyle\mu=\frac{1}{m\times n}\sum\limits_{l=1}^{m}\sum\limits_{j=1}^{n}% q_{j}^{\left(l\right)}$ $\displaystyle p=0.75\times\frac{1}{m\times n}\sum\limits_{l=1}^{m}\sum\limits_% {j=1}^{n}q_{j}^{\left(l\right)}$ $\displaystyle\sigma^{2}=\frac{1}{m\times n}\sum\limits_{l=1}^{m}\sum\limits_{j% =1}^{n}{(q_{j}^{\left(l\right)}-\mu)}$

Update the parameters ( $\bm{W}$ , $\bm{b}$ ) using the gradient of the $L_{\textit{sparsity}}$ term according to Eqs (15) and (16).

For a better understanding of the dynamic sparsity method, a histogram of activation probability of hidden units and its regularization term on the ISOLET dataset have been shown in Fig. 4. Although the average activation probability of hidden units are about 0.5, the normal function parameters have been set dynamically for the average activation probability of hidden units to tend towards zero (by adjusting sparsity target less than the average activation probability of hidden units). Also, the selected variance can have an appropriate effect on all hidden units.

Figure 4.

Histogram of activation probability of hidden units and its regularization term. Although the average activation probability of hidden units are about 0.5, the normal function parameters have been set dynamically for the average activation probability of hidden units to tend towards zero (by adjusting sparsity target to less than the average activation probability of hidden units).

The reduction in the average activation probability of hidden units and the dynamic change of regularization terms during learning are shown in Fig. 5. According to this figure, the normal function has been moved toward zero due to a decrease in the activation probability of hidden units during learning. It should be noted that in a classic RBM, the activation probability of hidden units does not change.

Figure 5.

Reduction in average activation probability of hidden units and dynamic change of regularization term during learning. In this figure, the normal function has been moved to zero due to decreasing activation probability of hidden units during learning.

5. Results

The method proposed in this paper was evaluated on different datasets with different applications and input types. These datasets are in the tasks of digit recognition on the MNIST dataset with discrete feature values between 0 and 255, spoken letter recognition on the ISOLET dataset with real feature values and document topic classification on the 20 Newsgroups dataset with binary feature values. In these experiments, we used the DeeBNet toolbox [10] (the toolbox implemented by the authors). In addition, we used the new FEPCD method [17] to approximate the gradient of the log likelihood.

5.1 MNIST dataset

MNIST1

¹
Available online at “http://yann.lecun.com/exdb/mnist/”.

dataset includes images of handwritten digits [12] (10 classes of digits 0–9). Each digit is carefully located in the center of each 28*28 image. The image pixels have discrete values between 0 and 255, most of which have the values at the edge of this interval [16]. The dataset was divided into train and test parts including 60,000 and 10,000 images, respectively. In our experiments, these discrete values have been mapped to the interval [0–1] using the min-max normalization method.

For better comparison and to demonstrate the superiority of this method in feature learning application, several experiments were conducted. In these experiments, different sparsity methods (qsRBM and rdsRBM, nsRBM and dynNsRBM) are compared with classic RBM, principal component analysis and raw features. Also, all models have been trained on 20,000 training samples of the MNIST dataset (similar to [8]) to extract 500 features from visible data (except in raw features where all input features are used).

Now with these learnt models, features of 10, 20, 50, 100, 500 and 1000 images per class are used in training a linear classifier. The reason for using a simple linear classifier is for a better comparison between the strength of learnt features. Also, for better classifier evaluation, image selection in each class and linear classifier training were done twenty times separately and the average results were reported.

Figure 6.

Digit recognition error rates on the MNIST dataset on training (with 10, 20, 50, 100, 500 and 1000 samples per class) and test sets obtained by a linear classifier that was trained on raw data, transformed codes produced by PCA and active probability of hidden units produced by RBM, qsRBM, rdsRBM, nsRBM and dynNsRBM.

In Fig. 6, the recognition performance based on raw data (784 features), transformed codes produced by PCA (500 features), RBM, qsRBM, rdsRBM, nsRBM and dynNsRBM (all with 500 hidden units) are compared. In this figure, the results of the qsRBM, rdsRBM and nsRBM are presented according to best parameter combinations that were obtained by dozens of experiments [11]. For brevity, only the best model results are reported. In dynNsRBM, the parameters were obtained dynamically.

According to Fig. 6, interesting results can be observed. The first interesting result is that using all types of RBMs in feature learning, the error rate decreases considerably. This improvementin the small number of training samples is more evident. The second interesting result is that all types of sparse RBMs have extracted better features with more discrimination and they have less error rate than classic RBM. The third interesting result is that even though finding the hyper parameter values in the proposed method is dynamic, the performance is close to the best results in nsRBM and in some cases the error is less. However, it is noteworthy that these error rates are achieved by dozens of experiments to find the best hyper-parameter values, but in the proposed method, the hyper-parameter values are obtained dynamically at runtime.

Figure 7.

Digit recognition error rates on MNIST training (with 10, 20, 50, 100, 500 and 1000 samples per class) and test sets obtained by a linear classifier that was trained on active probability of hidden units produced by the first and second layers in DBN, qsDBN, rdsDBN, nsDBN and dynNsDBN.

Since a DBN can learn higher levels of representation in its layers [1], in the next experiments, a layer with 100 hidden neurons has been added to the last models (RBM, qsRBM, rdsRBM, nsRBM and dynNsRBM). After learning new models with their learning methods, a feature vector of learnt features in the first and second layers has been used for training the linear classifier (500 and 100 features in the first and second layers, respectively). Figure 7 depicts the error rate on the test set. According to Fig. 7, it can be seen that all types of sparse DBNs have extracted better features with more discrimination and they have less error rates than the classic DBN. In addition, the proposed method performance is close to the best results in nsDBN and in some cases, the error is less.

Figure 8.

Spoken letter recognition error rates on ISOLET dataset on training (with 10, 20, 50 and 100 samples per class) and test sets obtained by a linear classifier that was trained on raw data, transformed codes produced by PCA and active probability of hidden units produced by RBM, qsRBM, rdsRBM, nsRBM and dynNsRBM.

5.2 ISOLET dataset

In the ISOLET2

²
Available online at “https://archive.ics.uci.edu/ml/datasets/ISOLET”.

dataset, each of the 150 speakers utters each of the 26 letters of the alphabet twice. Hence, there are 52 training examples from each speaker. The speakers are grouped into sets of 30 speakers each, and are referred to as isolet1, isolet2, isolet3, isolet4, and isolet5. The training data appears in the isolet1

+

+

+

4 data file in sequential order, i.e. first the speakers from isolet1, then from isolet2, and so on. The test set, isolet5, is a separate file. Due to three missing examples, there are 7797 examples in total, referred to as isolet1-isolet5 (6238 training examples and 1559 test examples). The feature vector has 617 features including spectral coefficients, contour features, sonorant features, pre-sonorant features, and post-sonorant features [4]. Since features have real values, the Gaussian visible units are used [6].

Similar to the MNIST test, different sparsity methods (qsRBM and rdsRBM, nsRBM and dynNsRBM) are compared with classic RBM, principal component analysis and raw features. All models were trained on 3,640 feature vectors (i.e. 140 feature vectors per class) of the ISOLET dataset to extract 500 features from visible data (except in raw features where all input features were used).

Now with these learnt models, features of 10, 20, 50 and 100 samples per class are used in training a linear classifier. For each combination of training data size and algorithm, we trained 20 classifiers with randomly chosen training sets for the linear classifier and then used the average classification error to evaluate the performance of the corresponding algorithm.

In Fig. 8, the recognition performance based on raw data (617 features), transformed codes produced by PCA (500 features), RBM, qsRBM, rdsRBM, nsRBM and dynNsRBM (all with 500 hidden units) are compared. In this figure, the results of the qsRBM, rdsRBM and nsRBM are presented according to best parameter combinations obtained by dozens of experiments. In dynNsRBM, the parameters were obtained dynamically. According to these results, even though finding the hyper parameter values in the proposed method is dynamic, the performance is close to the other sparse techniques.

Similar to experiments done on MNIST, in order to compare the recognition ability of representations learnt by DBN, qsDBN, rdsDBN, nsDBN and dynNsDBN, we trained a layer with 100 hidden neurons. After learning new models with their learning methods, a feature vector of learnt features in the first and second layers was used for training the linear classifier (500 and 100 features in the first and second layer, respectively). Figure 9 depicts the error rates on the test set. According to Fig. 9, it can be seen that even though finding the hyper parameter values in the proposed method is dynamic, the performance is close to the other sparse techniques.

Figure 9.

Spoken letter recognition error rates on ISOLET training (with 10, 20, 50 and 100 samples per class) and test sets, obtained by a linear classifier that was trained on active probability of hidden units produced by the first and second layers in DBN, qsDBN, rdsDBN, nsDBN and dynNsRBM.

5.3 20 Newsgroups dataset

The 20 Newsgroups3

³
Available online at “http://qwone.com/ $\sim$ jason/20Newsgroups”.

dataset is organized into 20 different news groups, each corresponding to a different topic. The 20 newsgroups dataset has become a popular dataset for experiments in text applications of machine learning techniques, such as text classification and text clustering.

For a text classification experiment, we used a version of the 20 Newsgroups4

⁴

Available online at “http://qwone.com/ $\sim$ jason/20Newsgroups/20news-bydate-matlab.tgz”.

with 18774 documents in which the training and test sets (60% and 40%, respectively) contain documents collected at different times. We used the 5000 most frequent words for binary input features. Since the features have binary values, the binary visible units are used.

Figure 10.

Document topic classification error rates on 20 Newsgroups dataset on training (with 10, 20, 50, 100, 200 and 350 samples per class) and test sets obtained by a linear classifier that was trained on raw data, transformed codes produced by PCA and active probability of hidden units produced by RBM, qsRBM, rdsRBM, nsRBM and dynNsRBM.

Similar to the MNIST test, different sparsity methods (qsRBM and rdsRBM, nsRBM and dynNsRBM) are compared with classic RBM, principal component analysis and raw features. All models are trained on 7,500 feature vectors (i.e. 375 feature vectors per class) of the 20 Newsgroups dataset to extract 500 features from visible data (except in raw features where all input features are used).

Figure 11.

Document topic classification error rates on 20 Newsgroups dataset on training (with 10, 20, 50, 100, 200 and 350 samples per class) and test sets, obtained by a linear classifier that was trained on active probability of hidden units produced by the first and second layer in DBN, qsDBN, rdsDBN, nsDBN and dynNsDBN.

Now with these learnt models, features of 10, 20, 50,100, 200 and 350 feature vectors per class are used in training a linear classifier. For each combination of training data size and algorithm, we trained 20 classifiers with randomly chosen training sets for the linear classifier and then used the average classification error to evaluate the performance of the corresponding algorithm.

In Fig. 10, the recognition performance based on raw data (5000 features), transformed codes produced by PCA (500 features), RBM, qsRBM, rdsRBM, nsRBM and dynNsRBM (all with 500 hidden units) are compared. In this figure, the results of the qsRBM, rdsRBM and nsRBM are presented according to the best parameter combinations that were obtained by dozens of experiments. However, in dynNsRBM, the parameters were obtained dynamically. According to the results, almost for all types of RBMs used in feature learning, the error rate decreases considerably. This improvementin the small number of training samples is more evident and even though finding the hyper parameter values in the proposed method is dynamic, the performance is close to the other sparse techniques.

Similar to experiments done on MNIST, in order to compare the recognition ability of representations learnt by DBN, qsDBN, rdsDBN, nsDBN and dynNsDBN, we trained a layer with 100 hidden neurons. After learning new models with their learning methods, a feature vector of learnt features in first and second layers were used for training the linear classifier (500 and 100 features in first and second layers respectively). Figure 11 displays the error rate on the test set. According to Fig. 11, it can be seen that even though finding the hyper parameter values in the proposed method is dynamic, the performance is close to the other sparse techniques.

In all of RBM experiments, we performed two-sample t-test and Wilcoxon test scores for our new method and nsRBM results to show that the average of error in our new method and Normal Sparse RBM method are similar.

6. Conclusion

In this paper, we presented a new sparse RBM method. Different methods are proposed to build sparse RBMs. The main problem in all of these sparsity methods is to find the appropriate values of hyper-parameters in sparsity regularization term. Based on experiments, these methods are very sensitive to hyper-parameters and finding their appropriate values is very time consuming.

In the proposed method in this paper, based on the features of nsRBM, we proposed a new dynamic method for adjusting hyper-parameters that does not require different experiments to find the appropriate values of hyper-parameters.

According to the results, in our new method, finding the hyper parameter values is dynamic and the performance is close to the best results in other sparse methods and in some cases, the error is less. However, it is noteworthy to mention that error rates on other sparse methods are obtained by dozens of experiments to find the best hyper-parameter values, but in the proposed method, the hyper-parameter values are obtained dynamically at runtime.

References

Bengio

, Learning deep architectures for AI, Found Trends Mach Learn 2 (2009), 1–127.

Bengio

Lamblin

Popovici

and Larochelle

, Greedy layer-wise training of deep networks, Advances in Neural Information Processing Systems 19 (2007), 153.

Deng

and Yu

, Deep Learning: Methods and applications, Foundations and Trends® in Signal Processing 7 (2014), 197–387.

Fanty

M.A.

and Cole

R.A.

, Spoken Letter Recognition, In: Advances in Neural Information Processing Systems, 1991, pp. 220–226.

Fischer

and Igel

, Training restricted Boltzmann machines: An introduction, Pattern Recognition 47 (2014), 25–39. doi: 10.1016/j.patcog.2013.05.025.

Hinton

, A practical guide to training restricted boltzmann machines, Machine Learning Group, University of Toronto, 2010.

Hinton

G.E.

Osindero

and Teh

Y.-W.

, A fast learning algorithm for deep belief nets, Neural Computation 18 (2006), 1527–1554.

N.-N.

Zhang

J.-S.

and Zhang

C.-X.

, A sparse-response deep belief network based on rate distortion theory, Pattern Recognition 47 (2014), 3179–3191.

Keyvanrad

M.A.

and Homayounpour

M.M.

, Deep Belief Network Training Improvement Using Elite Samples Minimizing Free Energy, arXiv:1411.4046 [cs], 2014.

10.

Keyvanrad

M.A.

and Homayounpour

M.M.

, A brief survey on deep belief networks and introducing a new object oriented MATLAB toolbox (DeeBNet), arXiv:1408.3264 [cs], 2014.

11.

Keyvanrad

M.A.

and Homayounpour

M.M.

, Normal sparse Deep Belief Network, In: 2015 International Joint Conference on Neural Networks (IJCNN), 2015, pp. 1–7.

12.

LeCun

and Cortes

, MNIST handwritten digit database, AT&T Labs [Online]. Available: http://yann.lecun.com/ exdb/mnist, 1998.

13.

Lee

, Unsupervised Feature Learning Via Sparse Hierarchical Representations, PHD Thesis, Stanford University, 2010.

14.

Lee

Ekanadham

and Ng

, Sparse deep belief net model for visual area V2, Advances in Neural Information Processing Systems 20 (2008), 873–880.

15.

Mohamed

Dahl

and Hinton

, Deep belief networks for phone recognition, In: NIPS Workshop on Deep Learning for Speech Recognition and Related Applications, Canada, 2009, pp. 1–9.

16.

Nair

, Visual Object Recognition Using Generative Models of Images, PHD Thesis, University of Toronto, 2010.

17.

Nair

and Hinton

, 3D object recognition with deep belief nets, Advances in Neural Information Processing Systems 22 (2009), 1339–1347.

18.

Salakhutdinov

, Learning deep generative models, PHD Thesis, University of Toronto, 2009.

19.

and Deng

, Automatic Speech Recognition, Springer London, London, 2015.

Dynamic sparsity control in Deep Belief Networks

Abstract

Keywords

1. Introduction

2. Background

5.1 MNIST dataset

1 Available online at “http://yann.lecun.com/exdb/mnist/”.

2 Available online at “https://archive.ics.uci.edu/ml/datasets/ISOLET”.

3 Available online at “http://qwone.com/ ∼ jason/20Newsgroups”.

References

¹
Available online at “http://yann.lecun.com/exdb/mnist/”.

²
Available online at “https://archive.ics.uci.edu/ml/datasets/ISOLET”.

³
Available online at “http://qwone.com/ $\sim$ jason/20Newsgroups”.