Adversarial training for few-shot text classification

Abstract

In recent years, Deep Learning methods have become very popular in classification tasks for Natural Language Processing (NLP); this is mainly due to their ability to reach high performances by relying on very simple input representations, i.e., raw tokens. One of the drawbacks of deep architectures is the large amount of annotated data required for an effective training. Usually, in Machine Learning this problem is mitigated by the usage of semi-supervised methods or, more recently, by using Transfer Learning, in the context of deep architectures. One recent promising method to enable semi-supervised learning in deep architectures has been formalized within Semi-Supervised Generative Adversarial Networks (SS-GANs) in the context of Computer Vision. In this paper, we adopt the SS-GAN framework to enable semi-supervised learning in the context of NLP. We demonstrate how an SS-GAN can boost the performances of simple architectures when operating in expressive low-dimensional embeddings; these are derived by combining the unsupervised approximation of linguistic Reproducing Kernel Hilbert Spaces and the so-called Universal Sentence Encoders. We experimentally evaluate the proposed approach over a semantic classification task, i.e., Question Classification, by considering different sizes of training material and different numbers of target classes. By applying such adversarial schema to a simple Multi-Layer Perceptron, a classifier trained over a subset derived from 1% of the original training material achieves 92% of accuracy. Moreover, when considering a complex classification schema, e.g., involving 50 classes, the proposed method outperforms state-of-the-art alternatives such as BERT.

Keywords

Semi-supervised learning generative adversarial network kernel-based embedding spaces universal sentence encoding

1 Introduction

One of the drawbacks of complex deep architectures is the large amount of annotated data required for an effective training. This means relying on large-scale annotated material to lead a neural network to learn suitable representations in its intermediate (i.e. hidden) layers. Unfortunately, annotated material is often scarce in many languages and domains. This is mainly due to the large costs and time consumption required to annotate new instances of the targeted phenomena. Unlabeled examples, instead, are less expensive, and typically more abundant. When only a few examples are available, deep learning methods are not competitive: this setting is usually known as few-shot learning.

Usually, in Machine Learning, this problem is mitigated by the usage of semi-supervised methods [5] or, more recently, using Transfer Learning mechanisms [27]. Semi-supervised learning methods aim at improving the generalization capability of a learner when few labeled data is available, while the acquisition of unlabeled sources is possible. With Transfer Learning, an already pre-trained architecture is exploited to transfer the knowledge embedded in its weights to a new task.

In Computer Vision, one recent method for semi-supervised learning in deep architectures is formalized within the so-called Semi-Supervised Generative Adversarial Networks (SS-GANs) framework. Generative Adversarial Networks (GANs) [18, 30] are a class of neural generative models based on game theory. The goal of GANs is to train a generator network $G$ in producing samples resembling the ones from a given data distribution. The training of $G$ “adversarially” depends on a discriminator network $D$ , trained to distinguish samples produced by the generator from those characterizing instances of the real data distribution. $G$ is trained to fool $D$ into accepting its outputs as being real. SS-GAN [30] is an extension of GANs where $D$ is devoted to both assigning a category to each example and discriminating whether it was generated by $G$ .

In SS-GANs, the training is extended to use unlabeled examples. While the labeled material is used to train the classifier, the unlabeled one improves the inner representations of $D$ . The aim is to let $D$ i) recognize the examples generated by $G$ , ii) assign the correct category to the labeled examples, iii) do not confuse the unlabeled examples with the ones coming from $G$ . Results within Image Classification have been shown SS-GANs to be very effective: when exposed to a few dozens of labeled examples (but thousands of unlabeled ones), an SS-GAN is capable of obtaining a quality similar to state-of-the-art convolutional neural networks. However, the usage of GANs in natural language problems has been limited so far; this is mainly due to the discrete input space typically adopted in this domain, i.e., raw words for which propagating gradients can be difficult.

In weakly supervised problems within NLP tasks, exploiting linguistic information can be beneficial. For example, syntactic information has been demonstrated to be useful for many language understanding tasks [29]. Recently, many approaches have been proposed for producing vector representations of sentences that try to capture syntactic or semantic information. In [7], the authors showed how to successfully embed linguistically motivated kernel functions into low-dimensional embeddings using the Nyström method [36]. In [9], the authors showed that these linguistically motivated embeddings are effective also within deep architectures, in the so-called Kernel-based Deep Architecture (KDA). The authors demonstrated how the syntactic information captured by a Semantic Tree Kernel can be embedded into vectors to be used as input for neural network learning.

A different approach to encode sentences is proposed in [4], where the semantics of short texts is captured in vector spaces, where the geometric distances reflect pretty well the semantic similarity of sentences. The Universal Sentence Encoder (USE, proposed in [4]), is trained over very large datasets; when vectors generated by a USE are used in input to deep architectures, these are trained by reaching very good performances in different natural language inference (NLI) tasks.

It is worth noting that the above methods are not designed to solve specific tasks. They are specifically designed to properly encode texts in metric spaces where the underlying distance successfully reflects linguistic generalizations of lexical and grammatical properties. Such properties are thus induced from unlabeled textual material to model relations between sentences and texts. However, to tackle a new NLI task, e.g. a sentence classification task, a specific (potentially deep) neural architecture needs to be trained. Whereas the above methods provide linguistically meaningful inputs to the target networks, a large labeled dataset is still mandatory, as it embodies knowledge about the classification task. The greater the number of required annotations, the higher the cost of applying the targeted neural classifier.

In this paper, we investigate how to improve the applicability of deep architectures by exploiting the encoding of rich linguistic information in expressive metric spaces, fully capitalizing a semi-supervised learning perspective. The aim is to improve the robustness of deep architectures against too small annotated data sets, by exploiting unlabeled material properly encoded in a linguistically expressive vector space. We will study the impact of syntactic encoding of textual data as presented in [4, 9]. In particular, we will use the information contained in a Semantic Tree Kernel, a semantic similarity metrics sensitive to grammatical and lexical information. The embedding of such linguistic evidence is obtained by using the Nyström method and used to feed a Multi-Layer Perceptron (MLP). In this work, we will augment that reference learning architecture through the adoption of the SS-GAN framework. An adversarially trained generator network $G$ is coupled with the target MLP classifier, in line with what we’ve experimented in [8]. The MLP will thus act as a discriminator $D$ , designed to separate fake examples from the real ones and, at the same time, to assign the correct class to each real example. $G$ , instead, will be devoted to the generation of fake examples with the aim of “fooling” the discriminator, by creating examples as much similar as possible to the Nyström kernel approximation. We expect that the fake and unlabeled examples will boost the capability of $D$ in improving its inner representation of input examples. These networks will be trained simultaneously so that $G$ is penalized when fake examples are “unmasked” by $D$ , while the latter will be penalized when fake examples are incorrectly assigned to one of the k classes.

In addition to our previous work presented in [8], in this paper we also show how a similar schema is effective with representations acquired from the USE. It allows us to verify the capability of the SS-GAN framework to deal with a different kind of representation not specifically focusing on syntactic information. Moreover, we show that the information provided by Tree Kernels and USE are complementary and that their combination provides further improvements in the semi-supervised setting provided by the SS-GAN. The resulting architecture enables semi-supervised learning in linguistically rich spaces. We will evaluate this framework on two text classification tasks, both based on the Question Classification TREC dataset. We will show that our approach is beneficial both when using the coarse-grained setting of QC, i.e., on 6 output categories, and when using the fine-grained setting, i.e., with 50 categories. The experimental evaluation shows significant improvements in the few-shot setting, i.e., when using only 1%, 2% and 5% of labeled examples. By applying such adversarial schema to a simple Multi-Layer Perceptron, a classifier trained over a subset derived from 1% of the original training material achieves 92% of accuracy. Moreover, when considering a complex classification schema, e.g., involving 50 classes, the proposed method outperforms state-of-the-art alternatives such as BERT [12].

In the remaining, section 2 provides a discussion on the related works. Section 3 discusses the methods for representing sentences in vector space. In section 4, the proposed SS-GAN is presented. In section 5, the experimental evaluations are reported. Finally, in section 6 the conclusions are derived.

2 Related work

This work is in the area of Deep Learning for NLP. In particular, it relates to Generative Adversarial Networks and Transfer Learning.

Generative Adversarial Networks [18, 30] have been introduced in the field of Computer Vision. They are a class of generative models based on neural networks. A generator network is trained to produce samples resembling a given data distribution. A discriminator “adversarially” fights the generator. In NLP, the usage of GANs has been very limited as of the discrete nature of the NL outputs. Propagating the gradients during training between discriminator and generators is impossible. Different works tried to build natural language generators, as [13 , 38]. In [38], a policy-gradient method is adopted to train a generator within a GAN framework for natural language. In [32], the authors propose a method to force the discriminator to observe a sequence of probabilities over every token in the vocabulary from the generator and a sequence of 1-hot vectors from the true data distribution. In [23], the discrete outputs are approximated by the usage of the Gumbel-softmax distribution [19], which is a continuous approximation to a multinomial distribution parameterized in terms of the softmax function. In [13], an autoencoder is trained to learn low-dimensional representations of sentences. A GAN is then trained to generate vectors in this space, which decode to realistic utterances.

Transfer Learning is a way to exploit already acquired knowledge in pre-existing models. In NLP, this technique is predominant in many works since the appearance of models like Universal Sentence Encoder (USE) [4], ELMO [28], and BERT [12]. In particular, BERT gave rise to a whole new series of work in using the Transformer [34] model for Transfer Learning.

In this work, we will adopt the USE model to provide initial representations for the SS-GAN framework. We aim at improving the applicability of the above embedding methods in scenarios where very few labeled data are provided, while large collections of unlabeled texts exist.

3 Embedding sentences in geometrical spaces

Lexical embeddings have been largely adopted to encode lexical and textual information in metric spaces, where the corresponding vector operations reflect syntactic as well as semantic properties. In this work, we explore two kinds of embedding techniques. In section 3.1, we will describe a vector space obtained by the approximation of a Semantic Tree Kernel function, aiming at capturing meaningful syntactic and lexical information. In section 3.2, a different representation for sentences, obtained by a deep learning model, i.e., the Universal Sentence Encoder is studied.

3.1 Nyström -based approximation of tree kernels

Syntax-based vector representations aim at capturing information reflecting syntactic aspects of sentences. For example, it could be possible to enumerate each possible syntactic relations and associate each of them to a specific vector dimension. Given a sentence, it could be possible to compute its syntax-based vector and use it to perform algebraic operations with other vectors, e.g., the cosine similarity for measuring how two sentences are syntactically similar. However, such a naive approach will not capture all the facets behind the syntax.

One powerful way of dealing with syntax in Machine Learning is given by the Tree Kernels [6]. A Tree Kernel captures the similarity between two parse trees by looking at the common structures in the two parse trees. Tree Kernels have been demonstrated successful in many different tasks, e.g., as in [2 , 10]. More formally, given an input training dataset $L$ , a kernel K (o_i, o_j) is a similarity function over $L^{2}$ that corresponds to a dot product in the implicit kernel space, i.e., K (o_i, o_j) = Φ (o_i) · Φ (o_j). The advantage of kernels is that the projection function $Φ (o) = \vec{x} \in ℝ^{n}$ is never explicitly computed [31]. In general, this operation is prohibitive for Tree Kernels as the number of syntactic substructures corresponds to the dimension n of the underlying kernel space: it grows exponentially for Semantic Tree Kernels that generalize syntactic structures and lexical relations across large sentence collections and very large lexicons. Fortunately, kernel functions are used by learning algorithms, such as Support Vector Machines [33], to operate only implicitly on instances in the kernel space: as a consequence, the complexity is never bound to the space dimension n, as the explicit vector definition is never used.

In [9] it has been shown that neural networks can be trained in low-dimensional spaces which approximate a generic Reproducing Kernel Hilbert Space. These low-dimensional approximations are derived as a reconstruction from a set of real reference training (unlabeled) examples, called landmarks, which can be used to compile the representation of any unseen test instance.

Let us assume we apply the projection function Φ over all examples from $L$ to derive representations $\vec{x}$ being the rows of the matrix X. The Gram matrix of the kernel can be computed as M = XX^⊤, with every single element corresponding to $M_{ij} = Φ (o_{i}) \cdot Φ (o_{j}) = K (o_{i}, o_{j})$

The Nyström method [36] aims to derive a new low-dimensional embedding $\tilde{x}$ in a l-dimensional space, with l ⪡ n so that $\tilde{M} = \tilde{X} {\tilde{X}}^{⊤}$ and $\tilde{M} \approx M$ . In other words, $\tilde{M}$ is an approximation of M that is obtained by using a subset of l columns of the matrix of the available examples, called landmarks. Suppose we randomly sample l columns of M, and let $C \in ℝ^{| L | \times l}$ be the matrix of these sampled columns. Then, we can rearrange the columns and rows of M and define X = [X₁ X₂] such that: $\begin{matrix} M = X X^{⊤} & = [\begin{matrix} W & X_{1}^{⊤} X_{2} \\ X_{2}^{⊤} X_{1} & X_{2}^{⊤} X_{2} \end{matrix}] \\ and C & = [\begin{matrix} W \\ X_{2}^{⊤} X_{1} \end{matrix}] \end{matrix}$ (1) where $W = X_{1}^{⊤} X_{1}$ , i.e., the subset of M that only considers landmarks. The Nyström approximation can be defined as: $M \approx \tilde{M} = C W^{†} C^{⊤}$ (2) where W^† denotes the Moore-Penrose inverse of W. Notice that $\tilde{M}$ is the best approximation of the Gram matrix M according to the Frobenius norm (as discussed in [14]) and it can be expressed as a multiplication of three l-dimensional matrices. The Singular Value Decomposition (SVD) is used to obtain W^† as follows. First, W is decomposed so that W = USV^⊤, where U and V are both orthogonal matrices, and S is a diagonal matrix containing the (non-zero) singular values of W on its diagonal. Since W is symmetric and positive definite W = USU^⊤. Then $W^{†} = {US}^{- 1} U^{⊤} = {US}^{- \frac{1}{2}} S^{- \frac{1}{2}} U^{⊤}$ and the Equation 2 can be rewritten as

$\begin{matrix} M \approx \tilde{M} & = C {US}^{- \frac{1}{2}} S^{- \frac{1}{2}} U^{⊤} C^{⊤} \\ = (C {US}^{- \frac{1}{2}}) (C {US}^{- \frac{1}{2}})^{⊤} = \tilde{X} {\tilde{X}}^{⊤} \end{matrix}$

Given an input example $o \in D$ , a new low-dimensional representation $\tilde{\vec{x}}$ can be thus determined by considering the corresponding item of C as

$\tilde{\vec{x}} = Θ (o) = \vec{c} {US}^{- \frac{1}{2}}$ (3)

where $\vec{c}$ is the vector whose dimensions contain the evaluations of the kernel function between o and each landmark o_j ∈ L. Therefore, the method assigns to individual examples an l-dimensional vector, based on its Nyström reconstruction. Regardless of the input examples (vectors, trees or graphs), given a valid kernel function, the Nyström method always projects examples in a l-dimensional feature space whose metrics depend on the kernel. The inner product among vectors is, in fact, the best approximation of the kernel function (according to the Frobenius norm) of the corresponding examples.

In this work, we follow the idea of [9] to generate vector representations of sentences to be fed into an MLP architecture acting as the discriminator in an SS-GAN setting. In particular, we will approximate the space defined by the Compositionally Smoothed Partial Tree Kernel discussed in [1] as it suitably captures both lexical syntactic and compositional semantic information.

3.2 Universal sentence encoder

The Universal Sentence Encoder (USE) [4] is a deep learning model providing 512-dimensional vector representations of sentences or short paragraphs. It belongs to the family of Transfer Learning models, which try to provide good out-of-the-box representations for NLP tasks, ranging from text classification to text similarity. The model we refer to in this work is the one trained with the Deep Averaging Network 1 , as presented in [4]. The model takes in input embeddings for words and bi-grams; it then averages them and then passes them through a feed-forward deep neural network to produce sentence embeddings. The model is trained over a combination of unsupervised and supervised learning.

In [4], it has been demonstrated that using the representations computed by the USE with simple MLP can provide very strong performances on multiple NLP tasks. In this work, we aim at verifying the capability of these representations to provide useful semantic information as opposed to the representations described in section 3.1 in a few shot scenario. Similar to the Nyström -based representation, we will adopt the USE vectors in an MLP architecture in the SS-GAN setting. We will show that the two types of vector representations provide complementary information.

4 Adversarial training for few-shot text classification

4.1 Semi-supervised GANs

In order to be effective, deep neural networks are usually trained on large labeled datasets. The availability of labeled data is not always guaranteed, as the process of annotating the data for a given task/language/domain is a costly and time-consuming process: on the contrary, unlabeled data can be easily accessed. Deep architectures were initially adapted to the semi-supervised case by using concepts coming from the theory of graph-based methods [22 , 37]. In this setting, both labeled and unlabeled data are represented in the same graph, and their relationships are exploited to “transfer” knowledge from the unlabeled instances to the labeled ones.

Semi-Supervised Generative Adversarial Networks (SS-GANs) [30] support a semi-supervised setting within the GAN framework. GANs are traditionally used for their generative capabilities. In GANs, the generator aims to produce new examples resembling an existing distribution; the discriminator (i.e., a binary classifier) aims to distinguish real examples from the ones generated by the generator. As shown in Figure 1, in SS-GAN the discriminator $D$ is trained over a (k + 1)-class objective: true examples are classified in one of the target k classes, while the generated samples are classified into the last k + 1 class. This (k + 1)-class discriminator objective leads to strong empirical results in image classification since [30]. It has been also shown that the adoption of a small labeled dataset improves the effectiveness of the generative component, improving the quality of the generated images.

Fig.1

SS-GAN architecture. $G$ generates from noise a set of fake examples F. These along with unlabeled U and labeled L examples are used as input for the discriminator $D$ .

More formally, let $D$ and $G$ denote the discriminator and generator, and p_d and $p_{G}$ denote the distribution of real data and the generated fake examples, respectively. To train a k-class classifier with a small number of labeled samples, the objective of $D$ is extended as follows.

Let us define $p_{m} (\hat{y} = y | x, y = k + 1)$ the probability provided by the model m that a generic example x is classified to the fake class, while the probability $p_{m} (\hat{y} = y | x, y \in (1, . . ., k))$ corresponds to x as considered real and belonging to one of the target classes.

$D$ is expected to perform well both in the supervised classification task (with respect to the k-classes) and on the unsupervised classification task. It means that $D$ should not erroneously reject real examples (labeled and unlabeled) and should not accept fake examples, i.e., the ones generated by $G$ ). The loss function of $D$ is defined as: $L_{D} = L_{D_{\sup .}} + L_{D_{unsup .}}$ where: $\begin{array}{l} L_{D_{sup .}} & = - E_{x, y ~ p_{d}} \log [p_{m} (\hat{y} = y | x, y \in (1, ..., k))] \\ L_{D_{unsup .}} & = - E_{x ~ p_{d}} \log [1 - p_{m} (\hat{y} = y | x, y = k + 1)] \\ - E_{x ~ G} \log [p_{m} (\hat{y} = y | x, y = k + 1)] \end{array}$

$L_{D_{\sup .}}$ measures the error in assigning the wrong class to a real example. $L_{D_{unsup .}}$ measures the error in incorrectly recognizing a real (unlabeled) example as fake and not recognizing a fake example. It is straightforward to observe that labeled examples are used to minimize $L_{D_{\sup .}}$ , while the unlabeled examples are used to minimize $L_{D_{unsup .}}$ thus improving its generalization capability.

At the same time, $G$ is expected to generate examples that are similar to the real ones sampled from the real distribution p_data. The more $G$ will succeed in this task, the better it will fool $D$ . Moreover, as suggested in [30] a good $G$ should generate data approximating the statistics of real data as much as possible. In other words, the average example generated in a mini-batch by $G$ should be as much similar as possible to the real prototypical one. Formally, let f (x) denote the activation on an intermediate layer of $D$ , this difference between such averaged inner representations should be minimized, introducing the feature matching loss of $G$ defined as: $L_{G_{feature matching} = {∥ E_{x \sim p_{d}} f (x) - E_{x \sim G} f (x) ∥}_{2}^{2}}$

This is combined with the unsupervised loss of $G$ induced by the fake examples correctly rejected: $L_{G_{unsup .}} = - E_{x ~ G} \log [1 - p_{m} (\hat{y} = y | x, y = k + 1)]$

Finally, the $G$ loss is $L_{G} = L_{G_{feature matching}} + L_{G_{unsup .}}$ .

While in literature an SS-GAN is usually stimulated with input examples x encoding images (e.g. from the MNIST dataset), in the next section we will show that they can be easily adopted over input spaces encoding linguistic information.

4.2 SS-GAN for text classification

In [8] we showed how to incorporate the SS-GAN perspective for the training of the KDA architecture, defined in [9]. The KDA, made of Nyström specific layers and an MLP, was used as a discriminator $D$ in the SS-GAN. The generator $G$ was an MLP that took in input random noise drawn from the Normal Distribution, and outputted vectors that, at the end of the training, have to be as much similar as possible to the Nyström ones.

In this work, we extend this schema to any vectors representing sentences. In particular, we substitute the KDA architecture with a more generic MLP as the discriminator $D$ . In particular, $D$ takes in input a vector x representing a sentence, and it is then made of a series of relu-activated hidden layers. Finally, an output softmax-activated layer is used for the final classification. The generator $G$ is, again, a series of relu-activated hidden layers. It takes in input random noise drawn from a normal distribution and it outputs a vector, whose dimensionality matches the one of $D$ input.

Applying a GAN perspective on such an architecture promotes two main benefits: the semi-supervised nature of the GAN approach and the expressiveness of linguistically-motivated vector representations for the involved sentences. This paves the way to cost-effective and highly reliable complex NLP inference, even in poor training conditions, e.g., small labeled data sets.

During the training phase, the input dataset is expected to contain labeled and unlabeled examples: these are randomly extracted and grouped in mini-batches of size b to be provided to $D$ . Consistently with [30], at each iteration $G$ will also provide b examples, which are added to the mini-batch provided to $D$ . After the classification of each mini-batch, the assignments provided by the discriminator with respect to the k + 1 target classes are used to evaluate the losses $L_{D}$ and $L_{G}$ .

We expect that “fake" and unlabeled examples will boost the capability of $D$ in improving its inner representation of input examples. If only a few (let’s say less than one hundred) examples are provided to an MLP, the ability to derive a robust parameter estimation through training is very limited. Moreover, there is no guarantee that a Transfer Learning approach works in any downstream task: its success indeed depends on the distance between the collection used in the initial training and the collections of examples used for the fine-tuning. We expect that an SS-GAN will boost its performance since its adaptation to the new dataset is empowered by using a possibly huge collection of real sentences when only few annotations are provided.

The application of the resulting architecture after training is straightforward [30]. $G$ is discarded after training and only $D$ is adopted at inference time. In other words, there is no additional cost at inference time with respect to the application of $D$ , which here is a simple MLP. From a computational perspective, it results in a highly scalable and efficient solution.

5 Experimental evaluation

In this section, we provide a set of evaluations of the proposed architecture based on the SS-GAN approach with respect to text classification tasks involving short sentences, in particular questions.

5.1 Experimental setup

In this work, we aim to assess the impact of the proposed semi-supervised approach in boosting the performances of a neural classifier in poor training conditions, i.e., when a minimal set of labeled data is available. To evaluate the benefit of adopting the SS-GAN, we consider the setting where only a subset L of a dataset is labeled, while the rest U is unlabeled. This second subset will be used to estimate the unsupervised losses of $D$ and $G$ , as discussed in Section 4. We will systematically train the SS-GAN and compare it within a classification task characterized by different levels of complexity, e.g., considering different numbers of target classes. Moreover, the beneficial impact of the proposed approach will be evaluated at different sizes of L. In practice, given an annotated dataset, its subsets corresponding to 1%, 2%, 5%, 10%, 20%, 30%, 40%, 50% and 100% are used as growing training material. The remaining part of the data in each setting is used as the reference set of unlabeled examples. Consistently with [8], we address the Question Classification task. We adopt the UIUC dataset [24]: this includes a training and test set of 5, 452 and 500 questions. Each question is annotated with respect to 50 fine-grained categories reflecting its focus. A coarse-grained set of 6 categories, i.e LOC as locations, ENTY as entities, HUM as humans, NUM as numbers, DESC as descriptions and ABBR as abbreviations, is used to cluster the 50 fine-grain classes.

Fine-grained labels, such as ENTY:animal, are used to indicate question over specific subclasses, i.e. animals as kinds of entities. LOC:country indicates that a question is specifically about a country. For example, “What was the first domesticated bird ?” is a question annotated as ENTY:animal, while “What country did King Wenceslas rule ?” is a question annotated as LOC:country.

In our experiments, we measure the performances of the SS-GAN approach for text classification against both the coarse-grained and the fine-grained target categories. The former task allows to assess the performances of the model on a quite common setting (6 classes), widely discussed in literature. The latter allows to assess the performances of the model in a more complex setting, i.e., made of 50 categories for 5, 452 training example. This task will be even more challenging when reducing the number of labeled instances to one or two examples per class.

We thus compare the contribution of the proposed technique when different embedding and learning methods are used, as discussed hereafter.

Embedding Techniques. In order to assess the applicability of the SS-GAN approach with different types of embeddings, we encode both the training and test questions according to the different methods described in the previous sections.

Kernel (K): In order to provide the input to the KGAN architecture, we generated the Nyström representations of the examples by using the Compositionally Smoothed Partial Tree Kernel (CSPTK) [1]. It combines both syntactic and compositional semantic information derived from the dependency graphs of questions. Notice that to compute the Tree Kernel, the dependency graphs have been transformed in trees, according to [1]. In particular, explicit compositional information is used to augment a Lexically Centered Tree [10]. Lexical vectors needed in the computation of the CSPTK are generated using a Skip-gram model 2 [26].

Universal Sentence Encoding (USE): we encoded questions using the Universal Sentence Encoder model that is available on Tensorflow HUB. 3

Joint model (KUSE): we also will experiment with the concatenation of the above two representations to exploit the information encoded in the both embeddings.

Learning algorithms and Architectures. The proposed SS-GAN architecture is essentially an extension of an MLP. In order to verify the beneficial impact of the proposed semi-supervised schema, we compare it with two competitive methods.

LinSVM: the first method is a linear Support Vector Machine (SVM), directly applied over the embeddings from the subsets of labeled examples. The linear SVM implemented within the KeLP framework 4 is considered. This simple algorithm is parametrized only regarding the regularizer C, tuned on a separated 10% held-out, in the [0.1, 10] range, with increments of 0.1 at each step.

MLP: the second competitive method is a Multi-Layer Perceptron. It is essentially the proposed SS-GAN architecture considering only the Discriminator $D$ trained over the subset of labeled examples. It is composed of a sequence of h hidden layers each separated by a non-linear relu function, where each layer has the same size of the input space. On top of the last hidden layer, a softmax fully connected layer is used to classify the example among the k target classes. At the output of each hidden layer, a dropout layer is added to prevent overfitting. A cross-entropy loss function is finally applied and the entire architecture is optimized using the ADAM optimizer, with the learning rate set to 10^-4. This architecture is tuned with respect to the following hyper-parameters: i) the number of hidden layers h in the discriminator among 1, 2, and 3; ii) dropout level applied after each hidden layer among 0.5, 0.75, 0.95 and 0.99; iii) whether using or not feature matching in calculating $L_{G}$ ; iv) batch size of 8 and 16: larger batch sizes are not considered since they tend to provide unstable training, especially for $G$ .

GAN: it implements the proposed SS-GAN, where the Discriminator $D$ is characterized exactly by the same architecture of the previous MLP architecture, extended to a classification task involving k + 1 classes. The Generator $G$ is again an MLP, where random vectors generated according to a normal distribution $N (0, 1)$ are used as input. A sequence of h hidden layers is used, each activated by a relu function. Differently to the discriminator $D$ , no dropout is used for $G$ , as suggested in [30]. The generator produces embeddings characterized by the same dimensionality of the original input space to evaluate the losses $L_{G}$ and $L_{D_{unsup .}}$ . The same hyper-parameters of the previous MLP architecture are considered. In order to reduce the number of possible configurations, the same number of hidden layers h is always used both for the $D$ and $G$ . In order to get more stable training, the labeled examples within each batch are replicated with a factor f = 1/l, where l is the percentage of labeled examples considered in training.

It must be said that the time required to train a GAN is higher with respect to the training of the corresponding MLP. This depends on the size of the unlabeled material that augments the labeled data and the replication factor f. Moreover, as pointed out in [25], the convergence of a GAN requires a higher number of epochs (in this work about 100 epochs for each training are required). However, as discussed in the previous section, once the training phase is complete, no additional computational costs are introduced, since only the Discriminator (i.e., an MLP) is used at inference time.

We used a split of the training set (about 10% of the entire training dataset, the same for all the experiments) to tune the hyper-parameters of each model. We performed 5 runs shuffling the training set and reporting the average classification accuracy. The MLP and SS-GAN are implemented in Tensorflow 1.14 5 and trained on an NVIDIA Tesla T4 GPU. Accuracy, i.e., the percentage of the test questions assigned to their correct classes, is used in the evaluations.

State-of-the-Art. The following methods represent the state-of-the-art with respect to the QC task (at least with respect to the coarse-grained settings). However, their accuracy is very competitive when trained on the entire labeled material. We will train and evaluate them with the same subsets of labeled material L to verify the actual benefit of our proposed method.

BERT: we will measure the BERT architecture proposed in [12], in particular the English BERT Cased model, which is made of 12-layer and 110M parameters. We will fine-tune this model over the QC task at different sizes L with the standard provided parameters 6 . We neglected larger versions of BERT: first exploratory evaluations provided lower accuracy scores when using small training sets and it was not competitive with respect to the BERT-Base model.

Kernel-based SVM (KSVM): the standard Kernel-based approach proposed in [1] is used, with the same parameters 7 . This comparison is particularly informative since part of the embeddings are derived by approximating the implicit representation space generated by the Kernel function used in this method.

5.2 Experimental results

Coarse-Grained Setting. Results are analytically reported in Table 1 and the most significant learning curves are reported in Figure 2. Each set of rows in the table refers to an embedding technique, while the last row refers to the state-of-the-art baseline system, i.e., BERT [12]. For each model, we used an increasing number of labeled material composed by 1%, 2%, 5%, 10%, 20%, 30%, 40%, 50% and 100% of the original dataset.

Table 1
Results in terms of accuracy with respect to the coarse-grained setting within the Question Classification task. Rows report the different input embeddings and different learning algorithms. Columns express the percentage of training material used. In bold the highest accuracy score with respect to each column

Model Percentage of labeled material

1% 2% 5% 10% 20% 30% 40% 50% 100%

Kernel K-SVM [1] 58.0% 65.8% 77.0% 86.6% 92.2% 93.2% 92.8% 93.2% 95.0%

K-LinSVM [7] 57.5% 66.6% 77.2% 84.5% 88.8% 90.1% 91.0% 91.3% 92.6%

K-MLP [9] 57.8% 65.7% 78.2% 85.3% 89.2% 90.7% 92.1% 92.3% 93.4%

K-GAN [8] 64.9% 69.1% 78.7% 85.1% 88.9% 91.5% 91.3% 91.8% 93.2%

USE USE-LinSVM 79.6% 84.6% 87.0% 89.2% 91.2% 93.4% 92.8% 93.6% 93.6%

USE-MLP 77.8% 85.7% 88.6% 91.1% 91.2% 92.8% 93.0% 93.4% 93.8%

USE-GAN 87.5% 89.5% 91.4% 91.8% 93.6% 93.5% 94.0% 94.2% 93.7%

Kernel KUSE-LinSVM 81.4% 84.6% 87.6% 91.0% 93.4% 93.6% 93.6% 94.8% 93.6%

+ KUSE-MLP 80.5% 86.9% 89.5% 91.6% 92.8% 93.9% 94.6% 94.5% 94.2%

USE KUSE-GAN 92.0% 92.0% 91.5% 92.5% 93.7% 93.6% 95.0% 95.4% 94.9%

BERT [12] 20.1% 21.4% 50.0% 81.6% 93.4% 95.1% 96.0% 96.5% 96.9%

Fig.2

Learning curves for the QC Coarse-Grained task. (a) QC Coarse-Grained - Kernel; (b) QC Coarse-Grained - USE; (c) QC Coarse-Grained - USE + Kernel; (d) QC Coarse-Grained - Best systems

In the coarse-grained setting, we consider 6 classes. This specific classification task is characterized by a random baseline (obtained by a system that randomly assigns questions to classes) of 16%. Since the dataset is not balanced, a system that always assigns the most frequent class (i.e., ENTITY) would achieve a 22% of accuracy.

The first group of results is obtained by applying the different learning architectures over the Kernel-based embeddings. A Kernel-based SVM (K-SVM) achieves 95.0% when trained over the entire L. It demonstrates the expressiveness of the (implicit) representation space provided by the kernel function. However, when the size of the dataset decreases, performances drop as well. When using 10% of labeled material (i.e., about 550 questions), an accuracy of 86% is obtained. When reducing this amount to only 50 questions, the accuracy drops to 58%. It is still higher with respect to the baselines, but the drop is nevertheless significant.

When the Kernel space is approximated by applying the Nyström method, both a linear SVM (the K-LinSVM, proposed in [7]) and an MLP (the K-MLP, which is essentially a KDA reported in [9]) shows similar trends. Even though the explicit 500-dimensional space provides an expressive reconstruction of the original kernel space, both methods suffer a similar performance drop. The first application of the semi-supervised approach is reported in the K-GAN row. When very few training materials is adopted, it can provide higher performances with respect to the previous supervised approaches: the K-GAN obtains 64.9% in accuracy while the Kernel SVM and the KDA obtain 58, 0% and 57.8% respectively; it results in an error reduction of about 17%. As reported in Figure 2a, the differences between the models get smaller when the size of L grows: starting about at 30 - 40% of the labeled examples the K-GAN, K-MLP and K-SVM perform almost equally. At 100% of the examples, all the model performances are very similar: the K-GAN, K-MLP, the K-SVM obtain 93.2%, 93.4% and 95.0%, respectively. This confirms our idea that linguistically motivated information, combined with adversarial learning, can help in reducing the amount of annotated data needed to train text classification models. However, we are still quite far from providing a solution that can be used in a realistic scenario, when using only 50 questions to train a neural architecture.

Results obtained when training the above algorithms over the embeddings obtained through the Universal Sentence Encoder (USE) show similar trends, but also some significant differences. A linear SVM and an MLP trained over the USE representation (i.e. the USE-LinSVM and USE-MLP) do not outperform the Kernel-based SVM when trained on the entire L, achieving 93.6% and 93.8%. However, these classifiers achieve higher results when only the 1% of the training material is used, i.e. 79.6% and 77.8%, respectively. Most importantly, when applying the SS-GAN approach, the robustness of the discriminator increases, as shown by the USE-GAN curve in Figure 2b: with only the 1% of the training material, accuracy boosts to 87.5%. Such improvements are more noticeable until all methods are trained over the 20% of L.

The most straightforward results are obtained when combining the above representation, in the Kernel + USE setting. The embedding spaces complement each other, improving the quality of the overall representation. As reported in rows KUSE-LinSVM and KUSE-MLP, this expanded representation space achieves better results when compared with any methods trained over individual embeddings. Most importantly, as shown in Figure 2c, when the SS-GAN is adopted a 92.0% of accuracy is achieved with only the 1% of L. These results outperform all previous methods with respect to any percentage of labeled material.

The proposed solution is highly competitive in poor-training conditions with respect to state-of-the-art methods such as BERT. Although this achieves an impressive accuracy of 96.9% when the entire L is used, its capability of generalizing in the QC downstream task is compromised with reduced training datasets. As shown in Figure 2d, BERT trained with only 50 questions diverges, obtaining an accuracy comparable with the lowest baselines. More than 1, 600 labeled questions (i.e., 30% of L) are required for a competitive training. Overall, the proposed approach based on the SS-GAN allows acquiring a classification model whose quality, with respect to the addressed QC task, is stable between 92% and 94% even with a bunch of annotated examples, even only 50 labeled examples from a dataset of 5, 500 instances.

Fine-grained Setting. The fine-grained setting considers 50 classes and it represents a far more complex scenario, especially when very few labeled material is available. When only 1% of L is available, we expect a learning machine to generalize the notion of questions having about 1 example per class. The complexity of this classification task is evident when considering the baselines: a random baseline corresponds to 2%, while a system always assigning the most frequent class, i.e., Human, is about 17%. Results are analytically reported in Table 2 and the most significant learning curves are reported in Figure 3. In general, the trends observed in the previous experiments are confirmed, although with lower performances. As shown in Figure 3a, a K-SVM trained over the entire dataset achieves 86.6%. As for the previous coarse-grained setting, the reduction of L impacts the results, up to 42.6% accuracy when 1% of L is considered. The 500 dimensional Nyström embedding allows approximating the kernel space even though less effectively with respect to the previous coarse-grained setting: a higher number of dimensions, in terms of additional landmarks, seems required to reduce the gap between K-LinSVM/K-MLP and K-SVM. Anyway, the SS-GAN is again beneficial, improving the quality of the results, even though in correspondence of very small labeled datasets: at 1% and 2% of L, K-GAN obtains 45.8% and 54.5% of accuracy, respectively. The beneficial effect of the USE-based embeddings is confirmed again in this setting. When training a Linear SVM or an MLP over such embeddings, an error reduction is always obtained with respect to the kernel-based architectures, although differences are less evident. It is interesting to notice that the USE-LinSVM achieves slightly higher results with respect to the USE-MLP with very few labeled data: again, the non-linearity of an MLP requires more data to provide an effective classification, but this phenomenon is mitigated when the size of the training material increases. As shown in Figure 3b, the SS-GAN variant again improves the results, so that a 54.1% of accuracy at 1% of L is obtained, higher if compared to the 49.0% of USE-LinSVM and 47.6% of USE-MLP. Finally, as shown in Figure 3c, the combination of both representations again guarantees a further boost of performances with respect to the adoption of individual representations. Most importantly, the adoption of the semi-supervised schema guarantees a further boost: at the 1% of L an accuracy of 60.0% is obtained.

Table 2

Results in terms of accuracy with respect to the fine-grained setting within the Question Classification task. Rows report the different input embeddings and different learning algorithms. Columns expresses the percentage of training material used. In bold the highest accuracy score with respect to each column

Model		Percentage of labeled material
		1%	2%	5%	10%	20%	30%	40%	50%	100%
Kernel	K-SVM [1]	42.6%	49.2%	62.7%	69.8%	76.5%	80.8%	82.9%	83.8%	86.6%
	K-LinSVM [7]	44.6%	50.6%	59.8%	67.2%	74.8%	77.6%	78.4%	80.4%	80.4%
	K-MLP [9]	42.6%	49.6%	59.4%	66.1%	73.4%	76.0%	77.9%	79.4%	83.2%
	K-GAN [8]	45.8%	54.5%	62.8%	68.2%	74.9%	76.8%	78.9%	80.6%	82.3%
USE	USE-LinSVM	49.0%	59.4%	63.8%	66.8%	74.6%	79.4%	79.4%	81.4%	85.6%
	USE-MLP	47.6%	57.8%	62.4%	67.6%	73.5%	79.6%	80.9%	82.6%	85.1%
	USE-GAN	54.9%	63.3%	68.4%	72.4%	78.9%	82.2%	82.3%	85.0%	85.1%
Kernel	KUSE-LinSVM	53.8%	61.6%	66.6%	69.2%	79.2%	84.0%	83.8%	84.6%	88.2%
+	KUSE-MLP	54.2%	60.5%	64.9%	72.3%	79.8%	83.8%	85.6%	85.5%	87.9%
USE	KUSE-GAN	60.0%	64.8%	69.9%	76.5%	83.0%	85.2%	85.8%	87.3%	88.7%
BERT [12]		0.8%	3.5%	17.0%	30.5%	54.7%	64.9%	69.8%	73.3%	83.9%

Fig.3

Learning curves for the QC Fine-Grained task. (a) QC Fine-Grained - Kernel; (b) QC Fine-Grained - USE; (c) QC Fine-Grained - USE + Kernel; (d) QC Fine-Grained - Best systems

This result is straightforward for two reasons. First, the adversarial training allows improving the robustness of the discriminator $D$ , obtaining the highest results with respect to any percentage of training material. Second, and most importantly, the proposed KUSE-GAN outperforms also BERT both in poor training conditions and when the entire dataset is used, as shown in 3d. It means that when few training materials is used or classification tasks are complex (e.g. has many classes) the proposed adversarial training is always crucial to achieve acceptable performances.

6 Conclusions

In this paper, the application of Generative Adversarial Networks (GAN) for semi-supervised learning within natural language inference tasks is presented. The approach is specifically tailored to amplify the applicability of combined Kernel-based and Deep Learning architectures. The strict requirement of large-scale annotated corpora, strongly characterizing most complex NLP inference tasks in Deep Learning, is here mitigated by the semi-supervised perspective offered by the Semi-Supervised GANs. In the proposed approach, the specific SS-GAN formulation has been adopted in combination with sentence embeddings derived by approximating a rich linguistic kernel space and by adopting the Universal Sentence Encoder. This allows to bootstrap the training of a classifier in all the situations when a significant amount of labeled examples is not available, but a large unlabeled dataset can be obtained. The experimental evaluations of the SS-GAN in sentence-level semantic classification tasks confirm several benefits. When a small dataset is made available, up to 1% of the original training material, an impressive 92% of accuracy is achieved in the coarse-grained Question Classification task. Moreover, when considering complex classification schema, e.g. involving 50 classes, the proposed method outperforms state-of-the-art methods such as BERT.

In the future, we aim at targeting more tasks and explore the role of other kernel embeddings. Finally, this paper does not fully explore the possibility to adopt SS-GAN settings against transformer-based embeddings (e.g. [11]) and its variants. This is certainly a core issue to be explored in future work.

Footnotes

We adopted the model that is available on Tensorflow HUB.

For the remaining kernel parameters, the same setting of [] is used.

http://www.kelp-ml.org, presented in [15, ].

We adopted the implementation which is available at referred to as Question Classification.

Acknowledgments

We would like to thank Carlo Gaibisso, Bruno Luigi Martino and Francis Farrelly of the Istituto di Analisi dei Sistemi ed Informatica “Antonio Ruberti” (IASI) for supporting the experimentations through access to dedicated computing resources made available by the Artificial Intelligence & High-Performance Computing laboratory.

References

Annesi

, Croce

, Basili.

. Semantic compositionality in tree kernels. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014, pp. 1029–1038. ACM, 2014.

Castellucci

, Filice

, Croce

, Basili

. UNITOR: Combining syntactic and semantic kernels for twitter sentiment analysis. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Vol. 2: Proc. of the 7th International Workshop on Semantic Evaluation, (2013), pp. 369–374, Atlanta, Georgia, USA.

Castellucci

, Filice

, Croce

, Basili.

. UNITOR: Aspect based sentiment analysis with structured learning. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), (2014), pp. 761–767, Dublin, Ireland.

Cer

, Yang

, Kong

S-y.

, Hua

, Limtiaco

, St. John

, Constant

, Guajardo-Cespedes

, Yuan

, Tar

, Strope

and Kurzweil

. Universal sentence encoder for English. In Proc. of EMNLP: System Demonstrations, (2018), pp. 169–174, Brussels, Belgium.

Chapelle

, Schlkopf

, Zien.

. Semi-Supervised Learning. 1st edition, The MIT Press, 2010.

Collins

, Duffy.

. Convolution kernels for natural language. In Proceedings of Neural Information Processing Systems (NIPS’2001), (2001), pp. 625–632.

Croce

, Basili.

. Large-scale kernel-based language learning through the ensemble nystrom methods. In Advances in Information Retrieval - 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings, volume 9626, (2016), pp. 100–112. Springer.

Croce

, Castellucci

, Basili.

. Kernel-based generative adversarial networks for weakly supervised learning. In AI*IA 2019 – Advances in Artificial Intelligence, pp. 336–347, Cham, 2019. Springer International Publishing.

Croce

, Filice

, Castellucci

, Basili.

. Deep learning in semantic kernel spaces. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, (2017), pp. 345–354.

10.

Croce

, Moschitti

, Basili.

. Structured lexical similarity via convolution kernels on dependency trees. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, 27-31 July 2011, Edinburgh, UK, (2011), pp. 1034–1046.

11.

Dai

, Yang

, Cohen

W.W.

, Salakhutdinov.

. Good semi-supervised learning that requires a bad gan. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, (2017), pp. 6513–6523, Red Hook, NY, USA.

12.

Devlin

, Chang

M.-W.

, Lee

, Toutanova.

. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), (2019), pp. 4171–4186, Minneapolis, Minnesota.

13.

Donahue

, Rumshisky.

. Adversarial text generation without reinforcement learning. CoRR, abs/1810.06640, 2018.

14.

Drineas

, Mahoney.

M.W.

. On the nyström method for approximating a gram matrix for improved kernelbased learning, Journal of Machine Learning Research6 (2005), 2153–2175.

15.

Filice

, Castellucci

, Croce

, Basili.

. KeLP: a kernel-based learning platform for natural language processing. In Proceedings of ACL-IJCNLP 2015 System Demonstrations, (2015), pp. 19–24, Beijing, China.

16.

Filice

, Castellucci

, San Martino

G.D.

, Moschitti

, Croce

and Basili

. Kelp: a kernel-based learning platform, Journal of Machine Learning Research18(191) (2018), 1–5.

17.

Goldberg

. A primer on neural network models for natural language processing, Journal of Artificial Intelligence Research57(1) (2016), 345–420.

18.

Goodfellow

, Pouget-Abadie

, Mirza

, Xu

, Warde-Farley

, Ozair

, Courville

, Bengio

. Generative adversarial nets. In Advances in Neural Information Processing Systems27 (2014), pp. 2672–2680. Curran Associates, Inc.

19.

Jang

, Gu

, Poole.

. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview. net, 2017.

20.

Kim.

. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, pp. 1746–1751, 2014.

21.

Kim

, Jernite

, Sontag

, Rush.

A.M.

. Character-aware neural language models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., (2016), pp. 2741–2749, 2016.

22.

Kipf

T.N.

, Welling.

. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, 2017.

23.

Kusner

M.J.

, Hernández-Lobato.

J.M.

. Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv:1611.04051, 2016.

24.

, Roth.

. Learning question classifiers: the role of semantic information, Natural Language Engineering12(3) (2006), 229–249.

25.

Mescheder

, Geiger

, Nowozin.

. Which training methods for GANs do actually converge? In Dy

Jennifer

and Krause

Andreas

, editors, Proceedings of the 35th ICML , volume 80, pages 3481–3490, Stockholm Ad’ssan, Stockholm Sweden 2018. PMLR.

26.

Mikolov

, Chen

, Corrado

, Dean.

. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, 2013.

27.

Pan

S.J.

, Yang

, Fan

, Jialin Pan

(ph. D. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 2010.

28.

Peters

M.E.

, Neumann

, Iyyer

, Gardner

, Clark

, Lee

, Zettlemoyer.

. Deep contextualized word representations. In Walker

Marilyn A.

, Ji

Heng

, and Stent

Amanda

, editors, Proceedings of the 2018 NAACL Conference, pp. 2227–2237, 2018.

29.

Hovav

M.R.

, Levin.

. The Syntax-Semantics Interface, chapter 19, pp. 593–624. John Wiley & Sons, Ltd, 2015.

30.

Salimans

, Goodfellow

, Zaremba

, Cheung

, Radford

, Chen

, Chen.

. Improved techniques for training gans. In Lee

D. D.

, Sugiyama

, Luxburg

U. V.

, Guyon

, and Garnett

, editors, Advances in Neural Information Processing Systems 29, (2016), pp. 2234–2242. Curran Associates, Inc.

31.

Shawe-Taylor

, Cristianini.

. Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge, UK, 2004.

32.

Subramanian

, Rajeswar

, Dutil

, Pal

, Courville.

. Adversarial generation of natural language. In Proceedings of the 2nd Workshop on Representation Learning for NLP, (2017), pp. 241–251, Vancouver, Canada.

33.

Vapnik.

V.N.

. Statistical Learning Theory. Wiley-Interscience, 1998.

34.

Vaswani

, Shazeer

, Parmar

, Uszkoreit

, Jones

, Gomez

A.N.

, Kaiser

Łukasz

and Polosukhin

. Attention is all you need. In Advances in Neural Information Processing Systems30 (2017), pp. 5998–6008. Curran Associates, Inc.

35.

Weston

, Ratle

, Collobert.

. Deep learning via semi-supervised embedding. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, (2008), pp. 1168–1175, New York, NY, USA, ACM.

36.

Williams

C.K.I.

, Seeger.

. Using the nyström method to speed up kernel machines. In Leen

T. K.

, Dietterich

T. G.

, and Tresp

, editors, Advances in Neural Information Processing Systems 13, (2001), pp. 682–688. MIT Press.

37.

Yang

, Cohen

W.W.

, Salakhutdinov.

. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, (2016), pp. 40–48. JMLR.org.

38.

, Zhang

, Wang

, Yu.

. Seqgan: Sequence generative adversarial nets with policy gradient. In Singh

Satinder P.

and Markovitch

Shaul

, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, (2017), pp. 2852–2858. AAAI Press.