CPCL: Conceptual prototypical contrastive learning for Few-Shot text classification

Abstract

As prototype-based Few-Shot Learning methods, Prototypical Network generates prototypes for each class in a low-resource state and classify by a metric module. Therefore, the quality of prototypes matters but they are inaccurate from the few support instances, and the domain-specific information of training data are harmful to the generalizability of prototypes. We propose a Conceptual Prototype (CP), which contains both rich instance and concept features. The numerous query data can inspire the few support instances. An interactive network is designed to leverage the interrelation between support set and query-detached set to acquire a rich Instance Prototype which is typical on the whole data. Besides, class labels are introduced to prototype by prompt engineering, which makes it more conceptual. The label-only concept makes prototype immune to domain-specific information in training phase to improve its generalizability. Based on CP, Conceptual Prototypical Contrastive Learning (CPCL) is proposed where PCL brings instances closer to its corresponding prototype and pushes away from other prototypes. “2-way 5-shot” experiments show that CPCL achieves 92.41% accuracy on ARSC dataset, 2.30% higher than other prototype-based models. Meanwhile, the 0-shot performance of CPCL is comparable to Induction Network in the 5-shot way, indicating that our model is adequate for 0-shot tasks.

Keywords

Prototypical network text classification Few-Shot learning prompt learning

1 Introduction

Deep learning models rely heavily on the large amount of high-quality labeled training data which is time-consuming to collect and costly to annotate. Few-Shot Learning (FSL) seeks to achieve superior performance with limited amount of labeled data, bridging the sample-efficiency gap between deep learning models and their practical application in fields such as Computer Vision, Reinforcement Learning and Speech Recognition [1].

Prototype-based few-shot methods, like Prototypical Network [2] and Induction Network [3], generate prototypes for each class and perform classification based on the distances between queried examples and class prototypes. These methods have found wide applications in scenarios where acquiring instances is costly, such as Medical Image Analysis [4]. However, the learning of prototypes in these methods is dominated by the support set, which is limited and sparse in the feature space. The presence of noise and disturbance in the support set can lead to deviations in the prototypes [5, 6], making it challenging to learn robust prototypes solely from the few support instances.

The valuable interaction between query set and support set is ignored in previous works [7], which is informative for generating prototypes. As the query set –with its large and more diverse instances –can better stand for the distribution of the entire dataset, it is worthy to leverage the abundant query instances. By utilizing these query instances to guide the distribution of support set to make it align closely with that of query set, we can create more robust and representative prototypes that better capture the dataset’s characteristics. Specifically, we introduce a query-detached set, which is generated from the query set but without their labels. This detachment ensures that the labels of query instances remain confidential during training, thus preserving their authenticity. As a result, the obtained Instance Prototype becomes more typical at an instance level, thereby reducing the influence of noise and bias insupport set.

Another challenge in prototype-based methods is that classification accuracy deteriorates when transferring to new domains (categories) during the testing phase. This occurs because the categories present in training data are different from those encountered in the testing data, resulting in inevitable inter-domain dissimilarities. For example, in sentimental analysis task, a model trained on “cosmetics” and “mother & baby products” may extracts domain-specific features which affect its classification when classifying test sets such as “appliances”. Therefore, it is necessary to create prototypes that focus solely on sentiment concepts, disregarding domain-specificfeatures.

The visual-language model CLIP [8 –10] and zero-shot detection method ZSD [11 –13] have made significant advancements in completing few-shot and zero-shot tasks by incorporating label semantics as concepts and knowledges. Inspired by these approaches, we propose a novel method that leverages concepts as prototypes through prompt engineering. By extracting textual semantics of classes from labels (e.g., ‘Positive’ and ‘Negative’ for sentiment classes), we create Concept Prototype which is immune to domain-specific features from training data.

We synthesize the Instance Prototype and the Concept Prototype into a novel representation called the Conceptual Prototype (CP) which incorporates both individualized features specific to instances and general features shared among categories. Meanwhile, Prototypical Contrastive Learning (PCL) [30] is adopted to address the issue of unclear classification boundary. PCL utilizes a contrastive loss to measure the distance between instances and prototypes. By doing so, it encourages instances to have representations that are closer to their corresponding prototypes while pushing them away from prototypes of other classes. The contributions of this paper can be summarized as follows:

We introduce a novel Instance Prototype, which effectively models the interaction between the support and query sets by an interactive network. This helps to reconstruct the distribution of support set, resulting in a more robust and typical prototype that represents the entire dataset.

We propose the concept as a general prototype, which enhances the semantics of classes to mitigate the impact of domains-specific features when testing on different domains. This enables our model to handle few-shot and zero-shot tasks more robustly, as it can better capture the core concepts and semantics associated with each class.

We combine the Instance Prototype and Concept Prototype into Conceptual Prototype (CP), which integrates both individualized and general features. This comprehensive prototype is utilized in the classification process through Prototypical Contrastive Learning.

2 Related work

2.1 Few-shot learning

Most existing few-shot learning models follow the meta-learning method, employing an episodic training setup in which the meta-learner iterates through episodes in the meta-training phase. In each episode, a task is drawn from base classes C_base and a limited amount of support and query data from that task is made available. The meta-learner then learns a task-specific classifier based on the support data and this classifier make predictions on the query data. Updates to the meta-learner is computed based on the performance of the classifier on the query set. Evaluation of the meta-learner is also carried out in episodes in a similar fashion, except that the meta-learner is no longer updated and the performance on query data across multiple episodes is aggregated. This training method is known as “N-way K-shot” where each episode involves N classes and each class contains K support samples.

Metric learning is a kind of meta-learning method that learns the distance distribution of samples, ensuring that similar samples are positioned closely together in the metric space. The Siamese Neural Network [14] calculates the distance between two representations to measure their similarity. Prototypical Network creates a prototype representation c_k for each class [14]. This is achieved by the method of averaging, where the model calculates the mean of the support samples belonging to a specific class.

$c_{k} = \frac{1}{| S_{k} |} \sum_{(x_{i}, y_{i}) \in S_{k}} f_{Φ} (x_{i})$ (1)

Where f_Φ (x) represents the feature vector of samples. The distance from the query sample to each prototype is calculated to classify. $p_{Φ} (y = k ∣ x) = \frac{\exp (- d (f_{Φ} (x), c_{k}))}{\sum_{k^{'}} \exp (- d (f_{Φ} (x), c_{k^{'}}))}$ (2) $L (Φ) = - log p_{Φ} (y = k ∣ x)$ (3)

Based on the Prototypical Network, several advancements have been made. The Relation Network [15] replaces the traditional distance metric with neural networks. The Induction Network [3] improves the averaging method and introduces dynamic routing to generate prototypes. In the case of ProtoPShare [16], ProtoPool [17], and Deformable ProtoPNet [18], all similarity scores are processed through a fully connected layer to produce the final classification. ProtoTree [19], on the other hand, uses these similarity scores to compute a path across a soft decision tree, where each leaf corresponds to one specific class. In another approach, [20] utilizes word distribution to improve the performance.

2.2 Prompt learning

Language models (LMs) at scale, such as BERT [21] and GPT [22] have become cornerstones in the field of Natural Language Processing (NLP) due to their extensive knowledge. These models are already central to several products with millions of users, such as the coding assistant Copilot [23] and more recently ChatGPT [24]. In recent years, prompting LMs has become a dominant paradigm [22] which mines the knowledge of LMs to achieve superior performance in various downstream tasks such as question and answer, natural language understanding, and text classification in a low-resource state. [25] transforms the input into fill-in-the-blank phrases to aid the language model in understanding a given task. [26] introduces CoT, where the prompt comprises examples of a task, with inputs followed by intermediate reasoning steps leading to the final output.

The success of prompt learning has also been applied to the field of Computer Vision. ZSD [11, 12] leverages the semantic information carried by image labels as an intermediate representation to transfer knowledge learned from known classes to unknown classes, enabling zero-shot classification. CLIP [8] trains a vision model with supervised signals in natural language which achieves performance close to or even surpassing supervised methods in over 20 tasks in a zero-shot manner. [13] performs object-centric alignment of the language embeddings obtained from CLIP to improve precise localization of objects.

2.3 Prototypical contrastive learning

Contrastive learning aims to learn meaningful representations in a self-supervised setting. The goal of Instance-wise Contrastive Learning (ICL) is to bring the embedding of different views from the same instance closer to each other while pushing embeddings of views from different instances further apart by instance-level contrastive loss. This is commonly achieved by a large batch size that allows for the accumulation of positive and negative pairs within the same batch [27, 28], or a momentum encoder to update negative instances from a large and consistent dictionary in real-time [29]. $L_{InfoNCE} = \sum_{i = 1}^{n} - log \frac{exp (v_{i} \cdot v_{i}^{'} / τ)}{\sum_{j = 1}^{r} exp (v_{i} \cdot v_{j}^{'} / τ)}$ (4)

However, ICL has weakness: the representation is not encouraged to encode the semantic structure of data. In ICL, different instances are treated as negative pairs, regardless of whether they share similar semantics. As a consequence, negative instance pairs that actually have similar semantics are undesirably pushed apart in the feature space, preventing the model from learning high-level semantic information.

Prototypical Contrastive Learning (PCL) is more concerned with the learning of semantic information. In PCL [30] the traditional InfoNCE loss is replaced with ProtoNCE loss, which encourages representations to be closer to their assigned prototypes and farther from other prototypes.

$\begin{matrix} L_{ProtoNCE} & = L_{InfoNCE} \\ + \sum_{i = 1}^{n} - \frac{1}{M} \sum_{m = 1}^{M} log \frac{exp (v_{i} \cdot c_{s}^{m} / φ_{s}^{m})}{\sum_{j = 1}^{r} exp (v_{i} \cdot c_{j}^{m} / φ_{j}^{m})} \end{matrix}$ (5)

SCCL [31] better facilitates the classification of clustered clusters by optimizing clustering loss and contrast loss. SPCL [32] employs an offline prototype spawn approach with k-means clustering and regularizes samples to their corresponding prototypes explicitly. [33] proposes a label obfuscation strategy for dynamically refining the constraint relations between prototypes in the semantic space, improving the accuracy of the mapping between the feature space and the semantic space.

3 Methodology

Our Method is built upon the Prototypical Contrastive Learning framework, where we introduce a novel Conceptual Prototype (CP) to replace the traditional prototype in contrastive loss, as illustrated in Fig. 1. (the case of “2-way 2-shot”).

Fig. 1

CPCL model (in the case of 2-way 2-shot). $[s_{p}^{1}, s_{p}^{2}, s_{n}^{1}, s_{n}^{2}]$ is support set where p and n denote label “positive” and “negative”. [q₁, q₂, q₃, q₄] is query set and (p₁, p₂, …, p_|P|) is prompt templates. In the interactive network, query-detached set will be Q while support set be K and V of attention mechanism.

3.1 Instance prototype

Given a training set T consists of a support set S_t with a label set C_t, a query set Q_t, and the query-detached set Q_detached:

$\begin{matrix} C_{t} = {c_{1}, c_{2}, \dots, c_{| C |}} \\ S_{t} = {(s_{c}^{1}, c), \dots, (s_{c}^{| S |}, c) | for each c ϵ C} \\ Q_{t} = {(q_{c}^{1}, c), \dots, (q_{c}^{| Q |}, c) | for each c ϵ C} \\ Q_{detached} = {q_{1}, q_{2}, \dots {, q}_{| Q |}} \end{matrix}$ (6)

The vanilla Prototypical Network [2] generates prototypes by assigning equal weightage to each support instance, which may lead to inaccuracies. This is especially problematic since the support set contains only a handful of instances, or worse, instances that are affected by noise. Such as in Fig. 2(a), the class prototype p′ from support set S_t is biased away from the real class center p of the entire dataset, due to the presence of deviated support instances s₁,s₂. As a consequence, the generated prototypes may be not optimal, thereby compromising the accuracy of classification. Besides, the prototype itself is not suitable enough when it comes to classifying instances in the query set. This is largely due to the fact that the prediction is dependent on a one-way information flow from support set to query set. This leads to the prototype suffering from representation bias, which can have a negative impact on the performance of downstream tasks.

Fig. 2

(a) shows the case of deviated support instances, the interactive network makes Prototype p′ closer to p. The blue line indicates instances with less importance while the red line indicates instances with greater importance. (b) shows the process of reducing intra-class variance. The red line illustrates how the support instances are drawn closer together.

Statistically, the vast query instances in query set Q_detached can better stand for the whole data in terms of center and variance. We develop a novel interactive network f (S_t, Q_detached) for improving the support set S_t by leveraging the information present in Q_detached which ultimately lead to a new and improved support set, denoted by S_t′. $S_{t^{'}} = f (S_{t}, Q_{detached}) * S_{t}$ (7) To obtain a more typical S_t′, there are a few key metrics that must be taken into account. The distance between distribution center of S_t′ and real center p should be kept to a minimum. Additionally, it is equally important to ensure that the intra-class variance of S_t′ is reduced to the greatest extent possible. $\begin{matrix} \min {(E}_{S_{t^{'}} \sim p_{s^{'}}} [S_{t^{'}}] - p) \\ \min {(D}_{S_{t^{'}} \sim p_{s^{'}}} [S_{t^{'}}]) \end{matrix}$ (8) In this paper, Cross Attention mechanism is proposed to learn f (S_t, Q_detached) between support and query set to acquire S_t′ with modified distribution (center μ′ and variance σ′). The Cross Attention mechanism facilitates the bi-directional information flow between support and query set which enables S_t′ to better represent the entire dataset and learn more robust prototypes. $μ^{'} = \sum_{j = 1}^{n_{i}} f (S_{t}, Q_{detached}) * s_{j^{'}} | (s_{j^{'}} ϵ S_{t^{'}})$ (9) $σ^{'} = \frac{1}{(n_{i} - 1)} \sum_{j = 1}^{n_{i}} (s_{j^{'}} - μ^{'}) {(s_{j^{'}} - μ^{'})}^{T} | (s_{j^{'}} ϵ S_{t^{'}})$ (10)

3.1.1 Algorithm

We argue that not all instances in support set hold the same importance with a query set. In Fig. 2(a), s₃ is closer to the real center p and as a result, it should be given higher weight when generating the prototype. Conversely, s₁ and s₂ are much further away from p, and therefore should be assigned lower weights to minimize their influence on the final prototype. The interactive network assign weight a_i to each support sample $s_{c}^{i}$ which reflects its relevance to the query set through the Cross Attention mechanism. it ensures that the support set is optimized in a way highly responsive to the query set, thereby facilitating more interactive Instance Prototype, $Instance P = \sum_{i = 1}^{| S |} a_{i} * s_{c}^{i}$ (11)a_i is defined as follows, $a_{i} = Softmax (e_{i})$ (12) $e_{i} = Sum {tanh (s_{c}^{i} ⊙ [q_{1}, q_{2}, \dots, q_{| Q |}])}$ (13) where ⊙ is element-wise production and Sum{ · } means the sum of all elements which delegate the similarity of $s_{c}^{i}$ and whole Q_detached.

3.1.2 Instance loss

In the perspective of variance, Instance loss $L_{instance}$ is proposed to maximize the inter-class variance while simultaneously to minimize intra-class variance in support set. $L_{instance}$ also takes into account the interaction between support set and query set. Specifically, for a support sample $s_{c}^{i}$ , we project it on the whole Q_detached, obtaining a similarity distribution $d_{c}^{i}$ : $\begin{matrix} d_{c}^{i} & = s_{c}^{i} \cdot [q_{1}^{T}, q_{2}^{T}, \dots, q_{| Q |}^{T}] \\ = [{s_{c}^{i} q}_{1}^{T}, {s_{c}^{i} q}_{2}^{T}, \dots, s_{c}^{i} q_{| Q |}^{T}] \end{matrix}$ (14)

Where dot production is used to delegate the similarity between $s_{c}^{i}$ and every q_i in Q_detached. The resulting $d_{c}^{i}$ reflects the overall similarities and differences between support instance and query set. In $d_{c}^{i}$ , q_i with the same label c will get higher values and others get lower values.

Among all the support instances in S_t, the distribution for instances with the same label c, namely $d_{c}^{1}, . . ., d_{c}^{K},$ should be similar, whereas for instances with other labels should be different. In order to calculate the similarity between each $d_{c}^{i}$ , distribution matrix D is constructed as shown in Fig. 3, $\begin{matrix} D & = [s_{c}^{1}, s_{c}^{2}, \dots, s_{c}^{| S |}] \cdot [q_{1}^{T}, q_{2}^{T}, \dots, q_{| Q |}^{T}] \\ = [d_{c}^{1}, d_{c}^{2}, \dots, d_{c}^{| S |}] \end{matrix}$ (15)

$L_{instance}$ minimizes the distribution distance among instances with the same label (as $d_{p}^{1}$ and $d_{p}^{2}$ in Fig. 3), and maximizes the distribution distance among instances with different classes (as $d_{p}^{1}$ and $d_{n}^{1}$ in Fig. 3). $L_{instance} = \frac{\sum_{c ϵ C} f (d_{c}^{i}, d_{c}^{j})}{\sum_{c ϵ C} \sum_{c^{'} ϵ C \land {c}} f (d_{c}^{i}, d_{c^{'}}^{j})}$ (16)

Fig. 3

Calculation process of Instance loss.

c′ϵ C∧ { r } means c′ is not of the same class as c, and f ( · ·) represents the Euclidean distance between two distributions.

3.2 Concept prototype

By effectively utilizing query instances, Instance Prototype achieves excellent performance from support set to query set during the training phase. Nevertheless, there are inevitably inter-domain differences between the training set and testing set. When testing on new domains, the model trained exclusively on train domains, without information about the test domains, may generates prototypes that contain specific domain features unique to the train data. This could introduce pernicious bias and interference on predicting. For example, when conducting sentimental analysis, the presence of product category features from the training set could exert an unexpected impact on the model’s predictions on the testing set, leading to inaccurate results.

CLIP obtains outperforming generalizability through prompt learning based on class labels. Similarly, ZSD also leverages labels as knowledge on new classes classification. Inspired by these works, we integrate class concepts into prototypes by prompt engineering, which further enhances generalizability. In sentimental analysis, the label ‘Positive’ and ‘Negative’ would be used as concept in a ‘2-way’ setup. By capturing the semantic knowledge directly from the labels, we propose Concept Prototype which can be easier to discriminate between different sentiments, and immune to the negative effects of transfer to new domains.

Three kind of Concept Prototype strategies are proposed to leverage labels effectively: (1) class labels as prototype, (2) prompt template as prototype, and (3) prompt engineering for prototype.

Class labels

By feeding the class labels “Positive” and “Negative” into text encoder, the resulting feature vectors are directly acted as the Concept Prototype. This strategy is perfectly in line with ZSD.

Prompt template

Relying solely on labels may not capture all the nuances and complexities of the task. To address this, we employ described natural language templates as prompts, which can effectively capture more detailed semantics.

By utilizing a prompt template like “This is a {positive} comment.”, the model becomes capable of understanding that the task is related to sentiment analysis, which makes the prototype more interpretable.

Prompt engineering

It is unclear which templates are the most effective in representing semantic on manual prompt. CLIP integrates 80 templates, we hereby manually compose a set of unique templates to describe the task for the dataset [36]. CLIP treats every templates equally while we assume these templates should be customized for different scenarios. We adopt a self-attention method to focus on the templates that are more important, as shown in Fig. 4:

Fig. 4

Prompt Engineering for Concept Prototype.

Given a set of prompt templates P, $P = [p_{1}, p_{2}, \dots, p_{| P |}]$ (17) ${Sim}_{i} = Sum {tanh (p_{i} \cdot P^{T})}$ (18)Sim_i represents the relevance of template p_i to the entire set P. By determining the quality of p_i based on this relevance metric, it becomes possible to select the most effective templates. $β_{i} = softmax ({MAX}_{k} ({Sim}_{i}))$ (19) $Concept P = \sum_{i = 1}^{| P |} β_{i} * p_{i}$ (20)MAX_k (·) means top-ranked k templates are selected to construct the Concept Prototype.

3.3 Conceptual prototype

Concept Prototype is an abstract representation of class that does not contain a great deal of detail. However, by focusing on the intrinsic characteristics “Positive” and “Negative”, Concept Prototype can point to a wider range of domains, making it a powerful tool for achieving a more comprehensive understanding of sentiments.

In contrast, Instance Prototype is the concrete representation of class that contains diverse details about the instances like “satisfied”, “pleased”, “discontented” and “complaint”. This level of detail makes it an excellent tool for achieving specific analysis of particular problems.

To leverage the strengths of both prototypes, we synthesize the two prototypes into a Conceptual Prototype (CP). CP takes into account both the individual characteristics of the instance and the general attributes of the class simultaneously, enabling a more nuanced and comprehensive understanding.

To enhance the ability to learn and perform classification tasks, an MLP is used to project the abstract Concept Prototype and concrete Instance Prototype into a shared space where they are seamlessly integrated into a CP. $CP = MLP (Instance P, Concept P)$ (21)

By learning to adjust the importance of Instance Prototype and Concept Prototype in different situations, CP can effectively handle instances of different situations. In situations where instances are of low-quality, the Concept Prototype will play a more prominent role in guaranteeing the overall performance of the model. Conversely, when instances are high-quality and carry strong discriminative information, Instance Prototype will be given a higher weighting in the representation.

3.4 Framework

3.4.1 Prototypical contrastive learning

Contrastive learning is adopted as the metric module in our method. Further, Conceptual Prototypical Contrastive learning (CPCL) is proposed where the prototype is substituted by CP.

By enforcing the representations of instances to be more similar to their corresponding prototypes and far away from others, CPCL generates a clear classification boundary. The CPCL constrictive loss is defined as follows: ${L_{contrastive}}_{} = \sum_{i = 1}^{| S |} - log \frac{exp (s_{c}^{i} \cdot p_{c} / τ)}{\sum_{j = 1}^{K} exp (s_{c}^{i} \cdot {p_{c'}}^{j} / τ)}$ (22)

In contrast to the instance-wise contrastive learning, where the InfoNCE loss Equation (4) is computed between instance-level features, our $L_{contrastive}$ calculates the loss between features at the class level, which can be seen as a generalized version of the InfoNCE where the prototype for each instance is its augmented embedding.

CPCL imbues class semantics and instance diversity by means of CP, thereby empowering the model to obtain better and more precise representations. In addition to its impressive results in few-shot tasks, CPCL is remarkably suited to inter-domain transferring.

3.4.2 Objective function

The whole loss consists of three parts, as depicted in Fig. 5: (1) Instance loss $L_{instance}$ , (2) Prototypical Contrastive loss $L_{contrastive}$ and (3) Mean Square Error loss $L_{MSE}$ . $L = α L_{instance} + β L_{contrastive} + γ L_{MSE}$ (23) Where α, β, γ are the hyperparameters of the weights.

Fig. 5

Components of the objective function.

$L_{instance}$ is designed to reduce intra-class variance. $L_{MSE}$ better measures the correlation between instance and their corresponding prototype. But $L_{MSE}$ is unable to have any meaningful impact on the other prototypes, which can be complementarily realized by $L_{contrastive}$ to generate a clear classification boundary. $L_{MSE} = \sum_{i = 1}^{C} \sum_{j = 1}^{n} {(q_{i} - I (p_{j} = = p))}^{2}$ (24)

4 Experiments and conclusion

4.1 Datasets

Amazon Review Sentiment Classification (ARSC) The dataset comprises English reviews for 23 types of products on Amazon. For each product domain, there are three different binary classification tasks. These buckets then form 69 (23 × 3) tasks in total. Following [34], we select 12 (4 × 3) tasks from 4 domains (Books, DVD, Electronics and Kitchen) as the test set, and there are only five examples as support set for each label in the test set.

The Stanford Sentiment Treebank (SST-2) SST-2 consists of movie reviews from the Rotten Tomatoes website and includes sentence-level labels for binary sentiment classification (positive or negative). The sentences in SST-2 are short and simple, consisting of a single sentence that expresses a sentiment about a movie. We use 3000 sentences for training and 500 sentences for testing.

4.2 Parameters and settings

For the text encoder, we use RoBERTa-base [35], a pre-trained language model known for its effectiveness in various natural language processing tasks. The MLP consists of two hidden layers with dimensions of 1024 each, and the output layer has a dimension of 768. For the objective function, we take the values 0.4, 0.4, 0.2 for α, β, γ respectively. All models are trained on the base classes then the best-performing model on the validation are picked. The performance is evaluated by accuracy and F1 score, both of which have been widely recognized and acclaimed in the field of few-shot learning [2, 15].

The baseline models are trained by the ‘2-way 5-shot’ approach. In few-shot experiments, we repeat the experiments 5 times and perform K = [1, 3, 5, 10, 20] for comparison. For zero-shot experiments, we repeat experiments 3 times.

In addition to K instances as support set, each batch is populated with an equal number of positive and negative query instances. e.g. the ‘2-way 5-shot’ task has 27 positive and 27 negative query instances when the batch size is 64. It is worth noting that our model does not undergone any fine-tuning phase as its inherent generalization ability allows it to perform well across various tasks and domains.

The baseline models in our experiments are introduced as follows:

Prototypical Network (Proto) [2] is the first designed and applied to image classification and has also been used to deal with the text classification issue in recent studies.

Induction network (Induction) [3] proposes an induction module to induce the prototype by using dynamic routing.

Induction network with RoBERTa (Induction-R) replace the text encoder with RoBERTa on the base of induction network for fairly comparison.

HATT [6] is a hybrid attention-based prototypical network, which improves the model robustness greatly.

MLADA [37] is a meta-learning adversarial domain adaptation network, which aims to improve the adaptive ability and generate generalized text embeddings for new classes.

ContrastNet [38] trains the supervised contrastive representation model under the regularization of a task-level unsupervised contrastive loss and an instancelevel unsupervised contrastive loss, which can prevent overfitting and generate better representations.

4.3 Comparison with baselines

In order to verify the role and performance of the Instance Prototype and Concept Prototype, we introduce two separate models: Instance-PCL and Concept-PCL, which only use Instance Prototype and Concept Prototype respectively.

The classification results for the “2-way 5-shot” classification task on ARSC and SST-2 is are elucidated in Table 1.

Table 1
The “2-way 5-shot” classification results on ARSC and SST-2 datasets. Concept-PCL denotes the model only using Concept Prototype and PCL and Instance-PCL denotes the model only using Instance Prototype and PCL

Model ARSC SST-2

ACC(%) F1 ACC(%) F1

Proto 68.15 0.722 72.45 0.731

Induction 85.63 0.878 86.43 0.841

HATT 89.53 0.895 90.56 0.908

MLADA 90.94 0.910 91.45 0.916

ContrastNet 90.88 0.908 91.64 0.915

Induction-R 90.24 0.908 91.02 0.911

Concept-PCL 90.20 0.906 90.82 0.905

Instance-PCL 92.16 0.925 93.18 0.936

CPCL(Ours) 92.41 0.928 93.40 0.933

Model	ARSC	SST-2
Proto	68.15	0.722	72.45	0.731
Induction	85.63	0.878	86.43	0.841
HATT	89.53	0.895	90.56	0.908
MLADA	90.94	0.910	91.45	0.916
ContrastNet	90.88	0.908	91.64	0.915
Induction-R	90.24	0.908	91.02	0.911
Concept-PCL	90.20	0.906	90.82	0.905
Instance-PCL	92.16	0.925	93.18	0.936
CPCL(Ours)	92.41	0.928	93.40	0.933

It is worth mentioning that Concept-PCL, without any support instances, manages to achieve a comparable accuracy to Induction-R. We due the improvement to the fact that the Concept Prototype draws semantics from label through prompt engineering. The results indicate that Concept Prototype can acquire class knowledge and directly guide the query set complete the classification task.

On the other hand, by fully mining the interactions between support set and query set, Instance-PCL has the ability to focus on the support instances that are most critical, thereby effectively eliminating the impact of inaccurate support instances. The Instance Prototypes generated in Instance-PCL are more robust than those in Induction Networks, with an accuracy 1.82% higher.

The CPCL model has the best performance, achieving an impressive accuracy of 92.41% on the ARSC dataset. After learning from both low-level individual instances features and high-level semantics, CPCL is able to overcome the denotational ambiguity of purely labelled text, as well as acquire general concepts which is difficult to obtain from instances.

In our model, the loss function is composed of three parts: $L_{MSE}$ , $L_{contrastive}$ and $L_{instance}$ . Among these, $L_{instance}$ is associated with the Instance Prototype. For $L_{contrastive}$ and $L_{MSE}$ , we consider two ablated models: CPCL ( $L_{contrastive}$ ) that removes the $L_{MSE}$ from CPCL and CPCL ( $L_{MSE}$ ) that removes $L_{contrastive}$ from CPCL. From the ablation results shown in Table 2, we observe that CPCL ( $L_{contrastive}$ ) improves few-shot text classification performance upon CPCL ( $L_{MSE}$ ); CPCL further promotes CPCL ( $L_{contrastive}$ ). These results demonstrate the effectiveness of encouraging instances to have representations that are closer to their corresponding prototypes while pushing them away from prototypes of other classes. Additionally, $L_{MSE}$ and $L_{contrastive}$ are found to be complementary, as they utilize two different metrics to compute the distance between instances and prototypes in the feature space.

Table 2

The ablation result on objective function of CPCL for the “2-way 5-shot” task on ARSC dataset

$L_{instance}$	$L_{contrastive}$	$L_{MSE}$	ACC(%)
✓ 51	✓ 51	✗ 53	92.32
✓ 51	✗ 53	✓ 51	92.24
✓ 51	✓ 51	✓ 51	92.41

4.3.1 K-shot analysis

In order to meticulously evaluate CPCL, we conducted experiments considering the scale of few shots, with varying values of K. Figure 6 demonstrates that the models exhibit remarkable performance as K increases. This can be attributed to the fact that the prototypes contain more instance features become more robust. The few-shot task transforms into a 0-shot task when K decreases to 0 where Instance Prototype is invalidated and the model relies solely on the Concept Prototype to complete classification.

Fig. 6

The “2-way K-shot” classification results of CPCL and Induction-R on ARSC dataset. Note that when K is 0, it is meaningless for Induction-R since there is no support instances.

We can find that the Concept Prototype serves as a reliable foundation for ensuring advanced accuracy. Although 0-shot learning is meaningful and has a strong academic value, it may still be a tad distant from the practicalities of real-world scenarios. In fact, even for people, it is an arduous task to grasp new knowledge without any instances but limited extra information. Furthermore, it is challenging to construct a concept that is entirely pertinent to the current task. As such, the instance knowledge provided by the Instance Prototype is indeed essential, which can contribute substantially to the overall performance of the model.

When K is set to 1, the accuracy actually suffers and turns out to be even worse than 0 shot. This is attributed to the fact that the Instance Prototype only relies on a single instance, which can easily be influenced by low-quality instances, ultimately leading to unstable performance. This result aligns with the observations from experiments of CLIP, where excellent 0-shot performance is achieved, but the model falls short when it comes to 1 or 2 shots scenarios.

As K grows larger, the performance of CPCL is gradually improved compared to Induction-R. The support data becomes more reliable when K is more than 5, making the model more robust than in the case of 0 or 1 shot. However, as K continues to increase, the improvement of accuracy slows down.

4.3.2 Qualitative analysis

To further analyze the effect of Instance Prototype, we design experiments by setting different batch sizes.

Table 3
The “2-way 5-shot” classification result of CPCL under different batch size on ARSC dataset

model Batchsize ACC(%) Recall(%) F1

CPCL 32 92.02 92.06 0.923

64 92.41 92.43 0.928

128 92.59 92.57 0.931

model	Batchsize	ACC(%)	Recall(%)	F1
CPCL	32	92.02	92.06	0.923
	64	92.41	92.43	0.928
	128	92.59	92.57	0.931

Under fixed support instances, the larger batch means more query instances which can provide more distribution features. Specifically, when the batch size is set to 128, Instance Prototype can better stand for the whole dataset, resulting in a 0.57% increase in accuracy.

We leverage a query-detached set to provide additional distribution features for support set, which may be viewed as a leakage of query set. To prove the authenticity, we come up with an experiment in which a query memory bank is established to store the query instance from the past three batches. The prototype generated by current batch will be applied to the memory bank for classification. The results in Table 4 shows that the prototype generated by current batch does not suffer a decrease in accuracy in the memory bank, indicating that Instance Prototype acquires the spatial feature rather than label.

Table 4

The “2-way 5-shot” classification result of origin CPCL and CPCL with a memory bank on ARSC dataset

Model	ACC(%)
CPCL (memory bank)	92.39
CPCL	92.41

In Table 5, we have also recorded the distance between the prototype and the query set center, as well as the intra-class variance of the support set.

Table 5

The Distribution feature of support set of Induction-R and CPCL on ARSC dataset

Model	Center distance	Intra-class variance
Induction-R	8.752	15.769
CPCL	5.629	12.355

It is evident that prototype in CPCL is closer to the query set and less deviant. This indicates that CPCL can better capture the distribution of the query data. Moreover, CPCL reduces the intra-class variance, which diminishes the impact of low-quality instances.

We also investigate the impact of three distinctive concept prototype strategies. The results are diaplayed in Table 6. The direct use of class labels like “positive” and “negative” enhances semantic data to the prototype. However, class labels may lead to inadequacy due to lack of context. The construction of prompt templates can enrich text semantic information. Based on templates, prompt engineering strategy select more relevant templates to aggregate concept by self-attention. Compared with a single template, the performance is improved by 0.14%.

Table 6

The “2-way 5-shot” classification result of different strategies for generating Concept Prototype on ARSC dataset

Strategies	ACC(%)
Class labels	91.21
Prompt template	92.02
Prompt engineering	92.41

4.3.3 Domain transferring analysis

To verify the generalization of the model, we extra create a new dataset that reconstruct the categories of training set and testing set. Specifically, we construct three training sets “daily necessities”, “entertainments” and “electronic products”. The composition of three training sets is shown in the Table 7. For the test set, we select “electronics”, “kitchen housewares” and “DVD” respectively, corresponding to the three training sets mentioned above.

Table 7
The composition statistics of three training sets in ARSC dataset

Train set Test set Number

Daily necessities Beauty 1536

Baby products 2270

Apparel 4932

Gourmet food 818

Health personal care 3884

Entertainments Video 13294

Toy games 7088

Magazine 2226

Electronic products Computer video games 1466

Camera photo 3942

Cell phone service 544

The result presented in Table 8 clearly shows that when the testing category is similar to the training category, such as “Kitchen housewares” to “Daily necessities”, “DVD Book” to “Entertainments” and “Electronics” to “Electronic products”, the performance of the model is satisfactory. However, when transferring to other categories with less similarity, the accuracy of induction network decreases significantly, with the maximum degradation reaching 11.91% between “DVD Book” and “Electronics” when the training set is “Electronic products”. In contrast, CPCL maintains a consistently high level of performance regardless of the change of domains. We owe the outperforming lower limit to the generalization of concept.

Table 8

The “2-way 5-shot” classification result of CPCL transferring from training domains to testing domains on ARSC dataset

Train set	Test set	Induction-R	CPCL
Daily necessities	DVD Book	86.34%	91.77%
	Kitchen housewares	93.35%	94.44%
	Electronics	88.32%	92.28%
Entertainments	DVD Book	91.13%	93.85%
	Kitchen housewares	84.89%	91.61%
	Electronics	86.52%	91.69%
Electronic products	DVD Book	80.54%	91.12%
	Kitchen housewares	86.35%	92.53%
	Electronics	90.45%	94.12%

4.4 Conclusion

This paper presents a novel Conceptual Prototype (CP), which is the combination of Concept Prototype and Instance Prototype that provide a robust representation of class through the lens of instance and label concepts, to address the problem of inaccurate and limited generalizability of origin prototypes. We further apply CP to PCL to propose a Conceptual Prototypical Contrastive Learning (CPCL) which aims to bring instances closer to their corresponding prototype and push away from others. Experiments results demonstrate that CPCL outperforms other prototype-based methods on ARSC and SST-2. Besides, CPCL has a very high lower limit and is suitable for 0-shot tasks.

References

Tianshi

, Marc

, Sanja

A Theoretical Analysis of the Number of Shots in Few-Shot Learning, in: International Conference on Learning Representations, 2020.

Snell

, Swersky

, Zemel

Prototypical networks for few-shot learning, in: Advances in Neural Information Processing Systems (NIPS), 2017, pp. 4077–4087.

Geng

, Li

, Zhu

, Jian

, Sun

Induction Networks for Few-Shot Text Classification, in: Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, Association for Computational Linguistics, 2019, pp. 3904–3913.

Xu-Darme

, Qu’enot

, Chihani

, Rousset

Sanity checks and improvements for patch visualisation in prototype-based image classification ArXiv, abs/2302.08508.

, Gao

, Xhonneux

, Tang

Few-shot relation extraction via bayesian meta-learning on relation graphs in: International Conference on Machine Learning, PMLR, 2020, pp. 7867–7876.

Gao

, Han

, Zhu

, Liu

, Li

, Sun

, Zhou

FewRel 2.0: Towards more challenging fewshot relation classification, in: Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, Association for Computational Linguistics, 2019, pp. 6250–6255.

Gao

, Han

, Liu

and Sun

, Hybrid attention-based prototypical networks for noisy few-shot relation classification, in: AAAI Conference on Artificial Intelligence 33 (2019), 6407–6414.

Alec Radford , Jong Wook Kim , Chris Hallacy , Aditya Ramesh , Gabriel Goh , Sandhini Agarwal , Girish Sastry , Amanda Askell , Pamela Mishkin , Jack Clark et al.Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning, PMLR, 2021, pp. 8748–8763.

Kaiyang Zhou , Jingkang Yang , Chen Change Loy , Ziwei Liu Learning to prompt for vision-language models, International Journal of Computer Vision (2022b), 1–12.

10.

Kaiyang Zhou , Jingkang Yang , Chen Change Loy , Ziwei Liu Conditional prompt learning for vision-language models, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022a), 16816–16825.

11.

Joseph

K.J.

, Khan

F.S.

, Balasubramanian

V.N.

Towards OpenWorld Object Detection, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 5826–5836, doi: 10.1109/CVPR46437.2021.00577.

12.

Rahman

, Khan

, Barnes

Transductive Learning for Zero-Shot Object Detection, in: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 6081–6090, doi: 10.1109/ICCV.2019.00618.

13.

Hanoona Rasheed , Muhammad Maaz , Muhammad Uzair Khattak , Salman Khan , Fahad Shahbaz Khan, Bridging the gap between object and image-level representations for open-vocabulary detection, In Advances in Neural Information Processing Systems, 2022.

14.

Koch

, Zemel

, Salakhutdinov

, ICML Deep Learning Workshop 2 (2015), 0.

15.

Sung

, Yang

, Zhang

, Xiang

, Torr

P.H.S.

, Hospedales

T.M.

Learning to Compare: Relation Network for Few-Shot Learning, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 1199–1208, doi: 10.1109/CVPR.2018.00131.

16.

Dawid Rymarczyk , Lukasz Struski , Jacek Tabor , Bartosz Zielinski Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification, in: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.

17.

Dawid Rymarczyk, , Lukasz Struski , Michal G’orszczak , Koryna Lewandowska, , Jacek Tabor , Bartosz Zieli nski Interpretable image classification with differentiable prototypes assignment, in: European Conference on Computer Vision, 2021.

18.

Jonathan Donnelly , Alina Jade Barnett , Chaofan Chen Deformable protopnet: An interpretable image classifier using deformable prototypes, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10255–10265, 2022.

19.

Meike Nauta , Ron van Bree , Christin Seifert Neural prototype trees for interpretable fine-grained image recognition in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition 709 (CVPR), pages 14928–4938, 2021.

20.

Kahana Jonathan , Cohen Niv , Hoshen Yedid Improving Zero-Shot Models with Label Distribution Priors, 2022, arXiv:2212.00784

21.

Devlin

, Chang

M.-W.

, Lee

BERT: Pretraining of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,NAACL-HLT2019, Minneapolis, MN, USA, June 2-7, 2019, Volume1(Long and Short Papers).Association for Computational Linguistics, pp. 4171–4186.

22.

Tom Brown

, Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Aditya Ramesh , Daniel Ziegler

, Jeffrey Wu , Clemens Winter , Christopher Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , Dario Amodei Language models are few-shot learners, in: 34th International Conference on Neural Information Processing Systems (NIPS’20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 2020, pp. 1877–1901.

23.

Mark Chen , Jerry Tworek, , Heewoo Jun, , Qiming Yuan, , Henrique Ponde de Oliveira Pinto, et al. Evaluating large language models trained on code, 2021.

24.

Yoav Goldberg, Some remarks on large language models, 2023, URL https://gist.github.com/yoavg/59d174608e92e845c8994ac2e234c8a9.

25.

Timo Schick , Hinrich Schütze Exploiting cloze questions for few shot text classification and natural language inference, 2020, arXiv preprint arXiv:2001.07676

26.

Jason Wei, , Xuezhi Wang, , Dale Schuurmans, , Maarten Bosma, , Ed Chi, , Quoc Le , Denny Zhou, Chain of thought prompting elicits reasoning in large language models, 2022c, arXiv preprint arXiv:2201.11903.

27.

Ting Chen, , Simon Kornblith, , Mohammad Norouzi , Geoffrey Hinton, A simple framework for contrastive learning of visual representations, in: 37th International Conference on Machine Learning (ICML’20). 2020, JMLR.org, Article 149, 1597–1607.

28.

Zhirong Wu, , Yuanjun Xiong, , StellaYu

, Dahua Lin, Unsupervised feature learning via non-parametric instance discrimination, in: CVPR, 2018, pp. 3733–3742,

29.

Kaiming He , Haoqi Fan , Yuxin Wu , Saining Xie , Ross Girshick, Momentum contrast for unsupervised visual representation learning, in: CVPR, 2020.

30.

, Zhou

, Xiong

, Hoi

Prototypical contrastive learning of unsupervised representations, in: International Conference on Learning Representation (ICLR), 2021.

31.

Xie Binhui

et al. SPCL: A New Framework for Domain Adaptive Semantic Segmentation via Semantic Prototypebased Contrastive Learning, 2021, ArXiv abs/2111.12358.

32.

Dejiao Zhang , Feng Nan , Xiaokai Wei , Shang-Wen Li, , Henghui Zhu, , Kathleen McKeown, , Ramesh Nallapati,

Andrew Arnold

Bing Xiang, Supporting Clustering with Contrastive Learning, in: 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5419–5430, Online. 2021, Association for Computational Linguistics.

33.

Bateni

, Barber

, van Wood

J.-W.

Enhancing Few-Shot Image Classification with Unlabelled Examples, in: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2022, pp. 1597–1606, doi: 10.1109/WACV51458.2022.00166.

34.

Mo Yu , Xiaoxiao Guo , Jinfeng Yi , Shiyu Chang , Saloni Potdar , Yu Cheng , Gerald Tesauro , HaoyuWang , Bowen Zhou Diverse few-shot text classification with multiple metrics, in: 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018, V olume 1 (Long Papers), pages 1206–1215.

35.

Liu

, Ott

, Goyal

, Du

, Joshi

, Chen

, Levy

, Lewis

, Zettlemoyer

, Stoyanov

Roberta: A robustly optimized bert pretraining approach, 2019, arXiv preprint arXiv:1907.11692.

36.

Huang Shaohan , Dong Li , Wang Wenhui , Hao Yaru , Singhal Saksham , Ma Shuming , Lv Tengchao, , Cui Lei , Mohammed Owais , Liu Qiang, , Aggarwal Kriti, , Chi Zewen , Bjorck Johan , Chaudhary Vishrav , Som Subhojit , Song Xia , Wei Furu Language Is Not All You Need: Aligning Perception with Language Models, 2023, 10.48550/arXiv.2302.14045.

37.

Han

, Fan

, Zhang

, Qiu

, Gao

, Zhou

Meta-Learning Adversarial Domain Adaptation Network for Few-Shot Text Classification, in: ACL/IJCNLP, 2021, 1664–1673.

38.

Chen

, Zhang

, Mao

, Xu

ContrastNet: A Contrastive Learning Framework for Few-Shot Text Classification, in: AAAI, 2022. 10492–10500.

Train set	Test set	Number
Daily necessities	Beauty	1536
	Baby products	2270
	Apparel	4932
	Gourmet food	818
	Health personal care	3884
Entertainments	Video	13294
	Toy games	7088
	Magazine	2226
Electronic products	Computer video games	1466
	Camera photo	3942
	Cell phone service	544

CPCL: Conceptual prototypical contrastive learning for Few-Shot text classification

Abstract

Keywords

1 Introduction

2 Related work

2.1 Few-shot learning

2.3 Prototypical contrastive learning

3.4.1 Prototypical contrastive learning

4.1 Datasets

4.2 Parameters and settings

4.3 Comparison with baselines

Table 3 The “2-way 5-shot” classification result of CPCL under different batch size on ARSC dataset model Batchsize ACC(%) Recall(%) F1 CPCL 32 92.02 92.06 0.923 64 92.41 92.43 0.928 128 92.59 92.57 0.931

References

Table 3
The “2-way 5-shot” classification result of CPCL under different batch size on ARSC dataset

model Batchsize ACC(%) Recall(%) F1

CPCL 32 92.02 92.06 0.923

64 92.41 92.43 0.928

128 92.59 92.57 0.931