Region-Attention Prompt Learning for CLIP

2.1 Theoretical development

Vision Language Models  In recent years, there have been a series of works on building the connection between computer vision and natural language processing [1–4], e.g., text-to-image retrieval [16], image caption [17], visual question answering [18, 19], referring segmentation [20] and so on.

Those developments in image-text joint learning are largely driven by advances in the following three areas: (1) text representation learning with Transformers [21], (2) large-minibatch contrastive representation learning, and (3) web-scale training datasets [2, 22].

A representative approach is CLIP [5], which trains two neural network-based encoders using a contrastive loss to match pairs of images and texts. After consuming 400 million data pairs, CLIP shows competitive performance in various downstream tasks such as image recognition, objective detection, and dense prediction. Moreover, CLIP shows an impressive transferable ability to other datasets. Inspired by CLIP [5], several follow-ups have been proposed to improve the training strategy (e.g., CoOp [8], CLIP-Adapter [23], Tip-adapter [24], CoCoOp [14]) or apply CLIP to other domains (e.g., ActionCLIP [25]). Our work is built on CLIP, aiming to construct a more instructive and region-sensitive prompt to model more fine-grained region-text learning.

Prompt Learning  This topic is derived from Natural Language Processing (NLP) domain. The motivation of prompt learning is to exploit the knowledge learned by large-scale pre-trained models to perform various downstream tasks. Prompt learning can be summarized as “pre-train, prompt, and predict” [26], in which the downstream task is reorganized into a form similar to a pre-training task. Timo Schick and Hinrich Schütze [27] proposed PET, which adapts the language model to the task by modifying the input to the text classification task to a fill-in-the-blank question, while the inference process is modified to a text generation task to take full advantage of the text generation capabilities of the language model. Fabio Petroni et al. [28] proposed LAMA, which obtained better relational extraction than knowledge bases by modifying the relational extraction task to fill-in-the-blank questions, without modifying the pre-trained language model. Shin et al. [29] proposed AUTOPROMPT, which uses prompt learning for text classification, and text implication determination tasks. However, AUTOPROMPT sometimes struggles when the training data is highly imbalanced, and also lacks interpretability.

Motivated by the achieved prompt learning in NLP, several works have adapted prompt learning into VLMs. CoOp [8] replaces the manual prompt used in CLIP with a set of learnable vectors. Despite demonstrating outstanding performance, interpreting the results of CoOp, like other continuous prompt learning methods in natural language processing (NLP), poses a significant challenge. Furthermore, the conducted experiments indicate that CoOp is highly susceptible to the presence of noisy labels. Moreover, the learned context prompt is not generalizable to wider unseen classes within the same dataset,

Based on CoOp, prompt tuning methods for complex tasks have also been proposed, such as open-vocabulary object detection [30, 31], zero-shot semantic segmentation [32], continual learning [33], and multi-label image classification [34]. CoCoOp [14] extends CoOp by further learning a lightweight neural network to generate for each image an input-conditional token(vector). Although CoCoOp introduces instance-level image features for context prompt, it assigns the same image features to each category to construct prompt. Based on this, the model still fails to recognize the intrinsic semantic associations of different classes of text and images, resulting in semantic ambiguity and inaccuracy of prompt.

Our work aims to explore the use of image information to build a category-sensitive prompt, which we hope can adaptively focus on different regions in the image according to different category descriptions, so we call it Region Attention Prompt.

Zero-Shot Learning (ZSL)  ZSL is another relevant research area, i.e., to recognize new classes by training only on base classes [3, 35–37]. Moreover, the "seen-class bias" issue brought up in the ZSL literature is related to the generalization problem, where a model trained on base classes frequently fails on new classes. VLMs like CLIP [5] also suffer from the generalization problem, this problem is attributed to the text form mismatch between the pre-trained data and downstream tasks data, which is detrimental to alignment learning.

The most common approach to ZSL is to learn a semantic space based on auxiliary information such as attributes or word embedding. Different from existing ZSL methods, our work addresses the above problem by using image-based information to construct an instance-level prompt called Region-Attention Prompt (RAP). RA-CLIP is further proposed to improve image classification performance under generalization scenarios.

OPEN IN VIEWER

Fig. 1

Semantic Interaction Network(SINet), which makes prompt focus on those category-related regions (cat’s head, tail, paw). SINet can assign a higher attention weight to prompt, while a lower one to background regions (grass and trees).

CLIP  is a powerful vision-language pre-training model using contrastive learning to align image and text representation in the same vector space. CLIP consists of two encoders, including an image encoder (ResNet [38] or ViT [39] and a text encoder (Transformer [21]. When performing zero-shot learning, CLIP [5] uses a manual prompt “a photo of a CLASS” to transform the origin category text into the complete sentence, then feeds the sentence into text encoder and selects the category with the greatest similarity as the label of the image. It should be noticed in this process CLIP does not need to perform additional parameters training [8, 25].

Context Optimization (CoOp)  is proposed to extend CLIP’s ability in few-shot tasks. Motivated by prompt learning in the NLP field, CoOp [8] introduces M learnable context vectors (soft prompt) {v₁, v₂, … , v _M  } to replace the manual template “a photo of a {CLASS}”, where each has the same dimension with the word embedding. The key idea of the CoOp is to make use of the prompt learning method in the text encoder by only tuning the learnable context vectors, then the CLIP-like models can learn a better prompt representation.

Conditional Context Optimization (CoCoOp)  is proposed to improve CoOp’s generalizability. Specifically, CoCoOp [14] further learns a lightweight neural network to generate for each image an input-conditional token. Then is added to soft prompt to obtain an instance-level prompt for each image. Same as CoOp, CoCoOp maximizes the cross-entropy score from the ground-truth to update the learnable context vectors and light-weight linear-transformation head by gradient propagation, with both image encoder and text encoder freezing during training.

3.1 Region features analysis

A good prompt in CLIP should be instructive and precise, thus helping alignment learning more adapt to downstream tasks. For image classification tasks, the information in different regions of the image does not have the same relevance to the category. Moreover, assuming that the classification object in the image is not obvious, or there are other focal parts in the image irrelevant to the classification, then simply making the whole image interact with category text to construct the prompt may be harmful to CLIP’s alignment ability. It can be found that category texts provide conceptual-level semantic descriptions and regions offer the more precise and instance-level ones, so refining the image features with category features can obtain more instructive and fine-grained region information.

Therefore, in constructing the prompt for image classification tasks, it is necessary to pay more attention to regions that contribute more to the category texts, which makes prompt more sensitive and precise to important regions. This prompting method can enrich the semantic representation of text descriptions, thus improving CLIP’s image-text level alignment capabilities.

To illustrate that focusing on category-related regions is beneficial for better alignment in CLIP, the region-text similarity experiments are conducted on OxfordPets dataset. We split a complete image into 4 regions (region 1 - 4) and additionally extract the classification object region (region*) from the image, then the similarity between classes and regions/region* is calculated by the dot-product operation. Greater similarity indicates better alignment.

Table 1 shows that for a cat image, region* (which is more relevant to the class “cat”) achieves the highest similarity score 0.942 than the other regions and full image. It indicates that it is beneficial for CLIP’s alignment to use category-related regions to construct the prompt. Moreover, those regions that are not relevant to the “cat” (region1, region3) may further be classified into the wrong category (such as “dog” or “lion”) thus hurting CLIP’s alignment ability.

In general, an instance-level prompt called Region-Attention Prompt (RAP) is proposed to model fine-grained region-text joint embedding learning. RAP introduces region-attention features under the guidance of category texts, which can enrich the semantic representation of the text description and can avoid the referencing ambiguity caused by unimodal prompts.

SINet is built to generate RAP, which uses the Cross-Attention mechanism to make every category interact with the image to assign greater attention weight to the region with higher semantical relevance to the category. In this process, all category texts act as queries while images serve as keys/values. SINet allows each category text to obtain unique region-attention features that make prompt more instructive and precise to important regions. Also, benefiting from the properties of the Cross-Attention mechanism, SINet can maximize the use of semantic information from both modalities. Specifically, SINet consists of two modules:

Transformer Decoder Module  Let T _θ (·) denote the Transformer Decoder Module parameterized by θ, x ¯ indicates the image features generated by the image encoder. And let { c i } i = 1 K be word embedding (s) for the K category texts produced by the text encoder (suppose there are K classes in total). K category features { c i } i = 1 K and the image features x ¯ are passed through the Transformer Decoder Module T _θ (·) to get the region-attention features: { e i } i = 1 K = T θ ( { c i } i = 1 K , x ¯ )(1) where in Eq. (1) each region-attention features e _i corresponds to c _i of the same category name.

Projector Module  To generate RAP a lightweight neural network called Projector Module is further learned. Specifically, let G _η (·) denote the Projector Module parameterized by η, for the i - th category, region-attention features e _i are passed through the projector G _η (·). We can use Eq. (2) to get RAP: p i = G η ( e i )(2)

In general, CLIP is equipped with RAP (called RA-CLIP), the framework of RA-CLIP is shown in Fig. 2, which mainly includes the following parts:

OPEN IN VIEWER

Fig. 2

Framework of RA-CLIP, where both text encoders are original CLIP text encoders and share the same pre-training parameters. "concat" means that RAPs are concatenated with class embedding as inputs to the right text encoder(the number of RAPs is always equal to the number of classes). Meanwhile, in order to maintain the consistency of the text encoder, we pad the token length of the concatenated vectors to 77.

Modified Image Encoder  Motivated by DenseCLIP [40], we modify CLIP’s image encoder as f′ (·) to additionally adopt non-global averaged pooled features after MHSA (Multi-Head Self-Attention) layer (called the language-compatible features) as the input of SINet.

Language-compatible features contain sufficient image space features while being well aligned with text features. Eq. (3) can be used to obtain the language-compatible features: x ¯ = f ′ ( I )(3) where I denotes the original image.

Image Encoder  To maintain the uniformity of RA-CLIP, the original CLIP’s image encoder f (·) is used to generate the image features: x = f ( I )(4)SINet  Let SINet _θ,η (· , ·) be the SINet parameterized by θ and η, then SINet can be represented in the form of Eq. (5): SINet θ , η ( · , · ) = { T θ ( · , · ) , G η ( · ) }(5)

And RAP can be denoted by Eq. (6): p i = SINet θ , η ( c i , x ¯ )(6)Text Encoder  The original pre-trained CLIP’s text encoder g (·) is adopted as our feature extractor. Note that the two text encoders in the framework are the same and shared among all data.

Let’s assume a dataset τ with training data D train = { ( T i , I i ) } i = 1 K ,where K denotes the number of classes, T denotes category texts. The first text encoder is used to get word embedding(s) and take M token lengths as the input of SINet. c i = g ( T i ) [ : , : M , : ](7) where in Eq. (7) M is a hyperparameter.

Then the K classes features are prepended after RAP to get the input of the second text encoder: { ω i } i = 1 K = concat ( { p i } i = 1 K , g ( { T i } i = 1 K ) [ : , M : , : ] )(8)

For example, the input of the second text encoder for the i ^th category now becomes: ω i = concat ( p i , g ( T i ) [ : , M : , ; ] )(9)Loss Function  RA-CLIP is performed to minimize the standard classification loss, and the gradients can be back-propagated all the way through the text encoder to optimize the SINet.

Specifically, the cross-entropy loss is adopted as our loss function to maximize the score from the ground-truth, which can be denoted by Eq. (10): p ( y ∣ x ) = exp ( sim ( x , g ( ω y ) ) / τ ) ∑ i = 1 K exp ( sim ( x , g ( ω i ) ) / τ )(10) where K denotes the number of classes.

During training, only the parameters of SINet are updated, both text encoders and image encoders are frozen.

In this work, the Transformer Decoder Module is built on six-layer Transformer Decoder Blocks with eight heads, and the Projector Module is built with a two-layer bottleneck structure (Linear-ReLU-Linear), with the hidden layer reducing to 64 dimensions.

RA-CLIP is evaluated over 7 datasets in the base-to-new generalization setting, the results are averaged over three runs. CoOp [8], CoCoOp [14], and RA-CLIP are learning-based methods, CLIP [5] is the zero-shot prompt. The detailed results are shown in Table 3, which strongly justifies the strong generalization performance of RA-CLIP.

RA-CLIP vs. CLIP  As shown in Fig. 3, for new classes RAP outperforms CLIP in 5 datasets and suffers the loss in 2 datasets (FGVCAircraft and DTD) But RAP still achieves a better performance in new classes on average, which indicates that RAP already has similar or even better generalizability than CLIP on most datasets.

OPEN IN VIEWER

Fig. 3

Comparisons of RA-CLIP and CLIP in the base-to-new generalization setting.

For base classes, RA-CLIP outperforms CLIP by a large margin on all 7 datasets (up to 33.59%, least to 0.64%). This indicates that learning-based prompt methods can more effectively adapt training data, thus providing better semantic representation for base classes. Moreover, RA-CLIP’s gains in base classes far outweigh its losses in new classes than CLIP, thus achieving better harmonic performance, e.g., on DTD dataset in base classes is +27.94% while its losses in base classes are -3.06% or RAP’s gains on FGVCAircraft dataset in base classes are +9.63% while its losses in new classes are just -2.18%.

RA-CLIP vs. CoOp  In comparison to CoOp (see Fig. 4), performance drops in the base classes occur for RA-CLIP on 3 datasets (StanfordCars, FGVCAircraft, EuroSAT). This is reasonable because CoOp optimizes specifically for base classes, while RA-CLIP optimizes for each specific image in order to gain the instance-level prompt over an entire task (both new classes and base classes).

OPEN IN VIEWER

Fig. 4

Comparisons of RA-CLIP and CoOp in the base-to-new generalization setting.

But it is worth noting that although RA-CLIP’s base accuracy drops below CoOp on 3 of 7 datasets (range from -2.12% to -3.72%), RA-CLIP’s gains in new classes are significantly larger than its losses, which is enough to turn the averages improvement into positives, e.g., StanfordCars sees the worst base accuracy drop of -3.72% but obtains accuracy gain of +14.55% in new classes, which together bring a 10.83% positive improvement for RA-CLIP, or StanfordCars sees the base accuracy drop of -2.12% but also obtains accuracy gain of +16.64% in new classes, which together bring a 14.52% positive harmonic improvement for RA-CLIP.

RA-CLIP vs. CoCoOp  As shown in Fig. 5, when considering both base and new classes (harmonic performance), RAP shows a gain of 3.52% over CoCoOp on average (77.67% vs. 74.15%, see Table 3). Specifically, for new classes, RA-CLIP outperforms CoCoOp on all datasets (specifically on EuroSAT and FGVCAircraft) by a relatively large margin.

OPEN IN VIEWER

Fig. 5

Comparisons of RA-CLIP and CoCoOp in the base-to-new generalization setting.

For base classes, RAP still outperforms CoCoOp on all datasets, (specifically on DTD, StanfordCars, FGVCAircraft, and EuroSAT). In general, compared to CoCoOp (which adopts the image-condition features to construct prompt), RAP is more competitive in both base and new classes. It suggests that using region-attention features to construct prompt can provide a more precise and instructive semantic representation for text description.

To understand the working mechanism of RAP and explore how RAP improves image classification performance, we visualized the attention weights of RAP on oxfordPets and Food101 datasets. In our experiments, we visualized RAP’s attentional weights on the base class and the new class, respectively, and for each image, we sampled the attention feature maps corresponding to the three classes (only one of which is GroundTruth).

The detailed results are shown in Table 5 and Table 6, it can be inferred that regardless of whether it is on base classes or new classes, Ground-Truth-RAP always focuses on the correct regions in the image, and assigns higher attention weights to these regions. Meanwhile, the rest of the RAPs still maintain lower attention weights overall. This suggests that RA-CLIP is conditioned on each input instance, it always focuses on regions that are semantically similar to category and assigns high attention weights. Meanwhile, RAP can be optimized to characterize each instance, so it is more robust to class shift. In other words, RA-CLIP learns a pattern of extracting region-level features, when comes to the generalization scenario the learned extraction pattern helps RA-CLIP obtains the instance-level region features.

Moreover, to explore the advantages of this regional attention feature in model alignment learning, we further measured the cosine similarity between the Ground-Truth-RAP and the image on OxfordPets dataset(see Table 7). The larger the η, the closer the semantic distance between Ground-Truth-RAP and the image, which indicates stronger alignment ability. The results in Table 7 demonstrate that RAP - the prompting approach that utilizes the Cross-Attention mechanism to acquire region features - is able to bring images and classes closer in the vector space, thus improving the model’s alignment ability (image classification performance).

The results in Table 7 show that the larger η (RA-CLIP) helps to improve generalizability. It indicates that RAP has a more similar semantic representation to image features in vector space, thus enhancing the alignment ability. Moreover, RA-CLIP also improves base accuracy, which means RA-CLIP can reduce the generalizability trade-off between base classes and new classes (increase the harmonic performance).

The Necessity of Transformer Decoder Module  To demonstrate the key role of region-attention features in image classification tasks, an Image-Text prompt (IT-prompt) is proposed as a method of comparison. Specifically, IT-prompt makes image features pass through a linear projection layer and then directly point-wise-add to embedding(s) of category to obtain prompt. In the comparison experiment, RAP is replaced with IT-prompt, and the rest of the model structure and training parameters remain unchanged.

Comparison experiments over 7 datasets are shown in Fig. 6. The result shows that RA-CLIP outperforms CLIP with IT-Prompt on both base classes and new classes by 1.7% approximately, suggesting the key to enhancing alignment capability is using region features to construct prompt.

OPEN IN VIEWER

Fig. 6

RAP vs. IT-prompt.

The Effectiveness of Projector Module  In this experiment, the removal of the Projector Module in SINet, while keeping the model structure and training parameters unchanged, resulted in a degradation of classification performance on both the base classes and the new classes. This deterioration can be attributed to the incomplete representation and reduced parameterization capability.

The Projector Module plays a crucial role in extracting higher-level abstract features through additional nonlinear transformations and mappings. By removing the Projector Module, the model loses the ability to fully leverage the output of the Transform Decoder Module, leading to incomplete feature representations and consequently impacting performance.

OPEN IN VIEWER

Fig. 7

SINet vs. SINet without Projector Module on Caltech101 and StanfordCars datasets.

The Length of RAP  To figure out whether the length of RAP affects the generalizability, a study is conducted by adopting the different lengths (specifically 4, 8, and 16 token lengths) of RAP while keeping all the other parameters identical. Fig. 8 summarizes the average results on the 7 datasets, and it shows that RAP with 8 tokens length performs better in both base classes and new classes than the others.

OPEN IN VIEWER

Fig. 8

Study on the length of RAP over 7 datasets.

This indicates that a suitable length of RAP (which is 8 token lengths in this work from our experience) has contain the most precise region features (semantic representation), which can maximize the model’s alignment ability. On the contrary, RAP with longer length may suffer from information redundancy and ambiguous referents, while RAP with shorter length may suffer from information missing for important regions, both cases will result in inadequate representation ability of prompt.

The Scale of SINet  To find out if the scale of SINet affects the image classification performance of RA-CLIP, we conduct experiments over all 7 datasets, where the different scales of SINet are proposed (RA-CLIP-Small, RA-CLIP-Standard, RA-CLIP-Large). In addition, CoCoOp is introduced as a baseline to verify whether a smaller RA-CLIP can still achieve better results than CoCoOp.

Table 8 demonstrates that as the scale increases RA-CLIP also gains a slight performance improvement, but the increase in performance slows down significantly. However, it is important to note that the model also runs the risk of increased computational cost and overfitting as the size increases. Moreover, SINet-small can still achieve better performance on both the new classes and the base classes than CoCoOp. It indicates that region features adopted in RAP are the key to guaranteeing the alignment ability, rather than the training parameters of SINet.

Training Epochs  In order to explore the effect of training epochs on performance and whether fewer training epochs can also yield desirable results, different training epoch (5/10/20/30 epochs) is adopted in the experiment. Fig. 9 shows that as the number of training epochs increases RA-CLIP still gains in classification performance on both base and new classes. However, the performance growth rate on new classes is getting slower and slower, which may be due to the fact that too many training rounds lead to the overfitting of RA-CLIP on the base classes. Meanwhile, for fewer training epochs, RA-CLIP still outperforms CoCoOp, which suggests that RA-CLIP is an efficient prompt-tuning method that can ensure high performance while converging quickly during training.

OPEN IN VIEWER

Fig. 9

Training epoch experiments.

To understand the use of templated category descriptions (like CLIP) affects the RA-CLIPs’ performance, an experiment is conducted over 7 datasets. In the experiment, a template is adopted to expand the classes into one sentence as the input of SINet, following CLIP [5], we use "a photo of a CLASS" as a template to compare with the non-templated RA-CLIP. The detailed results are shown in Fig. 10.

OPEN IN VIEWER

Fig. 10

Template experiments over 7 datasets.

Fig. 10 shows that templated RA-CLIP has a certain degree of regression on both base classes and new classes (from 1.26% and 0.60% drops compared to the original RAP). This illustrates the original RA-CLIP already contains enough conceptual-level semantic representation to guide the image in extracting important region features. On the contrary, using additional templates for classes may complicate the text semantics and thus be detrimental to the generation of RAP.

In order to analyze the computational resource cost as well as the scalability of RA-CLIP, we conduct comparison experiments with 7 datasets (see Table 9) on each of the three computing platforms(Nvidia 3090, Nvidia 3080Ti, Nvidia Tesla T4). Meanwhile, we introduce CoCoOp on Nvidia 3090 computing platform as a baseline to explore the trade-off between performance improvement and computational cost.

Table 9 shows that on the Nvidia Tesla T4 platform, RA-CLIP’s training/inference times and GPU memory consumption are marginally longer than CoCoOp. This is likely due to the increased complexity and parameters of RA-CLIP, which necessitates more computational time. However, given the enhanced performance of RA-CLIP, this increase in computation time is acceptable. Besides, RA-CLIP exhibits robust scalability across a diverse range of datasets. It delivers stable and reliable performance on datasets of various scales, from those requiring fewer computational resources such as OxfordPets, to those imposing higher ones like StanfordCars. This illustrates the model’s capability to effectively scale across datasets of various types and magnitudes. Furthermore, RA-CLIP’s memory usage and GPU memory utilization remain relatively consistent across all platforms and datasets. This stability underlines the model’s proficient resource management, demonstrating its ability to maintain an efficient balance between computational resource expenditure and performance outcomes.

The first limitation pertains to the training process’s efficiency. RA-CLIP was found to be notably slow during its training phase, exhibiting a high consumption of GPU memory. Our runtime analysis indicates that when the batch size exceeds 4, it becomes untenable for standard civilian-grade GPUs. This is primarily attributed to the proposed SINET, which necessitates independent forward and backward propagation for each image in the minibatch during training to construct an instance-level Region Attention Prompt. This is markedly less efficient than both CLIP and CoOp, which require only a single forward and backward propagation through the text encoder for the entire minibatch, irrespective of its size.

The second limitation emerges in the detailed performance analysis. Despite RA-CLIP demonstrating the best overall performance across all 7 datasets, it does not consistently yield the most optimal results in either the base or new classes. Specifically, it failed to provide the top performance for base category recognition on 3 datasets and new category recognition on 2 datasets. This underscores the necessity for a substantial community effort in the pursuit of fully bridging or even reversing the gap between manually-designed prompts and learning-based prompts.

4.8 Conclusion

An instance-level and category-sensitive prompt called Region-Attention Prompt (RAP) is proposed in this paper to model fine-grained region-text joint embedding learning. RAP employs a unique strategy of integrating region features into prompting, obtained via a Cross-Attention mechanism between image and text data. In essence, RAP is a region-level prompt with category-sensitive attributes that adaptively assign a greater attention weight to image regions possessing higher semantic relevance to the category. Semantic Interaction Network (SINet) is proposed to generate RAP, which includes Transformer Decoder Module and Projector Module. Besides, CLIP is equipped with RAP (called RA-CLIP) to improve image classification performance in generalization scenarios. A series of extensive experiments across seven datasets show that RA-CLIP surpasses the current state-of-the-art method, CoCoOp, by a margin of 0.4% - 4.16% on base classes and 0.25% - 11.34% on new classes. Furthermore, our findings reveal that by focusing on category-related regions to construct prompts, as opposed to uniformly focusing on the entire image, the model’s alignment ability can be further improved, demonstrating the efficiency of RAP in boosting the image classification performance of VLMs.