Robustness test-time augmentation via learnable aggregation and anomaly detection

Abstract

Test-time augmentation (TTA) has become a widely adopted technique in the computer vision field, which can improve the prediction performance of models by aggregating the predictions of multiple augmented test samples without additional training or hyperparameter tuning. While previous research has demonstrated the effectiveness of TTA in visual tasks, its application in natural language processing (NLP) tasks remains challenging due to complexities such as varying text lengths, discretization of word elements, and missing word elements. These unfavorable factors make it difficult to preserve the label invariance of the standard TTA method for augmented text samples. Therefore, this paper proposes a novel TTA technique called Defy, which combines nearest-neighbor anomaly detection algorithm and an adaptive weighting network architecture with a bidirectional KL divergence entropy regularization term between the original sample and the aggregated sample, to encourage the model to make more consistent and reliable predictions for various augmented samples. Additionally, by comparing with Defy, the paper further explores the problem that common TTA methods may impair the semantic meaning of the text during augmentation, leading to a shift in the model’s prediction results from correct to corrupt. Extensive experimental results demonstrate that Defy consistently outperforms existing TTA methods in various text classification tasks and brings consistent improvements across different mainstream models.

Keywords

Test-time augmentation test-time robustification text classification language model anomaly detection

1 Introduction

Pre-trained language models (PLMs) have achieved remarkable success in various natural language processing (NLP) tasks [13 , 47]. Nevertheless, most of their works focus on improving the performance of PLMs during training, which involves the utilization of models with a higher number of parameters, improved network structures [13, 45], various data augmentation methods [44 , 58], and adversarial training [27]. But the robustness of PLMs during inference is often overlooked.

Typically, data augmentation is applied before or during model training, but rencent studies have shown that data augmentation can also be applied during model inference [29, 48], which is known as test-time augmentation (TTA). TTA is a tried-and-true method for improving the final prediction accuracy and robustness of language models by aggregating augmented samples. TTA does not rely on any hyper-parameter tuning and can improve the robustness of the model to test data perturbations without additional training. TTA has been widely used in computer vision tasks [2 , 52], and it has been proven to be effective in improving the accuracy and robustness of models. However, the application of TTA in the field of natural language processing (NLP) is still in its infancy, and there are still some obstacles to overcome. In contrast to the widespread use of TTA in computer vision tasks, where the mainly utilized data augmentation methods such as rotation, rescaling, and translation can largely preserve labels and convey crucial visual information about the described objects or scenes. However, the TTA methods used in text processing tasks, such as synonym replacement, random word deletion, and word position exchanging often alter the semantic and grammatical structure of the text, making it challenging to select effective augmented samples for aggregation and difficult to ensure that the labels of aggregated samples are not damaged, which is crucial for the accuracy and stability of the model.

Another obstacle is how to more effectively aggregate multiple augmented data samples to improve the robustness of the model. The standard TTA method simply averages the prediction results of all augmented samples, which can bring some performance improvements, but it ignores the fact that not every augmented sample can bring value gains.

Simply averaging the probability prediction results of all augmented samples often leads to significant prediction biases due to the influence of outlier augmented samples. Although large-scale pre-trained language models have been proven to have some ability to recognize out-of-distribution data [15], but the accuracy of these models is easily disrupted by slight perturbations and lead to a drop in performance [1]. Therefore, how to calibrate and filter outliers in the augmented samples [8, 17], reassign sample weights, and minimize the risk of damaging model performance caused by anomalous augmented samples is a challenging problem.

In this work, we focus on how to improve the robustness of TTA aggregation while avoiding overemphasizing the influence of anomalous augmented samples. [50] focuses on the selection of data augmentation methods during testing, whereas our focus lies in enhancing test-time aggregation and, therefore, does not involve discussions in this regard. We present the related work in Section 2, provide the problem definition in Section 3, and formally propose our methodology in Section 4. Furthermore, We conducted extensive ablation experiments on various factors that may affect TTA, including backbone models, data augmentation methods and the number of augmented samples, in order to seek effective means of applying TTA in the NLP field. And our main contributions are as follows:

By adding a regularization term based on the bidirectional KL divergence entropy between the distribution of original samples and the aggregated samples to the conventional cross-entropy method. This promotes consistency in predictions across different augmented samples and is better able to adapt to non-uniform weights among augmented samples.

Incorporating anomaly Nearest Neighbor (ANN) detection for calibrating abnormal augmentations, our method can effectively filter out abnormal augmented samples to minimize the risk of uncertainty amplification caused by different augmentation methods.

The versatility of our method is evident from its ability to be applied "plug-and-play" to any existing based model without any hyper-parameter tuning, allowing us to work seamlessly with other techniques that enhance model robustness.

2 Related work

2.1 Test-time augmentation

TTA is a technique applied to a trained model during testing, where multiple augmented samples are generated for each original sample, and the average prediction over the augmented samples is used as the aggregated result to improve the final output of the model. Although data augmentation is typically applied during model training but can also use during prediction. And TTA has been widely demonstrated to enhance model accuracy and robustness [24], as well as address distribution shift issues [60] and defend against adversarial attacks [40]. Researchers have proposed various TTA methods, such as [22] proposed an instance-aware TTA algorithm based on a loss predictor that dynamically selects samples for TTA [39] used Mixup algorithm [59] to mix the input with other randomly clean augmented samples, thereby improving the model’s ability to withstand adversarial attacks after TTA [31] introduced a greedy search strategy to select samples for augmentation [31] discussed the impact of various basic text data augmentation strategies on model accuracy in text classification applications.

2.2 Anomaly detection

Anomaly detection technique involves identifying unexpected behavior in data that leads to anomalies, deviations, or outliers, and assigning them uncertainty (anomaly) scores. Meanwhile, it is also commonly used to calibrate the uncertainty of the anomalous data [10]. Traditional anomaly detection techniques are mainly based on statistical methods, while machine learning-based techniques further employ self-supervised, unsupervised, or semi-supervised methods, such as One-Class SVM [47] and Local Outlier Factor (LOF), which use self-supervised and contrastive learning to achieve better performance in detecting neighboring samples [16]. Deep learning-based techniques, on the other hand, use neural networks for anomaly detection, such as unsupervised LSTM-based anomaly detection [38], BERT-based anomaly detection [12], and adaptive threshold-based streaming anomaly detection [43].

2.3 Ensemble selection

Ensemble learning is a machine learning strategy that makes overall decisions based on the prediction results of multiple models [5]. It can improve the overall performance and robustness of the model, but the cost is that a large amount of computational resources are required to train these models. Although TTA is often used in conjunction with a single model, it is meaningful to consider TTA as an aggregation of different models. This is because during TTA, each augmented data sample is equivalent to a new test sample, and when multiple test results based on these samples are aggregated, it is approximating a local ensemble learning process. A typical example is that [9] proposed to select the optimal clustering results from multiple clustering results for aggregation.

3 Problem formulation

Test Time Augmentation (TTA) is a technique used to improve the performance of a model during the inference phase. It involves creating multiple versions of each test sample using data augmentation techniques, and then averaging the model’s predictions for each version to produce the final prediction. This method is based on the assumption that the model may recognize different features in the different augmented versions of a sample, leading to a more robust prediction.

Let’s denote the original input sample as x and the model’s prediction function as f. In the usual scenario without TTA, the prediction for the sample would be f (x).

The TTA technique involves applying a set of augmentation transformations T = {t₁, t₂, . . . , t_m} to the sample x, creating a set of augmented samples $X_{N} = t_{1} (x), t_{2} (x), . . ., t_{m} (x)$ . The model then makes predictions for each of these augmented samples, resulting in a set of predictions P_tta = {f (t₁ (x)) , . . . , f (t_m (x))}.

The final prediction p_tta for the original sample x is then obtained by averaging the predictions in P_tta. This can be formally expressed as: $p_{tta} = \frac{1}{N} \sum_{i = 1}^{N} f (t_{i} (x))$ (1)

The problem of TTA thus lies in selecting the appropriate set of augmentation transformations T and in determining how to aggregate the set of predictions P to produce the final prediction. Different choices for T and the aggregation method can lead to different performance characteristics for the TTA technique.

One common challenge is deciding which augmentation techniques to include in T. Not all techniques may be beneficial for all types of data or models, and some may even harm the model’s performance if they introduce too much noise or distort the data in a way that the model cannot handle.

Another challenge is how to aggregate the set of predictions P_tta to produce the final prediction. It is necessary to assign appropriate weights to different data augmentation samples to eliminate the negative impact of certain samples.

Despite these challenges, TTA has been shown to be a valuable technique that can significantly improve model performance, making it a worthwhile area of study and experimentation.

3.1 Problem analysis

As shown in Fig. 3.1, when using the standard TTA method to generate augmented samples for aggregated prediction, it is prone to cause the final prediction results of the model to deviate from the ground truth. This is because the prediction results of the augmented samples may not all bring net value-added benefits, so we need to effectively measure, screen and allocate weights based on the intrinsic characteristics of the augmented samples to minimize the risk of damage caused by abnormal augmented samples. If we simply use the standard TTA method to average the prediction results of all augmented samples, the final prediction results are easily affected by abnormal augmented samples, leading to deviation from the ground truth.

Fig. 1

Visualization results of IG sampling of samples from the SST-5 dataset after different data augmentations 6.2. Different colors indicate different levels of importance, with darker colors indicating greater importance.

Therefore, we further attempt to use Integrated Gradients (IG) [41] to help us analyze and observe which augmented samples have a negative impact on the aggregated prediction.

Figure 3.1 shows the importance of different words in sentences obtained by different data augmentation methods. Specifically, when the input sample is "the story and characters are nowhere near gripping enough" with true label "negative". We observe that the data-augmented sample "story characters are near gripping enough." using the RWD method, which involves word deletion, removes the crucial feature "nowhere" and results in the model predicting the sample as "very positive",which is inconsistent with the original label. In the process of employing the RPI operation, an error was encountered in which a question mark("?") was erroneously inserted at the end of the sentence. As a consequence, the meaning of the sentence was transformed from a declarative to an interrogative form. Although such an alteration may have augmented the semantic richness of the sentence during training. However, the standard TTA method only employs a simple weighted average operation, which fails to reduce the weight of anomalous samples or remove them. As a result, the correct label of the original sample is compromised. Therefore, it is necessary to identify anomalous samples and allocate varying weights to the effective augmented samples.

4 Proposed methodology

To overcome the limitations of TTA, we have introduced nearest neighbor anomaly detection algorithms and network architectures with learnable weights to improve the prediction accuracy and robustness of deep learning models. Specifically, our approach involves two steps (see section 4.2-4.3): (1) We use a k-NN anomaly detection algorithm to select the most anomalous samples from the augmented samples. (2) We use a sub network architecture with learnable weights to aggregate the predictions of the remaining valuable augmented samples. In this section, we will introduce the details of our method.

4.1 Data augmentation on test time

We first consider $m \in ℕ$ different types of augmentation, and since some augmentation methods are stochastic, we repeat each augmentation method $n \in ℕ$ times to obtain n × m augmented samples. Therefore, for the original test input x, we can obtain the augmented samples $X_{N}$ , where N = n × m. And the augmented samples $X_{N}$ is represented as $X_{N} : T (X, N) \to {x_{0}, x_{1}, x_{2}, \dots, x_{N}}$ , where x₀ is the original sample and x_i is the augmented sample.

Then we can consider inputting the augmented samples $X_{N}$ into the classifier model f_base : x → p^C, where f_base represents the original unadjusted model, p^C is the predicted probability distribution and C is the number of categories. So we have the following definition:

$p_{i} = f_{base} (x_{i}), i = 0, 1, \dots, N .$ (2) In this equation, we define the predicted probability distribution matrix $ℙ (X)^{N \times C}$ of the augmented samples $X_{N}$ as follows:

$ℙ (X)^{N \times C} = {[p_{i}]}_{i = 0}^{N} = [\begin{matrix} p_{0}^{1} & p_{0}^{2} & \dots & p_{0}^{C} \\ p_{1}^{1} & p_{1}^{2} & \dots & p_{1}^{C} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ p_{N}^{1} & p_{N}^{2} & \dots & p_{N}^{C} \end{matrix}]$ (3)

where p_i is the predicted probability distribution of the i-th augmented sample x_i.

4.2 Similarity-based anomaly detection

Now, by the results of 6.1, we select the HNSW64 (64 represents the degree of nodes in the graph) algorithm based on graph retrieval method to screen out anomalous samples from the augmented sample set $X_{N}$ . Specifically, we input the augmentation matrix $ℙ (X)^{N \times C}$ into the index function $G (P)$ of the nearest neighbor algorithm to construct a distance index for the current augmented sample set $X_{N}$ , and obtain the nearest neighbors set $D (X^{N}) \to {d_{i}}_{i = 0}^{N} = {d_{0}, d_{1}, d_{2}, \dots, d_{N}}$ , here,d_i represents the distance between augmented sample x_i and original sample x₀. Therefore, d₀ is the distance value between x₀ and itself, and thus, d₀ = 0.

Then the remaining samples in the augmented sample set $X_{N}$ are selected according to the distance value d_i, and the selected samples are represented as $X_{K}$ , where K is the number of remaining samples. The remaining samples are then input into the classifier model f_base to obtain the predicted probability distribution matrix $ℙ (X)^{K \times C}$ of the remaining samples.

To aggregate the diversity of augmented samples while avoid generating excessive out-of-distribution samples, we propose a distance-based selection approach. and then takes the average of these distance values to obtain the mean of all elements in the distance matrix, denoted as $\bar{D} = \frac{1}{N} \sum_{i = 1}^{N} d_{i}$ . Next, it filters out samples with distance values less than the threshold $\bar{D}$ from $X^{N}$ , resulting in a updated probability matrix $ℙ (X)^{K \times C}$ where K represents the number of remaining samples. The above process can be expressed as follows:

$\begin{matrix} G (ℙ (X)^{N \times C}) = {[p_{i}]}_{i = 0}^{K} = [\begin{matrix} p_{0}^{1} & p_{0}^{2} & \dots & p_{0}^{C} \\ p_{1}^{1} & p_{1}^{2} & \dots & p_{1}^{C} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ p_{K}^{1} & p_{K}^{2} & \dots & p_{K}^{C} \end{matrix}] \end{matrix}$ (4)

Algorithm 1 Distance-based samples fillter

Require: Augmented sample set $X_{N}$ , threshold $\bar{D}$

1: x ← 1

2: for i ∈ N do

3: $d_{i} \leftarrow G (ℙ (X)^{N \times C})$

4: $\bar{D} \leftarrow \frac{1}{N} \sum_{i = 1}^{N} d_{i}$

5: if $d_{i} < \bar{D}$ then

6: $X_{K} \leftarrow X_{K} \cup X_{i}$

7: end if

8: end for

9: return Remaining sample set $X_{K}$

4.3 Weight adaptation

After obtaining the predicted probability distribution matrix $ℙ (X)^{K \times C}$ of the remaining samples, we need to update the weight ω_i of each augmented sample x_i in the augmented sample set $X_{K}$ according to the predicted probability distribution matrix $ℙ (X)^{K \times C}$ . The weight ω_i of each augmented sample x_i was initially set to $\frac{1}{N}$ , where N is the number of augmented samples. The weight ω_i of each augmented sample x_i is updated as $ω_{i} = \frac{1}{C} \sum_{j = 1}^{C} p_{i}^{j}$ . and,the parameters of $ℙ (X)^{K \times C}$ are updated according to the following formula:

$W (X)^{K \times C} = {[ω_{i}]}_{i = 0}^{K} = [\begin{matrix} ω_{0}^{1} & ω_{0}^{2} & \dots & ω_{0}^{C} \\ ω_{1}^{1} & ω_{1}^{2} & \dots & ω_{1}^{C} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ ω_{K}^{1} & ω_{K}^{2} & \dots & ω_{K}^{C} \end{matrix}]$ (5)

In some cases, only using the normalized values obtained from sigmoid function to assign weights may lead to over-reliance on certain augmented samples, resulting in an excessive dependence on the features of these samples while aggregating prediction results. To address this issue, we propose a new method for mixing weights. Specifically, multiply $W (X)^{K \times C}$ by a parameter β that follows the beta distribution, and multiply the predicted results of the original samples prediction p₀ by (1 - β). Then add them together to obtain the mixed-weight feature matrix $\tilde{W} (X)^{K \times C}$ . At times, the predicted results of the augmented samples may be more representative than the original samples. Therefore,our method can prevent the model from relying too much on local features, thereby improving the model’s predictive ability and robustness. The process can be expressed as follows:

${\tilde{ω}}_{i} = β \cdot ω_{i} + (1 - β) \cdot p_{0}^{i}$ (6)

The next step is to take the average along dimension C of $\tilde{I} (X)^{K \times C}$ , which gives us the aggregated prediction results p_β of the augmented samples:

$p_{β} = \frac{1}{K} \sum_{i = 1}^{K} {\tilde{ω}}_{i}$ (7)

Then we take the average of the maximum values of each row of $W (X)^{K \times C}$ to obtain $W (X)_{\max}^{K}$ , and then take the average of $W (X)_{\max}^{K}$ to obtain α:

$W (X)_{\max}^{K} \overset{\frac{1}{K} \sum_{i = 1}^{K} W (X)_{\max}^{K}}{\to} α$ (8)

Finally, we use α to control the weight of the prediction results based on the original samples and the aggregated prediction results based on the augmented samples. The final prediction results are obtained by mixing the two prediction results according to the following formula:

$p_{tta} = (1 - α) p_{0} + α p_{β}$ (9)

Considering that the aggregate prediction result p_tta and the prediction result p₀ based on the original sample are similar but slightly different outputs of the same model, it is similar to the situation where the same sample corresponds to two different prediction results given by two different models. Therefore, on the basis of minimizing the cross-entropy loss 11 between the model’s predicted result and the ground truth, we add calculations of the similarity between the original sample and the augmented sample to construct the objective function.Consequently, it contains two optimization objectives:

$L_{KL}^{i} = \frac{1}{2} (L_{KL} (p_{tta} | | p_{0}) + L_{KL} (p_{0} | | p_{tta})) .$ (10)

The first objective is to minimize the bidirectional Kullback-Leibler (KL) divergence $L_{KL}^{i}$ between the predicted probability distribution based on the original samples and predicted probability distribution based on the aggregated samples.

The second objective is to minimize the cross-entropy loss $L_{nce}^{i}$ between the predicted results based on the original samples and the ground truth.

$L_{nce}^{i} = - \sum_{j = 1}^{C} y_{j}^{i} \cdot log p_{tta}^{j}$ (11)

The final loss function of our method is as below:

$L^{i} = L_{nce}^{i} + L_{KL}^{i}$ (12)

Looking at formula 12 in a different perspective, by adding a regularization term to the cross-entropy loss function [56] which is equivalent to adding a penalty term to the cross-entropy loss function. This enhances the robustness and adaptability of the model to augmented samples, while also reducing sensitivity to certain perturbations and the likelihood of overfitting. This regularization method can better normalize the predicted results of the model, reducing the inconsistency between predicted results p_tta based on the aggregated samples and predicted results p₀ based on the original samples. It achieves a transition from "model fusion" to "weight fusion," which is more conducive to the stability and reliability of the final prediction results and is also more consistent with the core idea of soft voting.

Therefore, the pseudocode of our method is shown in Algorithm 2.

Algorithm 2 Proposed Approach Defy

Require: Dataset $D$ , Model M, Number of Augmentations N, Augmentation Method T, optimizer $O$ .

1: for instance $X \in D$ do

2: Get the augmented samples $X^{N}$ by $T (X)$ .

3: Get the predictions $ℙ (X)^{N \times C}$ by f_base.

4: Feed $ℙ (X)^{N \times C}$ to $G (P)$ to obtain $D (X^{N}) = {d}_{i}^{N}$ .

5: Filter out and get the remaining augmented samples $ℙ (X)^{K \times C}$ by 1.

6: Update remaining samples weight $W (X)^{K}$ by 5.

7: Apply Beta distribution β to $ω_{i} \in W (X)^{K}$ by 6.

8: Get the aggregated prediction p_tta by 9.

9: if $X \in D_{valid}$ then

10: Compute loss $L^{i}$ by 12.

11: Update the model weights by $O$ .

12: end if

13: end for

14: return Aggregate prediction p_tta.

5 Experimental design

5.1 Baselines and models

All experiments were conducted on the pre-trained DistilBERT-base-uncased [46], BERT-base-uncase [7], and RoBERTa-base [28] network from the Huggingface Transformers library, as used by [61 , 26], and fine-tuned on the MRPC, RTE, SST-2 [51], SST-5 [49], SUBJ [6], Trec-Fine, Trec-Coarse [26], AG-News [61]. During the test time adaptation process, the Baseline refers to a non-adaptive model. These datasets were sourced from either the Huggingface Datasets or their official websites. Apart from the baseline, we compare with the following typical baselines to verify the effectiveness of Defy: (1) Original (Raw Prediction): Directly use the prediction of the original sample without TTA; (2) Standard TTA: Average logit across all augmented samples [25], and this is the standard practice in TTA; (3) Max (Maximum Predicted Probability): Maximum logit across all augmented samples [11].

Note: Due to the limited research on TTA aggregation methods in the field of NLP, we also constructed some distance measure methods to compare with Defy. The specific ablation experiment results are shown in Section 6.4.

5.2 Implementation details

All our experiments follow the hyperparameter settings provided by the dataset official website. We use the model training code provided by Hugging face Transformers [55]. During test time, we adopt Adam [23] with initial learning rate 2e^-5, batch size 32 and max sequence length 128. All experiments were conducted on a single NVIDIA RTX A6000 GPU, and we performed the experiments with 3 different random seeds.

6 Experimental results

6.1 Overall results

Here we describe and discuss the results summarized in Table 1 and Fig. 6.1. The results obtained by Defy are compared with the performance of the standard TTA, Max TTA, and baseline models (without any type of TTA method). Table 1 displays a detailed comparison between Defy and the standard TTA method in terms of predicted results on eight benchmark datasets and three benchmark models. In the total of 24 experiments, our method showed a leading advantage in almost all of them.

Table 1
Overall results. Here the data augmentation method used by TTA is RWSR + RWI + RWS + RWD. BERT, DistilBERT, and RoBERTa are corresponding to BERT-base-uncased, DistilBERT-base-uncased, and RoBERTa-base respectively. Bold characters represent the best results

Model Datasets Baseline Standard TTA Max TTA Ours (HNSW64-L2) Ours (Annoy-Angular) Ours (Flat-L2) Ours (HNSW-Flat) Ours (HNSWSQ)

BERT AG-News 94.20 -0.14±0.03 -0.25±0.03 0.05±0.05 0.06±0.01 0.05±0.03 0.05±0.05 0.05±0.04

SST-2 91.43 -0.16±0.22 -1.18±0.43 0.11±0.13 0.14±0.07 0.11±0.13 0.11±0.13 0.14±0.15

SST-5 53.08 -0.02±0.15 -1.49±0.66 0.53±0.07 0.49±0.04 0.53±0.07 0.53±0.07 0.52±0.06

MRPC 81.57 -0.01±0.15 -0.57±0.49 0.33±0.13 0.64±0.05 0.33±0.13 0.33±0.13 0.29±0.09

RTE 63.18 -0.63±1.04 0.27±1.08 0.18±0.22 -0.09±0.18 0.18±0.21 0.18±0.21 0.27±0.18

SUBJ 97.00 -0.35±0.12 -0.41±0.13 0.05±0.01 0.01±0.03 0.05±0.01 0.05±0.21 0.05±0.21

Trec-Coarse 96.60 -0.30±0.20 -0.80±0.75 0.20±0.01 0.11±0.12 0.20±0.08 0.20±0.03 0.20±0.03

Trec-Fine 91.80 -0.65±0.19 -1.90±3.71 0.30±0.10 0.45±0.10 0.30±0.10 0.30±0.10 0.35±0.10

DistilBERT AG-News 93.97 -0.07±0.08 -0.15±0.09 0.13±0.02 0.10±0.03 0.13±0.02 0.13±0.02 0.13±0.02

SST-2 89.79 0.12±0.16 -1.17±0.25 0.43±0.11 0.62±0.12 0.43±0.11 0.43±0.11 0.44±0.13

SST-5 52.08 0.19±0.08 -0.52±0.49 0.18±0.08 0.24±0.38 0.18±0.08 0.18±0.08 0.19±0.10

MRPC 79.83 -0.43±0.18 -1.36±0.88 0.36±0.16 0.38±0.14 0.36±0.16 0.36±0.16 0.36±0.16

RTE 59.57 -1.62±0.47 -0.45±0.85 0.18±0.47 -0.45±0.18 0.18±0.47 0.18±0.47 0.09±0.45

SUBJ 96.90 -0.50±0.09 -0.74±0.28 0.01±0.02 0.06±0.02 0.01±0.02 0.01±0.01 0.01±0.01

Trec-Coarse 96.40 -0.05±0.10 -1.50±1.27 0.15±0.10 0.05±0.10 0.15±0.10 0.15±0.10 0.15±0.10

Trec-Fine 91.60 -0.55±0.44 -1.05±3.71 0.05±0.10 0.05±0.10 0.05±0.10 0.05±0.10 0.05±0.10

RoBERTa AG-News 94.36 -0.33±0.04 -0.31±0.13 0.12±0.06 0.10±0.03 0.12±0.06 0.12±0.06 0.12±0.06

SST-2 94.29 -0.07±0.21 -1.13±0.92 0.27±0.12 0.41±0.08 0.27±0.12 0.27±0.12 0.27±0.12

SST-5 56.11 0.14±0.08 -1.71±1.13 0.46±0.15 0.15±0.21 0.46±0.15 0.46±0.15 0.46±0.15

MRPC 86.43 -0.96±0.15 -1.43±0.19 -0.01±0.21 0.00±0.14 -0.01±0.21 -0.01±0.21 -0.02±0.19

RTE 75.09 -6.05±0.90 -8.39±1.57 0.54±0.21 0.45±0.35 0.54±0.21 0.54±0.21 0.54±0.21

SUBJ 97.00 0.00±0.04 -0.21±0.19 0.09±0.01 0.09±0.12 0.09±0.01 0.09±0.01 0.09±0.01

Trec-Coarse 96.20 -0.65±0.10 -2.20±0.75 0.20±0.16 0.15±0.10 0.20±0.16 0.20±0.16 0.20±0.16

Trec-Fine 92.80 -0.35±0.10 -1.50±3.01 0.25±0.10 0.30±0.12 0.25±0.10 0.25±0.10 0.25±0.10

Fig. 2

Figure 6.1 demonstrates that standard TTA method performance across different datasets. It can be seen that although there is a slight performance improvement on some individual datasets, the standard TTA method’s performance is unstable on most datasets and cannot significantly improve the performance. In specific datasets, it may even lead to a decrease in performance. This indicates that using a simple approach to average the model’s prediction results is not reasonable for text classification tasks at least. This will result in uncertain out-of-distribution augmented samples being averaged into the final results. However, [14]’s study also showed that although the Max method could improve the model’s ability to recognize out-of-domain distribution augmented samples, introducing this method into TTA to improve text classification accuracy was not very effective. This is because selecting the maximum predicted probability value in the augmented samples is the most volatile of all TTA methods, which erroneously calibrates the predictions on almost all datasets, greatly affecting the predictions.

Fig. 3

The performance of different TTA methods on different datasets, using RWSR + RWI + RWS + RWD for data augmentation with an augmentation magnitude of 32.Here,the standard deviation represents the model of BERT, RoBERTa and DistilBERT.

In this study, we present a novel TTA method for effectively aggregating multiple non-deterministic augmented samples, which ultimately leads to improved model performance. Specifically, reducing the impact of uncertain samples during the aggregation of augmented samples at test-time, thereby enhancing the robustness of the model’s prediction results. Some additional observations can be observed from Fig. 6, indicating that the effectiveness of TTA methods varies across datasets and is dependent on the model’s abilities. Interestingly, the study finds that for models with strong abilities, the effectiveness of the TTA method may not be significant, resulting in limited net gains. This is evidenced by the narrowing of benefits observed in BERT-base-uncased and RoBERTa-base models on the SUBJ dataset, as outlined in Subsection 6.3.

6.2 Effect of augmentation strategies and numbers

Fig. 4

The impact of different data augmentation methods on TTA net gains on BERT, DistilBERT, and RoBERTa. The standard deviation in the figure represents different augmentation magnitudes.

There are two factors that affect TTA net gains: data augmentation methods and the aggregation method of augmented samples. While we discussed the design of aggregation methods in detail earlier, and [53] has found that Neural networks have a high error tolerance for the low-frequency components of text, as these components usually correspond to the overall semantics and features of the text. Therefore, processing the low-frequency components will not have a significant impact on the model’s predictive results. Conversely, neural networks have a relatively low tolerance for errors in the high-frequency components of text, as these components typically correspond to the details and local features of the text, and even slight changes may cause the model to have larger prediction biases.

Due to the above reasons, we focus on the most representative and widely used data augmentation methods, including the following:

Random Punctuation Insertion(RPI) [21]: inserting specific punctuation marks at a randomly selected position within the range of the text sequence length with a probability of 0.3

Random Word Synonym Replacement(RWSR) [34]: Randomly select N words in the sentence that are not stop words, and then randomly choose possible synonyms for these words to replace them with a probability of 0.1.

Random Word Insertion(RWI) [57]: Randomly selecting a non-stop word in the sentence and inserting a synonym of that word at a random position in the sentenc with a probability of 0.1.

Random Word Swap(RWS) [30]: Randomly swapping the positions of two words in the sentence with a probability of 0.1.

Random Word Deletion(RWD) [3]: Randomly deleting words from the sentence with a probability of 0.1.

Although more complex data augmentation methods have been proposed, for example methods based on reinforcement learning [44], they may bring more uncertainty and additional computational burden.

Fig. 5

The impact of sample augmentation magnitude on TTA. Where the data augmentation method is RWSR + RWI + RWS + RWD, by averaging all the data sets and different models.

Table 2 shows the six data augmentation policies used in our experiments. Specifically, random punctuation insertion was denoted as RWSR, while random word synonym replacement was represented by RWSR. Additionally, random word insertion was denoted as RWI, random word swap was represented by RWS, and random word deletion was denoted as RWD. Finally, the combination of the four fundamental data augmentation methods was represented by RWSR + RWI + RWS + RWD. There is evidence that combining multiple data augmentation methods can improve the robustness of the augmented model at test time [29]. It is worth noting that the combination of various data augmentation methods, such as RWSR + RWI + RWS + RWD, is similar to what is discussed in [29]. And RPI modifies the semantics of the text more slightly and exceeds RWSR + RWI + RWS + RWD during training as mention in [21], so we choose RPI as another basic data augmentation method.

Table 2

Different data augmentation policies

Aug. 1: RPI	Aug. 2: RWSR
Aug. 3: RWI	Aug. 4: RWS
Aug. 5: RWD	Aug. 6: RWSR + RWI + RWS + RWD

We start to analyze the results of different data augmentation methods by the results in Fig. 6.2, We observe that RWSR + RWI + RWS + RWD is the best augmentation strategy in terms of performance on Bert in our method, regardless of which dataset is used, but considering that the performance of different base models is not all like this. For each base model, we have found different performance best augmentation strategies. Nonetheless, the results of RWSR + RWI + RWS + RWD are still relatively close to the best performance results in the first two places on the three base models. We also note that RWI has achieved unexpectedly good results on the RoBERTa model, while RWD has performed poorly on all three base models. This indicates that different models have different dependencies on different data augmentation methods, which is also one of the factors we need to consider when designing data augmentation methods.

For the standard TTA method, there is no significant performance improvement on various data augmentation methods on the three models, and RWSR causes a large variance. Although most data augmentation methods have damaged the performance of the model on Max TTA, RPI augmented method has shown a significant lower variance on the three models.

Next, we start to analyze the results in Fig. 5 to explore the impact of different data augmentation magnitudes on TTA. The main observation result obtained in this context is that our method Defy is the only method that shows a linear increase in model performance with the increase of the number of augmentations. In contrast, the standard TTA method further increases the variance of the model performance with the increase of the number of augmentations, while the Max TTA method will further reduce the model performance.

6.3 Effectiveness of pre-trained models

Fig. 6

The impact of different data augmentation methods on the accuracy of different pre-training models. Here, the average accuracy across all datasets is taken, using RWSR + RWI + RWS + RWD for data augmentation with an augmentation magnitude of 32. The standard deviation represents different datasets.

In the previous section, we observed that different models have different dependencies on data augmentation methods, so in this section, we further explore whether different models will have an impact on different TTA methods. To verify this, we compared the performance of the representative models, BERT-base and RoBERTa-base, as well as a lightweight model, DistilBERT-base, which has only 60% of the parameters of BERT-base.

We start to analyze the results of different models by the results in Fig. 6. The first observation is that our method Defy is the only method that can consistently improve the performance of different models. Contrarily, the standard TTA method and the Max method damage the performance of the model to varying degrees, especially on DistilBERT, the standard TTA method even causes significant fluctuations in the performance of the model. Although RoBERTa is a more robust model, it has not been able to reduce the damage of the Max method to the model performance.

Now,we focus on the best results obtained on each specific dataset, as shown in Table 1. Despite our method, Defy, achieving the best results on all datasets, it exhibits significant differences in performance on different models for certain datasets. For instance, we observe that the RTE dataset demonstrates lower benefits on BERT and DistilBERT, but performs significantly better on RoBERTa. When a baseline model has already achieved high performance levels, the improvement in model performance through the TTA method can be reduced, as in the case of MRPC, where RoBERTa’s baseline performance has already reached 86.43%, resulting in almost no benefit from TTA. However, for BERT and DistilBERT, with baseline performances of 81.57% and 79.83%, respectively, our TTA (Ours) method shows an average benefit of almost 0.4%.

6.4 Ablation studies for anomaly detection

Fig. 7

The impact of different k-NN algorithms on the performance of our method with anomaly augmented samples. Here, we compared different models and averaged the results across all datasets, using RWSR + RWI + RWS + RWD for data augmentation with an augmentation magnitude of 32.

In the previous subsection, it was observed that Defy (Ours) was the best TTA method when considering different benchmark models, despite variations in performance across specific datasets. Given this observation, this section aims to further explore the effectiveness of the anomaly detection methods employed in our approach. Simple methods commonly used in NLP tasks, such as Euclidean distance, often fail to capture the full potential of tensors and are sensitive to feature distribution [42]. Additionally, these methods are not well-suited to zero or sparse vectors and are prone to misclassifying anomalous samples. To address these issues, this subsection focuses on experiments conducted with several advanced k-NN methods, including Annoy-Angular, Flat-L2, HNSW64-L2, HNSWFlat, and HNSWSQ [19]. Through these experiments, we aim to find an anomaly detection method that is universally applicable across different datasets and models.

Annoy [4] use random projection and tree structure with hyperplane partitioning based on the Angular distance metric.

Flat-L2 [36] use linear scanning on a plane to search for nearest neighbors based on the Euclidean distance metric.

HNSW64-L2 [32] use a graph-based approach with Hierarchical navigable small world(HNSW) structure, 64 represents the number of node for each vector, and L2 represents the Euclidean distance metric.

HNSWFlat [33] only use a graph-based approach with HNSW.

HNSWSQ [18] use a graph-based approach with HNSW combined with scalar quantization.

The results are shown in Fig. 7. In the experiments, Defy conducted different k-NN algorithms across three models and employed data augmentation through RWSR + RWI + RWS + RWD with 32 repetitions. Notably, It was observed that the performances of HNSW-based methods (HNSW564-L2, HNSWFlat, and HNSWSQ) were almost identical to Flat. Specifically, the median performance of HNSWSQ was relatively high on the Bert model. Although Annoy-Angular achieved slightly better overall performance than other methods on Bert, it was not sufficient to surpass the performance of HNSW-based methods on many occasions, and its performance was not always stable on specific datasets, such as RTE. Additionally, the performance of Flat-Cos was consistently underperformed in comparison to other methods across all models. This may be attributed to the sensitivity of cosine similarity measurement to the angles between data points. In cases where the probability prediction distributions p_i of the model are close to each other, their cosine similarity may be high, resulting in inaccurate distance estimations.

7 Investigating label corruption and correction

Fig. 8

The percentage of label changes for different datasets using different TTA methods. Here,corrected label means that the model predicted inaccurately but aggregated correctly using TTA, and corrupted label means that the model predicted accurately but aggregated inaccurately through TTA.

Figure 8 portrays the standard TTA method, Max TTA method, and our method, concerning the proportions of label corruption and correction across eight datasets. Label corruption denotes the number of instances where the model predicted accurately but aggregated inaccurately through TTA, while label correction represents the number of cases where the model predicted inaccurately but aggregated correctly using TTA. We notice that the standard TTA method consistently has a higher proportion of label corruption than correction across most datasets. The Max TTA method exhibits an abnormally high proportion of label corruption, reaching 0.07 on the Trec-Fine dataset. Conversely, our proposed approach consistently has a higher proportion of label correction than corruption. To identify the data augmentation samples leading to incorrect predicted labels, we conduct a thorough investigation of the data augmentation samples responsible for label corruption.

8 Conclusion

In this paper, we presents a novel TTA method, Defy, which overcomes the limitations of the standard TTA method in the NLP field, by leveraging k-NN-based anomaly detection algorithm and weight adaptation mechanism to effectively adapt to the characteristics of different datasets and models. Unlike other robust methods, Defy does not interfere with the training process of the backbone network and can be used in conjunction with other robust methods to further improve model performance. Furthermore, the approach offers excellent plug-and-play capabilities, enabling easy integration into existing models. Moving forward, attention should be directed toward addressing the following two challenges:

How to selectively apply TTA to those samples which need to be corrected, rather than the entire test set, in order to minimize time and computational costs.

Design more reasonable and effective data augmentation methods, especially for test time.

Footnotes

Acknowledgments

This work was funded by the Ten Thousand Talent Plans for Young Top-notch Talents of Yunnan Province (Project No. YNWR-QNBJ-2018-351). The authors declare that there are no conflicts of interest associated with this funding.

Data availability statement

The benchmark datasets and models utilized in this study were previously introduced in Section 5.1. These datasets are openly accessible and can be retrieved through the links provided in that particular section.

Statements and declarations

No potential conflict of interest was reported by the authors.

References

Abdou

Mostafa

, Ravishankar

Vinit

, Barrett

Maria

, Belinkov

Yonatan

, Elliott

Desmond

and Søgaard

Anders

, The sensitivity of language models and humans to winograd schema perturbations, arXiv preprint arXiv:2005.01348 2020.

Amiri

Mina

, Brooks

Rupert

, Behboodi

Bahareh

and Rivaz

Hassan

, Two-stage ultrasound image segmentation using u-net and test time augmentation, International Journal of Computer Assisted Radiology and Surgery 15 (2020), 981–988.

Bayer

Markus

, Kaufhold

Marc-André

and Reuter

Christian

, A survey on data augmentation for text classification, ACM Computing Surveys 55(7) (2022), 1–39.

Bernhardsson

Erik

, Annoy: Approximate Nearest Neighbors in C++/Python (2018). Python package version 1.13.0.

Caruana

Rich

, Niculescu-Mizil

Alexandru

, Crew

Geoff

and Ksikes

Alex

, Ensemble selection from libraries of models, In Proceedings of the twenty-first international conference on Machine learning, page 18, 2004.

Conneau

Alexis

and Kiela

Douwe

, Senteval: An evaluation toolkit for universal sentence representations, arXiv preprint arXiv:1803.05449, 2018.

Devlin

Jacob

, Chang

Ming-Wei

, Lee

Kenton

and Toutanova

Kristina

, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018.

Dong

Yihe

, Hopkins

Samuel

and Li

Jerry

, Quantum entropy scoring for fast robust mean estimation and improved outlier detection, In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.

Fern

Xiaoli Z.

and Lin

Wei

, Cluster ensemble selection, Statistical Analysis and Data Mining: The ASA Data Science Journal 1(3) (2008), 128–141.

10.

Frosst

Nicholas

, Papernot

Nicolas

and Hinton

Geoffrey

, Analyzing and improving representations with the soft nearest neighbor loss, In International conference on machine learning, pages 2012–2020 PMLR, 2019.

11.

Guo

Chuan

, Pleiss

Geoff

, Sun

and Weinberger

Kilian Q.

, On calibration of modern neural networks, In International conference on machine learning, pages 1321–1330, PMLR 2017.

12.

Guo

Haixuan

, Yuan

Shuhan

and Wu

Xintao

, Logbert: Log anomaly detection via bert, In 2021 international joint conference on neural networks (IJCNN), pages 1–8. IEEE, 2021.

13.

Pengcheng

, Gao

Jianfeng

and Chen

Weizhu

, DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing, In The Eleventh International Conference on Learning Representations, 2023.

14.

Hendrycks

Dan

and Gimpel

Kevin

, A baseline for detecting misclassified and out-of-distribution examples in neural networks, arXiv preprint arXiv:1610.02136, 2016.

15.

Hendrycks

Dan

, Liu

Xiaoyuan

, Wallace

Eric

, Dziedzic

Adam

, Krishnan

Rishabh

and Song

Dawn

, Pretrained transformers improve out-of-distribution robustness, arXiv preprint arXiv:2004.06100, 2020.

16.

Hendrycks

Dan

, Mazeika

Mantas

, Kadavath

Saurav

and Song

Dawn

, Using self-supervised learning can improve model robustness and uncertainty, Advances in Neural Information Processing Systems 32 (2019).

17.

Huang

Yulong

, Zhang

Yonggang

, Zhao

Yuxin

, Shi

Peng

and Chambers

Jonathon A.

, A novel outlier-robust kalman filtering framework based on statistical similarity measure, IEEE Transactions on Automatic Control 66(6) (2021), 2677–2692.

18.

Jégou

Herve

, Douze

Matthijs

and Schmid

Cordelia

, Product quantization for nearest neighbor search, IEEE Transactions on Pattern Analysis and Machine Intelligence 33(1) (2011), 117–128.

19.

Johnson

Jeff

, Douze

Matthijs

and Jégou

Hervé

, Billion-scale similarity search with gpus, IEEE Transactions on Big Data 7(3) (2019), 535–547.

20.

Kandel

Ibrahem

and Castelli

Mauro

, Improving convolutional neural networks performance for image classification using test time augmentation: a case study using mura dataset, Health Information Science and Systems 9 (2021), 1–22.

21.

Karimi

Akbar

, Rossi

Leonardo

and Prati

Andrea

, Aeda: An easier data augmentation technique for text classification, arXiv preprint arXiv:2108.13230, 2021.

22.

Kim

Ildoo

, Kim

Younghoon

and Kim

Sungwoong

, Learning loss for test-time augmentation, Advances in Neural Information Processing Systems 33 (2020), 4163–4174.

23.

Kingma

Diederik P.

and Ba

Jimmy

, Adam: A method for stochastic optimization, In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings, 2015.

24.

Krizhevsky

Alex

, Sutskever

Ilya

and Hinton

Geoffrey E.

, Imagenet classification with deep convolutional neural networks, In F. Pereira, C.J. Burges, L. Bottou and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.

25.

Krizhevsky

Alex

, Sutskever

Ilya

and Hinton

Geoffrey E.

, Imagenet classification with deep convolutional neural networks, Communications of the ACM 60(6) (2017), 84–90.

26.

Xin

and Roth

Dan

, Learning question classifiers, In COLING 2002: The 19th International Conference on Computational Linguistics, 2002.

27.

Liu

Xiaodong

, Cheng

Hao

, He

Pengcheng

, Chen

Weizhu

, Wang

, Poon

Hoifung

and Gao

Jianfeng

, Adversarial training for large neural language models, ArXiv, abs/2004.08994, 2020.

28.

Liu

Yinhan

, Ott

Myle

, Goyal

Naman

, Du

Jingfei

, Joshi

Mandar

, Chen

Danqi

, Levy

Omer

, Lewis

Mike

, Zettlemoyer

Luke

and Stoyanov

Veselin

, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692, 2019.

29.

Helen

, Shanmugam

Divya

, Suresh

Harini

and Guttag

John

, Improved text classification via test-time augmentation, arXiv preprint arXiv:2206.13607, 2022.

30.

Luu

Son

, Nguyen

Kiet

and Nguyen

Ngan

, Empirical study of text augmentation on social media text in Vietnamese, In Proceedings of the 34th Pacific Asia Conference on Language, Information and Computation, pages 462–470, Hanoi, Vietnam, October 2020. Association for Computational Linguistics.

31.

Lyzhov

Alexander

, Molchanova

Yuliya

, Ashukha

Arsenii

, Molchanov

Dmitry

and Vetrov

Dmitry

, Greedy policy search: A simple baseline for learnable test-time augmentation, In Conference on Uncertainty in Artificial Intelligence, pages 1308–;1317 PMLR. 2020.

32.

Malkov

Yu A.

and Yashunin

D.A.

, Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs, IEEE Transactions on Pattern Analysis and Machine Intelligence 42(4) (2020), 824–836.

33.

Malkov

Yury

, Ponomarenko

Alexander

, Logvinov

Andrey

and Krylov

Vladimir

, Approximate nearest neighbor algorithm based on navigable small world graphs, Information Systems 45 (2014), 61–68.

34.

Miller

George A.

, Wordnet: a lexical database for english, Communications of the ACM 38(11) (1995), 39–41.

35.

Moshkov

Nikita

, Mathe

Botond

, Kertesz-Farkas

Attila

, Hollandi

Reka

and Horvath

Peter

, Test-time augmentation for deep learning-based cell segmentation on microscopy images, Scientific Reports 10(1) (2020), 1–7.

36.

Mulzer

Wolfgang

, Nguyen

Huy L.

, Seiferth

Paul

and Stein

Yannik

, Approximate k-flat nearest neighbor search, CoRR, abs/1411.1519, 2014.

37.

OpenAI. Chatgpt: Optimizing language models for dialogue, Open AI, blog, 2022.

38.

Pang

Guansong

, Shen

Chunhua

, Cao

Longbing

and Hengel

Anton Van Den

, Deep learning for anomaly detection: A review, ACM Computing Surveys (CSUR) 54(2) (2021), 1–38.

39.

Pang

Tianyu

, Xu

Kun

and Zhu

Jun

, Mixup inference: Better exploiting mixup to defend adversarial attacks, arXiv preprint arXiv:1909.11515, 2019.

40.

Prakash

Aaditya

, Moran

Nick

, Garber

Solomon

, DiLillo

Antonella

and Storer

James

, Deflecting adversarial attacks with pixel deflection, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8571–8580. 2018.

41.

Zhongang

, Khorram

Saeed

and Li

Fuxin

, Visualizing deep networks by optimizing with integrated gradients, In CVPR Workshops, volume 2, pages 1–4, 2019.

42.

, Qian

and Tang

, Large-scale distance metric learning with uncertainty, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8542–8550, 2018.

43.

Reiss

Tal

, Cohen

Niv

, Bergman

Liron

and Hoshen

Yedid

, Panda: Adapting pretrained features for anomaly detection and segmentation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2806–2814. 2021.

44.

Ren

Shuhuai

, Zhang

Jinchao

, Li

Lei

, Sun

and Zhou

Jie

, Text autoaugment: Learning compositional augmentation policy for text classification, arXiv preprint arXiv:2109.00523, 2021.

45.

Sanh

Victor

, Debut

Lysandre

, Chaumond

Julien

and Wolf

Thomas

, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108, 2019.

46.

Sanh

Victor

, Debut

Lysandre

, Chaumond

Julien

and Wolf

Thomas

, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108, 2019.

47.

Schölkopf

Bernhard

, Smola

Alexander J.

, Williamson

Robert C.

and Schürmann

Klaus-Robert

, Novelty detection, Neural Computation 13(7) (2001), 1435–1470.

48.

Shanmugam

Divya

, Blalock

Davis W.

, Balakrishnan

Guha

and Guttag

John V.

, Better aggregation in test-time augmentation, 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages –1203, 2020.

49.

Socher

Richard

, Perelygin

Alex

, Wu

Jean

, Chuang

Jason

, Manning

Christopher D.

, Ng

Andrew Y.

and Potts

Christopher

, Recursive deep models for semantic compositionality over a sentiment treebank, In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013.

50.

Son

Jongwook

and Kang

Seokho

, Efficient improvement of classification accuracy via selective test-time augmentation, Information Sciences 642 (2023), 119148.

51.

Wang

Alex

, Singh

Amanpreet

, Michael

Julian

, Hill

Felix

, Levy

Omer

and Bowman

Samuel R.

, Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint arXiv:1804.07461, 2018.

52.

Wang

Guotai

, Li

Wenqi

, Ourselin

Sébastien

and Vercauteren

Tom

, Automatic brain tumor segmentation using convolutional neural networks with test-time augmentation, In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part II 4, pages 61–72. Springer, 2019.

53.

Wang

Haohan

, Wu

Xindi

, Huang

Zeyi

and Xing

Eric P.

, High frequency component helps explain the generalization of convolutional neural networks, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8684–8694, 2020.

54.

Wei

Jason W.

and Zou

Kai

, EDA: easy data augmentation techniques for boosting performance on text classification tasks, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLPIJCNLP 2019, Hong Kong, China, November 3–7, 2019, pages 6381–6387. Association for Computational Linguistics, 2019.

55.

Wolf

Thomas

, Debut

Lysandre

, Chaumond

Victor

, Delangue

Clement

, Moi

Anthony

, Cistac

Pierric

, Sanh

Victor

, Chaumond

Julien

, Delangue

Clement

, Moi

Anthony

, et al., Huggingface’s transformers: State-of-the-art natural language processing, arXiv preprint arXiv:1910.03771, 2020.

56.

Lijun

, Li

Juntao

, Wang

Yue

, Meng

, Qin

Tao

, Chen

Wei

, Zhang

Min

, Liu

Tie-Yan

, et al., R-drop: Regularized dropout for neural networks, Advances in Neural Information Processing Systems 34 (2021), 10890–10905.

57.

Zhuofeng

, Wang

Sinong

, Gu

Jiatao

, Khabsa

Madian

, Sun

Fei

and Ma

Hao

, Clear: Contrastive learning for sentence representation, ArXiv, abs/2012.15466, 2020.

58.

Zhang

Hongyi

, Cisse

Moustapha

, Dauphin

Yann N.

and Lopez-Paz

David

, mixup: Beyond empirical risk minimization, arXiv preprint arXiv:1710.09412, 2017.

59.

Zhang

Hongyi

, Cissé

Moustapha

, Dauphin

Yann N.

and Lopez-Paz

David

, mixup: Beyond empirical risk minimization, In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, Conference Track Proceedings, OpenReview.net, 2018.

60.

Zhang

Marvin

, Levine

Sergey

and Finn

Chelsea

, Memo: Test time robustness via adaptation and augmentation, In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 38629–38642. Curran Associates, Inc., 2022.

61.

Zhang

Xiang

, Zhao

Junbo

and LeCun

Yann

, Character-level convolutional networks for text classification, Advances in Neural Information Processing Systems 28 (2015).

Model	Datasets	Baseline	Standard TTA	Max TTA	Ours (HNSW64-L2)	Ours (Annoy-Angular)	Ours (Flat-L2)	Ours (HNSW-Flat)	Ours (HNSWSQ)
BERT	AG-News	94.20	-0.14±0.03	-0.25±0.03	0.05±0.05	0.06±0.01	0.05±0.03	0.05±0.05	0.05±0.04
	SST-2	91.43	-0.16±0.22	-1.18±0.43	0.11±0.13	0.14±0.07	0.11±0.13	0.11±0.13	0.14±0.15
	SST-5	53.08	-0.02±0.15	-1.49±0.66	0.53±0.07	0.49±0.04	0.53±0.07	0.53±0.07	0.52±0.06
	MRPC	81.57	-0.01±0.15	-0.57±0.49	0.33±0.13	0.64±0.05	0.33±0.13	0.33±0.13	0.29±0.09
	RTE	63.18	-0.63±1.04	0.27±1.08	0.18±0.22	-0.09±0.18	0.18±0.21	0.18±0.21	0.27±0.18
	SUBJ	97.00	-0.35±0.12	-0.41±0.13	0.05±0.01	0.01±0.03	0.05±0.01	0.05±0.21	0.05±0.21
	Trec-Coarse	96.60	-0.30±0.20	-0.80±0.75	0.20±0.01	0.11±0.12	0.20±0.08	0.20±0.03	0.20±0.03
	Trec-Fine	91.80	-0.65±0.19	-1.90±3.71	0.30±0.10	0.45±0.10	0.30±0.10	0.30±0.10	0.35±0.10
DistilBERT	AG-News	93.97	-0.07±0.08	-0.15±0.09	0.13±0.02	0.10±0.03	0.13±0.02	0.13±0.02	0.13±0.02
	SST-2	89.79	0.12±0.16	-1.17±0.25	0.43±0.11	0.62±0.12	0.43±0.11	0.43±0.11	0.44±0.13
	SST-5	52.08	0.19±0.08	-0.52±0.49	0.18±0.08	0.24±0.38	0.18±0.08	0.18±0.08	0.19±0.10
	MRPC	79.83	-0.43±0.18	-1.36±0.88	0.36±0.16	0.38±0.14	0.36±0.16	0.36±0.16	0.36±0.16
	RTE	59.57	-1.62±0.47	-0.45±0.85	0.18±0.47	-0.45±0.18	0.18±0.47	0.18±0.47	0.09±0.45
	SUBJ	96.90	-0.50±0.09	-0.74±0.28	0.01±0.02	0.06±0.02	0.01±0.02	0.01±0.01	0.01±0.01
	Trec-Coarse	96.40	-0.05±0.10	-1.50±1.27	0.15±0.10	0.05±0.10	0.15±0.10	0.15±0.10	0.15±0.10
	Trec-Fine	91.60	-0.55±0.44	-1.05±3.71	0.05±0.10	0.05±0.10	0.05±0.10	0.05±0.10	0.05±0.10
RoBERTa	AG-News	94.36	-0.33±0.04	-0.31±0.13	0.12±0.06	0.10±0.03	0.12±0.06	0.12±0.06	0.12±0.06
	SST-2	94.29	-0.07±0.21	-1.13±0.92	0.27±0.12	0.41±0.08	0.27±0.12	0.27±0.12	0.27±0.12
	SST-5	56.11	0.14±0.08	-1.71±1.13	0.46±0.15	0.15±0.21	0.46±0.15	0.46±0.15	0.46±0.15
	MRPC	86.43	-0.96±0.15	-1.43±0.19	-0.01±0.21	0.00±0.14	-0.01±0.21	-0.01±0.21	-0.02±0.19
	RTE	75.09	-6.05±0.90	-8.39±1.57	0.54±0.21	0.45±0.35	0.54±0.21	0.54±0.21	0.54±0.21
	SUBJ	97.00	0.00±0.04	-0.21±0.19	0.09±0.01	0.09±0.12	0.09±0.01	0.09±0.01	0.09±0.01
	Trec-Coarse	96.20	-0.65±0.10	-2.20±0.75	0.20±0.16	0.15±0.10	0.20±0.16	0.20±0.16	0.20±0.16
	Trec-Fine	92.80	-0.35±0.10	-1.50±3.01	0.25±0.10	0.30±0.12	0.25±0.10	0.25±0.10	0.25±0.10