Confident sequence learning: A sequence class-label noise filtering technique to improve scene digit recognition

Abstract

Reading digits from natural images is a challenging computer vision task central to a variety of emerging applications. However, the increased scalability and complexity of datasets or complex applications bring about inevitable label noise. Because the label noise in the scene digit recognition dataset is sequence-like, most existing methods cannot deal with label noise in scene digit recognition. We propose a novel sequence class-label noise filter called Confident Sequence Learning. Confident Sequence Learning consists of two critical parts: the sequence-like confidence segmentation algorithm and the Confident Learning method. The sequence-like confidence segmentation algorithms slice the sequence-like labels and the sequence-like predicted probabilities, reorganize them in the form of the independent stochastic process and the white noise process. The Confident Learning method estimates the joint distribution between observed labels and latent labels using the segmented labels and probabilities. The TRDG dataset and SVHN dataset experiments showed that the confident sequence learning could find label errors with high accuracy and significantly improve the VGG-Attn and the TPS-ResNet-Attn model’s performance in the presence of synthetic sequence class-label noise.

Keywords

Scene digit recognition label noise confident learning

1 Introduction

Reading text (including digit) from natural images, referred to as scene text (digit) recognition, has been a vital computer vision task in many industrial applications [22]. Like other computer vision tasks, deep neural networks also have excellent performance on scene text recognition [25]. Since supervised learning requires training data labels, the outcome accuracy and algorithm performance of scene text(digit) models are positively related to the data used for training the model. However, text images’ annotation work is strenuous because there are multiple characters in a text image. So, the label noises in scene text recognition datasets are often inevitable. Previous results proved that the label noises in scene text recognition datasets could adversely impact scene text recognition models’ performance [43]. Therefore, training a robust scene text (digit) recognition model in the presence of label noise is an increasingly valued task.

Methods [10 , 37] to deal with label noise have been widely studied in recent years and applied to monotonic classification [9], distance metric learning [42], customer churn analysis [27]. However, training scene text recognition (STR) models require examples with a series of labels instead of a single label. Most previous methods cannot deal with a series of label noises, which we named sequence class-label noise. To deal with the sequence class-label noise in scene text recognition, inspired by the noise channel model [7], we formulate the stochastic process of sequence class-label noise. We also propose two sequence-like confidence segmentation algorithms: independent stochastic process segmentation algorithm and white noise segmentation algorithm. Besides, we integrate sequence-like confidence segmentation algorithms with the Confident Learning method [32] to find the corrupted labels. We named this method of confident sequence learning. According to the existing literature we found, confident sequence learning is the first class-label noise filtering technique applied to image-based sequence recognition. The main contributions of our work are as follows:

Inspired by the noisy channel model, we formulate the stochastic process of sequence class-label noise, filling the gap between scene text recognition and label noise. Compared with the noisy channel model, the stochastic process of sequence class-label noise considers the contextual information within a sequence of characters and models the noising process without relying on the word lexicon.

We propose two sequence-like confidence segmentation algorithms: independent stochastic process segmentation algorithm and white noise segmentation algorithm. We developed two confident sequence learning algorithms by integrating sequence-like confidence segmentation algorithms with the Confident Learning method. The confident sequence learning algorithms can filter the sequence class-label noise with linear time complexity.

We conduct extensive experiments on the TRDG dataset and SVHN dataset with different sequence class-label noise levels. The experiments show that confident sequence learning can find label errors with high accuracy and significantly improve the preference of the VGG-Attn model and the TPS-ResNet-Attn model. According to the existing literature, the confident sequence learning method is considered the state-of-art on scene digit recognition in the presence of the sequence class-label noise.

The rest of this paper is organized as follows. In Section 2, we discuss related work about scene text recognition and label noise. In Section 3, we review the scene digit recognition and sequence class-label noise. In Section 4, we formulate the stochastic process of sequence label noise and propose confident sequence learning. The experimental results are shown in Section 5. In Section 6, we compare the experimental results on the TRDG dataset with the experimental results on the SVHN dataset and discuss the difference between independent sequence learning (SCL: IN) and white noise sequence learning (SCL: WN). The conclusion is in Section 7.

2 Related work

This section reviews some of the previous work on topics closely related to our paper. We first briefly review scene text recognition, and then the studies of label noise are followed.

2.1 Scene text recognition

Reading text (including digit) from natural images is an important computer vision task. Optical Character Recognition (OCR) technique has led to its successful application on cleaned documents. However, most traditional OCR methods have failed to be as effective on the scene text recognition(STR) tasks due to the distinct text appearances in the real world and the imperfect conditions in which these scenes are captured. Unlike image classification tasks, STR tasks require the system to predict a series of labels instead of a single label, which means STR tasks are more complex than image classification tasks. Early STR models [6, 38] first detected individual characters and then recognized these detected characters with Deep Convolutional Neural Network models trained using labeled character images. However, such models required training a capable character detector for accurately detecting and cropping each character out from the original word image. Shi el at. [34] proposed an innovative neural network architecture CRNN, which can recognize character sequences without cropping each character out from the original word image. Although CRNN can handle regular character sequences in arbitrary lengths, it poorly performs when identifying text with an irregular shape. The rectification-based STR models [29, 35] attempted to transform irregular text patches into regular ones and then recognize them using regular text recognizers. Furthermore, the attention-based STR models [11, 26] achieved state-of-the-art performance on irregular text recognition benchmarks; they applied attention mechanisms to extract context information of features on the decoding stage. Nonetheless, training STR models require large amounts of labeled data. The training dataset used by irregular text recognizers may contain label errors because irregular text images’ annotation work is quite challenging. Moreover, the quality of training data has a significant impact on the performance of the STR models [15].

2.2 Label noise

Learning with noise has been a hot topic in the machine learning field. Anything that obscures the relationship between an instance’s features and its label can be seen as noise [2]. According to the previous literature [15], two types of noise are distinguished: feature noise and label noise. Feature noise affects the features’ observed values, e.g., the adversarial examples [17, 40]. The sequence noise studied in the Natural Language Processing (NLP) tasks [1 , 31] is also feature noise. Feature noise can affect the result outputted by deep neural network models at test time, while label noise is more harmful at training time. Label noise alters the observed labels assigned to instances, e.g., label flip attack [39]. Three types of methods to deal with label noise are distinguished [15]: label noise robust-models [16], label noise-tolerant algorithms [18], label noise filter approaches [10]. Confident learning is a kind of label noise filter approach that finds corrupted labels by estimating the joint distribution between observed labels and latent labels. Elkan [12] and Frank [14] pioneered counting methods to calculate false positive and false negative rates for binary classification, but these counting approaches perform poorly on unbalanced data sets. Elkan & Noto [13] introduced thresholding to learn a classifier from an unbalanced training set, but this method is limited to binary classification. For multi-classification, Lipton et al. [28] and Chen et al. [10] estimate label noise using approaches based on confusion matrices and cross-validation, but these methods fails to properly count errors for class imbalance or when a model is more confident for certain class than others. Recently, Northcutt et al. [32, 33] propose the Confident Learning (CL) method to directly estimate the joint distribution and prune error labels. The CL method outperforms seven recent state-of-the-art methods for learning with noisy labels. However, traditional confident learning methods can only filter class noise; they cannot deal with sequence class-label noise.

3 Preliminary

3.1 Scene digit recognition

The scene digit recognition task requires the system to predict a series of character labels using the images captured by scene digit detection systems. So, the feature space of scene digit recognition is $J \subset R^{a * b}$ . And, the label space of scene digit recognition is $L = {l_{0}^{k} : l_{0}^{k} = (l_{0}, l_{1}, \dots, l_{i}, \dots, l_{k})}, l_{i} \in L, k \leq K, K \in N$ , where L = {0, 1, 2 … , m} is the label set and K is the max length of digit sequence. We collect a training dataset with n samples $T : = {(I, \tilde{l_{0}^{k}})}^{n}$ , where samples $\tilde{l_{0}^{k}}$ are randomly sampled from sample space $J \times L$ . Like scene text recognition models [17 , 35], scene digit recognition models are also designed as an encoder-decoder framework. In the encoder stage, an image is transformed into a sequence of feature vectors $x_{0}^{T} = f_{1} (I)$ by CRNN [34]/ResNet [20]/VGG [36], and each feature vector corresponds to a region in the input image. In the decoder stage, the attention network [11] computes alignment factors and transforms them into a sequence of vector $y_{0}^{K} = f_{2} (x_{0}^{T})$ . The $y_{0}^{K}$ s termed as sequence-like predicted probability, and the ith vector y_i ∈ R^m is aligned with the ith target character. The jth element in the vector y_i is the predicted probability $\hat{p_{i}} (\hat{l_{i}} = j; I \in J, θ)$ .We use cross-entropy loss $O_{BCE} (\tilde{l_{0}^{k}}, y_{0}^{T}) = - \sum_{t = 1}^{T} log (\hat{p_{t}} (\hat{l_{t}} = \tilde{l_{t}}; I \in J, θ))$ to train STR models.

3.2 Sequence class-label noise

In scene digit recognition task, sequence label noise naturally occurs when human experts are involved. This phenomenon is termed as unknown noising process $Γ = P (\tilde{l_{0}^{k}} ∣ {l_{0}^{k}}^{*})$ [31], where ${l_{0}^{k}}^{*}$ is the true sequence label of the image I and $\tilde{l_{0}^{k}}$ is its distorted equivalent. According to the noisy channel model [7], four types of sequence label noises are distinguished:1.Deletion: A character in ${l_{0}^{k}}^{*}$ is deleted. 2. Insertion: A character is inserted into a random position in ${l_{0}^{k}}^{*}$ . 3. Substitution: A character is replaced with a random character in ${l_{0}^{k}}^{*}$ . 4. Swap: Two adjacent characters change position in ${l_{0}^{k}}^{*}$ .We only consider substitution and swap in this paper, and we term these two types of sequence label noises as sequence class-label noise. Deletion and insertion alter the distances between character sequences, which pose a disparate challenge for noise filter methods.

3.3 Sequence class-label noise filter

When training data is polluted by label noise, an obvious and tempting solution is cleansing the training data themselves. A classic type of data cleanses method is a noise filter. Noise filter methods detect noisy examples in the training dataset first, then remove them. Precisely, for a training dataset T in STR tasks, sequence class-label noise filter can remove the noisy examples in the training dataset T and get a clean dataset T^* ⊂ T. The STR models will perform better on the test dataset when training on a clean dataset T^*.

4 The proposed method

In this section, we propose a novel sequence class-label noise filter refer to as confident sequence learning. Compared with the Confident Learning method, confident sequence learning introduces the sequence-like confidence segmentation algorithm to segment the sequence-like noisy labels and sequence-like predicted probabilities. The sequence-like confidence segmentation algorithms are based on the stochastic process of sequence class-label noise in section 4.1.

4.1 The stochastic process of sequence class-label noise

To model the sequence class label noise, we formulate the stochastic process of sequence class label noise based on the noisy channel model [7], class conditional classification noise process(CNP) [2], and stochastic process [8]. Unlike the noisy channel model applied to spelling correction, the stochastic process of sequence class label noise did not rely on the word lexicon.

The stochastic process of sequence class-label noise:we call ξ a stochastic process of sequence class-label noise, if ξ consists of random vectors with a discrete parameter.

$ξ = {(\tilde{l_{i}}, {l_{i}}^{*}), i \leq k, k \in N}$ (1)

Where $\tilde{l_{i}}$ and l_i^* are random variable, $\tilde{l_{i}}$ represents the ith distorted label in the character sequence. l_i^* represents the ith latent(true) label in the character sequence, as shown in Fig. 1. Although the stochastic process of sequence class-label noise precisely models the noise process, it is hard to integrate this model with existing noise filter methods because we cannot estimate the joint distribution between random vectors. So, we simplify the model by using the independence condition.

Fig. 1

The stochastic process of sequence class label noise.

The independent stochastic process of sequence class label noise: We call ξ_ID an independent stochastic process of sequence class label noise, if ξ_ID consists of independent random vectors.

$\begin{matrix} ξ_{ID} = {(\tilde{l_{i}}, {l_{i}}^{*}), i \leq k, k \in N} \\ \forall i, j \leq k, p (\tilde{l_{i}}, {l_{i}}^{*}) = p_{i}, p (\tilde{l_{j}}, {l_{j}}^{*}) = p_{j} \\ p (\tilde{l_{i}}, {l_{i}}^{*}, \tilde{l_{j}}, {l_{j}}^{*}) = p_{i} \cdot p_{j} \end{matrix}$ (2)

In the independent stochastic process of sequence class label noise, each random vector is an independent class conditional classification noise process which means we can independently estimate the joint distribution between $\tilde{l_{i}}$ and l_i^*. To deal with class imbalance, we consider a stricter condition and propose the white noise of sequence class label noise.

The white noise of sequence class label noise:We call ξ_WN the white noise of sequence class label noise, if ξ_WN consists of independent and identically distributed random vectors.

$\begin{matrix} ξ_{WN} = {(\tilde{l_{i}}, {l_{i}}^{*}), i \leq k, k \in N} \\ \forall i, j \leq k, p (\tilde{l_{i}}, {l_{i}}^{*}) = p, p (\tilde{l_{j}}, {l_{j}}^{*}) = p \\ p (\tilde{l_{i}}, {l_{i}}^{*}, \tilde{l_{j}}, {l_{j}}^{*}) = p^{2} \end{matrix}$ (3)

In the white noise of sequence class label noise, all random vectors have the same joint distribution which brings huge convenience to estimate the joint distribution between $\tilde{l_{i}}$ and l_i^*. In section 4.2, we will introduce two sequence-like confidence segmentation algorithms: one is based on the independent stochastic process of sequence class-label noise, another is based on the white noise of sequence class-label noise.

4.2 Confident sequence learning

Although the sequence class-label noise models help us understand the relationship between the distorted labels and the latent labels, they cannot directly be used to find error labels without integrating with other noise filter methods. Therefore, we developed two sequence-like confidence segmentation algorithms: independent stochastic process segmentation algorithm and white noise segmentation algorithm. We combine sequence-like confidence segmentation algorithms with the CL method [32,33, 32,33] to propose two noise filter techniques. These techniques are called confident sequence learning. The confident sequence learning has three necessary steps as shown in Fig. 2. First, we segment the sequence-like noisy labels and sequence-like predicted probabilities using the sequence-like confidence segmentation algorithms. Then, we use the CL method (Cleanlab) to estimate the joint distribution between $\tilde{l_{i}}$ and l_i^* and find error labels. Finally, we count the number of character label errors in each sequence label, and we will remove the examples with the number of label errors higher than the threshold.

Fig. 2

The procedure of confident sequence learning.

Algorithm 1 Independent Stochastic Process Confidence Segmentation Algorithm

Input $\tilde{l_{0}^{k}}$ : an n × K matrix of sequence-like noisy labels; $\hat{P_{0}^{k}}$ : an n × m × K matrix consists of sequence-like predicted probabilities $y_{0}^{K}$ ; L: an n × 1 array of sequence length.

Output S: a list consists of $(\tilde{l_{i}}, \hat{P_{i}})$ , $\tilde{l_{i}}$ is an n_i × 1 array of the ith noisy labels, $\hat{P_{i}}$ is an n_i × m matrix of the ith predicted probability.

1: l_m← max(L)

2: S ← new empty list []

3: for i ← 1,l_m do do

4: append new empty tuple ([],[]) to S /* $\tilde{l_{i}}$ is S[i][0], $\hat{P_{i}}$ is S[i][1]*/

5: end for

6: for j ← 1,n do do

7: l ← L[j]

8: for k ← 1,l do do

9: append $\tilde{l_{0}^{k}} [j] [k]$ to S[k][0]

10: append $\hat{P_{0}^{k}} [j] [k]$ to S[k][1]

11: end for

12: end forreturn S

Algorithm 2 White Noise Confidence Segmentation Algorithm

Output $(\tilde{l}, \hat{P})$ , where $\tilde{l}$ is an N × 1 array of the noisy labels, $\hat{P}$ is an N × m matrix of the predicted probability.

1: $\tilde{l} \leftarrow$ new empty list []

2: $\hat{P} \leftarrow$ new empty list []

3: for j ← 1,n do do

4: l ← L[j]

5: for k ← 1,l do do

6: append $\tilde{l_{0}^{k}} [j] [k]$ to $\tilde{l}$

7: append $\hat{P_{0}^{k}} [j] [k]$ to $\hat{P}$

8: end for

9: end forreturn $(\tilde{l}, \hat{P})$

The sequence-like confidence segmentation algorithms: We developed two sequence-like confidence segmentation algorithms; the pseudo-codes are shown in Algorithm 1 and Algorithm 2. The independent confidence segmentation algorithm is based on the independent stochastic process of sequence class label noise. The white noise confidence segmentation algorithm is based on the white noise of sequence class label noise. Both two algorithms take three inputs: (1) $\tilde{l_{0}^{k}}$ : an n × K matrix of sequence-like noisy labels, (2) $\hat{P_{0}^{k}}$ : an n × m × K matrix consists of sequence-like predicted probabilities $y_{0}^{K}$ . (3)L: an n × 1 array of sequence length. We use a two-layer loop structure to slice the sequence-like noisy labels and the sequence-like predicted probabilities. The independent confidence segmentation algorithm reorganizes the noisy sliced label and sliced predicted probabilities in an independent stochastic process, and the white noise confidence segmentation algorithm reorganizes the noisy sliced label and sliced predicted probabilities in a white noise process. The time complexity of Algorithm 1 and Algorithm 2 is O (kn) which is a linear time complexity.

Confident Learning method (Cleanlab): Confident learning [32, 33] estimates the joint distribution between the observed labels and latent labels by taking noisy labels and predicted probabilities as inputs. We call it independent/white noise confident sequence learning if confident learning takes the noisy labels and predicted probabilities in independent stochastic process/white noise process as input. The first procedure of confident learning is computing confusion matrix $C_{\tilde{l}, l^{*}}$ . $\begin{matrix} C_{\tilde{l}, l^{*}} [i] [j] : = | {\hat{X}}_{\tilde{l}, l^{*}} | \\ {\hat{X}}_{l, l^{*}} : = {x \in {\hat{X}}_{\tilde{l} = i} : {\hat{p}}_{x, \tilde{l} = j} \geq t_{j}} \\ j = \underset{k \in 1, \dots, m : \tilde{l} = k \geq j_{k}}{argmax} {\hat{p}}_{x, \tilde{l} = k} \end{matrix}$ (4)

The threshold t_j is the expected (average) self-confidence for each class. $t_{j} = \frac{1}{| {\hat{X}}_{\tilde{l} = j} |} \sum_{x \in {\hat{X}}_{\tilde{l} = j}} \hat{p} (\tilde{l} = j; x, θ)$ (5)

The threshold t_j fixes the class sensitivity problem with $C_{\tilde{l}, l^{*}}$ and the label collisions are handled by selecting $\hat{l^{*}} \leftarrow \underset{j \in 1 \dots m}{argmax} {\hat{p}}_{x, \tilde{l} = j}$ . Given the confusion matrix $C_{\tilde{l}, l^{*}}$ , the Confident Learning method estimates the joint distribution ${\hat{Q}}_{\tilde{l}, l^{*}}$ ${\hat{Q}}_{\tilde{l}, l^{*}} = \frac{\frac{C_{\tilde{l} = i, l^{*} = j}}{\sum_{j \in 1.. m} C_{\tilde{l} = i, l^{*} = j}} \cdot | X_{\tilde{l} = i} |}{\sum_{i \in 1.. m, j \in 1.. m} (C_{\tilde{l} = i, l^{*} = j} \sum_{j \in 1.. m} C_{\tilde{l} = i, l^{*} = j} \cdot | X_{\tilde{l} = i} |)}$ (6)

Finally, for each off-diagonal entry in $C_{\tilde{l}, l^{*}}$ , the Confident Learning method selects the $n \cdot {\hat{Q}}_{\tilde{l} = i, l^{*} = j}$ examples $x \in X_{\tilde{l} = i}$ with max-margin ${\hat{p}}_{x, \tilde{l} = j} - {\hat{p}}_{x, \tilde{l} = i}$ and marks them as error labels. The time complexity of the independent confident sequence learning is O (k · (m² + mn)) and the time complexity of white noise confident sequence learning is O (m² + mkn).

Count & prune: Considering the small amount of label noise can improve the generalization of deep neural networks [43], we do not directly remove the error labels. We count the number of label errors in each sequence label. We will remove the examples that the number of label errors is higher than the threshold (we set it to 0 in experiments on the TRDG train dataset and 1 in experiments on the SVHN train dataset).

5 Experiment

In this section, we introduce the experiment architecture and analyze the experiment results. We conduct all the experiments on NICIDIA 2080Ti GPU. Besides, the operating system is Ubuntu 16.04LTS. The RAM size is 8G. Our code is available at our online code repository 1 .

5.1 Experiment setup

Dataset:To validate confident sequence learning effectiveness, we conduct our experiments on two benchmark datasets with different sizes and complexity. The first dataset is the TRDG dataset that includes 16000 synthetic images generated by TextRecognitionDataGenerator(TRDG) 2 . TRDG is an open-source synthetic data generator for text recognition. It can generate text image samples with different types to train lightweight scene digit recognition(SDR) models. The TRDG dataset is divided into three subsets: TRDG pretrain-4000 examples for pretraining models to generate predicted probabilities; TRDG test-2000 examples for testing; TRDG train-10,000 examples for official training. We have uploaded the TEDG dataset to the Baidu Netdisk 3 . The second dataset is Street View House Number (SVHN) dataset [33] which includes over 600,000 digit images captured from the realistic scenario. The SVHN dataset has been made available in two formats. We choose the full number format in our experiment. The SVHN dataset is also divided into three subsets: SVHN train-33397 examples for pretraining models to generate predicted probabilities, SVHN test-13066 examples for testing, SVHN extra-202311 examples for official training.

Network Architecture:We choose two SDR models [3] to evaluate confident sequence learning effectiveness. To interface with confident sequence learning, we made some changes to the original code and the latest version is available at our online repository 4 . For the TRDG dataset, we choose a lightweight scene text recognition model: the VGG-Attn network. The VGG-Attn network implements a VGG network [36] as an encoder in the feature extraction stage, and it implements a one layer LSTM attention decoder [11, 35] in the prediction stage. The VGG-Attn network is lightweight and it is suitable for simple scene digit recognition tasks (close to OCR). For the SVHN dataset, we choose the TPS-ResNet-Attn network to recognize the complex and irregular digits captured from real-world scenarios. The TPS-ResNet-Attn network adds a transformation stage that implements TPS [23] to transforms the input images to normalized images. The TPS-ResNet-Attn network implements a ResNet module [20] in the feature extraction stage to achieve better feature extractive performance. The TPS-ResNet-Attn network’s parameter size is 30 times larger than the parameter size of the VGG-Attn network.

Noise Generation:Following prior works [5, 21], we study the performance of confident sequence learning on substitutions type synthetic noise and swaps type synthetic noise. We generated two types of noisy data from clean data by randomly switching some labels to different classes non-uniformly according to the noise transition matrix [32]. The fraction of off-diagonals termed as sparsity in the noise transition matrix controls the strength of label noise. The noise transition matrixes we use in our experiments are in the Appendix.

Training:We implemented scene digit recognition models using the PyTorch framework. We used Adadelta [41] as an optimizer. The parameter LR was set to 0.9, and RHO was set to 0.95, EPS was set to 1e-08. The batch size was set to 32 for training the VGG-Attn model and was set to 64 for training the TPS-ResNet-Attn model. We trained our models for at most 40000 batches and used early stopping based on the validation set performance.

5.2 Performance measure

Following previous work [32], we use two types of metrics to measure the confident sequence learning method’s performance. The first type of metrics is used to evaluate the confident sequence learning method’s performance in finding misidentified characters in a sequence. The second type of metrics is used to evaluate the scene digit recognition models’ performance that trained with a dataset filtered by the confident sequence learning method.

Evaluation metrics for finding label errors: Finding misidentified characters in noisy scene digit recognition dataset is a multi-class problem. Accuracy, precision, recall, F1-Score are metrics that are usually used to evaluate multiple classification models and noise filter methods. For more clarity on these metrics, we present the confusion matrix for finding misidentified characters in a sequence with maximum length 2 in Table 2.

Table 1
Summary of related works regarding sequence confident learning

Task Noise Type Literature

Scene Text Recognition No Noise [6 , 38]

Scene Text Recognition Feature Noise [40]

Image Classification Feature Noise [17]

Image Classification Label Noise [10 , 39]

Natural Language Processing Feature Noise [1 , 31]

Task	Noise Type	Literature
Scene Text Recognition	No Noise	[6 , 38]
Scene Text Recognition	Feature Noise	[40]
Image Classification	Feature Noise	[17]
Image Classification	Label Noise	[10 , 39]
Natural Language Processing	Feature Noise	[1 , 31]

Table 2

Confusion matrix for finding Sequence class-label errors with maximum length 2

Actual\Predicted	Find zero misidentified character	Find one misidentified character	Find two misidentified characters
Zero misidentified character	X ₀₀	X ₀₁	X ₀₂
One misidentified character	X ₁₀	X ₁₁	X ₁₂
Two misidentified characte	X ₂₀	X ₂₁	X ₂₂

Accuracy is the most commonly used performance measure in multi-class problem; it is defined as the ratio of correct predictions to the total number of predictions. $Accuracy = \frac{\sum_{t = 0}^{2} X_{tt}}{\sum_{j = 0}^{2} \sum_{i = 0}^{2} X_{ij}}$ (7)

However, the use of accuracy measure improper as the rare samples significantly influences accuracy relative to the common ones [4]. So, we further consider precision and recall as metrics to measure the performance of finding sequence class-label errors. $Precision = \frac{\sum_{j = 0}^{2} \frac{X_{jj}}{\sum_{j = 0}^{2} X_{ij}}}{3}$ (8) $Recall = \frac{\sum_{i = 0}^{2} \frac{X_{ii}}{\sum_{j = 0}^{2} X_{ij}}}{3}$ (9)

F1-Score is another performance measure that is commonly used for imbalances classification problem. It considers both precision and recall. $F 1 = \frac{2 \cdot Precision \cdot Recall}{Precision + Recall}$ (10)

Evaluation metrics for scene text recognition models: To intuitively measure the confident sequence learning methods’ performance, we also evaluate the performance of scene digit recognition models trained with a dataset filtered by the confident sequence learning method. We mainly considered two metrics to measure the scene digit recognition models’ performance: sequence recognition accuracy and edit distance. Sequence recognition accuracy (SR Accuracy) is usually used to measure the performance of sequence recognition models, as defined by $SR Accuracy = \frac{M}{N}$ (11)

Where M represents the number of correctly recognized images, N represents the number of total images. The scene digit image is correctly recognized on the condition that no character is misidentified. We further consider Levenshtein distance (edit distance) to measure the performance of scene digit recognition models. The edit distance (ED) is a string metric for measuring the difference between two sequences. Informally, the edit distance between two words is the minimum number of single-character edits (i.e. insertions, deletions or substitutions) required to change one word into the other. It is defined as

$\begin{matrix} {ED}_{s_{1}, s_{2}} (i, j) \\ = {\begin{matrix} \max (i, j), if \min (i, j) = 0 \\ \min {\begin{matrix} {ED}_{s_{1}, s_{2}} (i - 1, j) + 1 \\ {ED}_{s_{1}, s_{2}} (i, j - 1) + 1 \\ {{ED}_{s_{1}, s_{2}} (i - 1, j - 1) + 1}_{s_{1} \neq s_{2}} \end{matrix}, otherwise \end{matrix} \end{matrix}$ (12)

5.3 Experiments on TRDG dataset

We generated sequence class label noise in the TRDG train dataset across varying noise rates and sparsities. All models evaluated on the unaltered TRDG test dataset. We used a pre-trained VGG-Attn model with a 0.8815 test accuracy to output the noisy TRDG train dataset’s predicted probabilities. The confident sequence learning methods estimated the joint distribution of noisy digits and true digits, as shown in Fig. 4.

Fig. 3

Examples of the TRDG dataset and the SVHN dataset.

Fig. 4

We estimate the joint distribution of noisy digits and true digits for the TRDG dataset with a 0.4 noise rate and a 0.6 sparsity using a white noise sequence learning method. Probabilities are scaled up by 100.

Table 3 shows the accuracy, F1, Precision, and recall measures for finding label errors in the TRDG dataset. According to the existing literature, we can’t find any other noise filter techniques that could be used on scene digit recognition task. We only compared the white noise sequence learning method (SCL: WN) and independent sequence learning method (SCL: IN). As shown in Table 3, the accuracy, precision, recall, and F1 of two confident sequence learning methods are all higher than 0.89 across varying noise rates and sparsities, which proves that confident sequence learning methods are able to find the label errors with a high success rate on TRDG dataset. Table 3 also shows that SCL: WN method performs better than SCL: IN method when finding label errors on the TRDG dataset with different noise levels except for a 40% noise rate with 0.2 sparsity.

Table 3

Accuracy, F1, precision, and recall measures for finding label errors in the TRDG dataset

Measure	ACCURACY				PRECISION				RECALL				F1
Noise rate(%)	20		40		20		40		20		40		20		40
Sparsity	0.2	0.6	0.2	0.6	0.2	0.6	0.2	0.6	0.2	0.6	0.2	0.6	0.2	0.6	0.2	0.6
SCL:WN	0.9027	0.9027	0.9001	0.9055	0.9400	0.9339	0.9270	0.9202	0.9027	0.9027	0.9001	0.9055	0.9159	0.9120	0.9097	0.9092
SCL:IN	0.9012	0.8994	0.9007	0.9013	0.9390	0.9310	0.9274	0.9168	0.9012	0.8994	0.9007	0.9013	0.9145	0.9089	0.9102	0.9053

After finding out label errors on the TRDG train dataset, we counted the number of character label errors in each sequence label. We removed the examples that have character label errors. We used the filtered TRDG train dataset to train the VGG-Attn network and evaluated the trained VGG-Attn network on the TRDG test dataset. We did not train the TPS-ResNet-Attn network on the TRDG dataset because the sample size of the TRDG dataset is too small, which will cause an overfit for the TPS-ResNet-Attn. Table 4 shows the evaluation results. Baseline refers to using the noisy TRDG train dataset without filtered noise. As shown in Table 4, the SCL: WN method and SCL: IN method significantly improve the test performance of the VGG-Attn model. Table 4 also indicates that SCL: WN method performs better when the sparsity of noise is low and SCL: IN method performs better when the sparsity of noise is high.

Table 4

Evaluation results for using the filtered TRDG train dataset to train the VGG-Attn network

Measure	SR Accuracy				Edit Distance
Noise rate (%)	20		40		20		40
Sparsity	0.2	0.6	0.2	0.6	0.2	0.6	0.2	0.6
BASELINE	81.95	73.20	74.75	48.50	86.70	204.27	128.11	455.30
SCL:WN	92.15	91.95	89.60	85.25	36.37	36.93	48.41	66.69
SCL:IN	93.00	91.85	90.20	82.2	30.74	38.78	44.59	74.96

5.4 Experiments on SVHN dataset

The SVHN dataset is a more challenging scene digit recognition benchmark compared with the TRDG dataset. Because the SVHN train dataset’s sample size is huge, the training scene digit recognition(SDR) models only need 1-3 epoch on the SVHN train dataset, which will reduce the effect of sequence class label noise. We increased the noise rate and sparsity when generating sequence class label noise in the SVHN train dataset. We used a pre-trained TPS-ResNet-Attn network with a 0.8832 test accuracy to output the noisy SVHN train dataset’s predicted probabilities. Table 5 shows the accuracy, F1, Precision, and recall measures for finding label errors in the SVHN train dataset.

Table 5
Accuracy, F1, precision, and recall measures for finding label errors in the SVHN dataset

Measure ACCURACY PRECISION RECALL F1

Noise rate (%) 40 60 80 40 60 80 40 60 80 40 60 80

Sparsity 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6

SCL: WN 0.9618 0.9633 0.9659 0.9649 0.9655 0.9664 0.9618 0.9633 0.9659 0.9627 0.9636 0.9660

SCL: IN 0.9621 0.9634 0.9658 0.9650 0.9657 0.9663 0.9621 0.9634 0.9658 0.9630 0.9637 0.9659

Measure	ACCURACY	PRECISION	RECALL	F1
SCL: WN	0.9618	0.9633	0.9659	0.9649	0.9655	0.9664	0.9618	0.9633	0.9659	0.9627	0.9636	0.9660
SCL: IN	0.9621	0.9634	0.9658	0.9650	0.9657	0.9663	0.9621	0.9634	0.9658	0.9630	0.9637	0.9659

As shown in Table 5, the accuracy, precision, recall, and F1 of two confident sequence learning methods are all higher than 0.96 across varying noise rates and sparsities. This result proves that confident sequence learning methods can find label errors with a high success rate on the SVHN dataset. Unlike the TRDG dataset experiments, Table 5 shows no significant difference between the two types of confident sequence learning for finding label errors in the SVHN train dataset.

To evaluate the effectiveness of confident sequence learning methods, we used the filtered SVHN train dataset to train the TPS-ResNet-Attn network and evaluated the trained TPS-ResNet-Attn network on the SVHN test dataset. We did not train the VGG-Attn network on the SVHN train dataset because the VGG-Attn network is ill-suited to the SVHN dataset. The SVHN train dataset images are corrupted by natural phenomena that are difficult to compensate for by hand. The VGG-Attn network cannot recognize images with severe blur, distortion, and illumination effects on top of wide style and font variations. Table 6 shows the evaluation results for using the filtered SVHN train dataset to train the TPS-ResNet-Attn network. As shown in Table 6, he accuracy increased by 1.5% and 2%, respectively, from the lowest noise rate (40%) to the highest noise rate (80%), which prove that confident sequence learning methods are effective to improve the performance of TPS-ResNet-Attn network under sequence class label noise. Table 6 also shows that SCL: WN is more effective for enhancing the sequence class-label noise robustness of the TPS-ResNet-Attn network compared with the SCL: IN method.

Table 6

Evaluation results for using the filtered SVHN train dataset to train the TPS-ResNet-Attn network

Measure	SR Accuracy			Edit Distance
Noise rate (%)	40	60	80	40	60	80
Sparsity	0.6	0.6	0.6	0.6	0.6	0.6
BASELINE	92.653	92.354	91.015	601.93	604.55	739.97
SCL:WN	94.341	93.722	93.278	496.18	530.63	593.27
SCL:IN	93.094	93.829	92.107	591.02	541.18	699.55

6 Discussion

This section compares the experimental results on the TRDG dataset with the experimental results on the SVHN dataset first. Besides, it discusses the difference between the SCL: WN method and the SCL: IN method.

When comparing the experimental results on the TRDG dataset with the experimental results on the SVHN dataset, one interesting finding is that the confident sequence learning methods’ performance is positively related to the robustness of the scene digit recognition models. As shown in Table 4, the confident sequence learning methods significantly improve the VGG-Attn model’s performance on the noisy TRDG dataset (the accuracy improved by an average of 19.92%). Meanwhile, the VGG-Attn model’s accuracy sharply decreases when the SVHN training dataset’s noise rate increases, which indicates that the VGG-Attn model is not robust to the sequence class-label noise. Similar results are presented in Table 6.

In Section 5, we also find that the SCL: WN method and the SCL: IN method perform differently across varying noise rates and sparsities. From a theoretical perspective, the independent and identically distributed condition is more restrictive to the probability distribution of random vectors in noisy sequence labels. This conclusion indicates that the SCL: WN method is more suitable for situations that the sparsity of sequence class label noise is high. The experiment results also show that when the sparsity of sequence class label noise is high, using the training dataset filtered by SCL: WN method to train scene digit recognition models improve the test accuracy by 1% more than using SCL: IN method.

7 Conclusion

In this paper, we reviewed the scene digit recognition task and the sequence class label problem. We formulated the stochastic process of sequence class-label noise. Furthermore, we proposed two sequence-like confidence segmentation algorithms: independent stochastic process confidence segmentation algorithm and white noise stochastic process confidence segmentation algorithm. We proposed confident sequence learning, the first class-label noise filtering technique applied to image-based sequence recognition, by integrating two sequence-like confidence segmentation algorithms with confident learning. The TRDG dataset and SVHN dataset experiments show that confident sequence learning methods can find label errors with 0.9 accuracies. Besides, our experiment shows that confident learning methods improve the SR Accuracy of the VGG-Attn model by an average of 19.92% and improve the SR Accuracy of the TPS-ResNet-Attn model by an average of 1.04%. We think the confident sequence learning methods’ performance is highly related to the scene digit recognition models’ robustness. We also found that the SCL: WN method is more effective when the sparsity of noise is high compared with the SCL: IN method.

The confident sequence learning methods have two limits: (1) the confident sequence learning method can only be applied to attention-based scene text recognition models. (2) the confident sequence learning methods cannot filter the sequence label noise caused by insertions and deletions. In the future, our work will involve the improvement in the proposed sequence label noise model to deal with the noise caused by deletion and insertion. Moreover, we plan to evaluate the confident sequence learning method on scene text recognition datasets (e.g., MJSynth[?] and SynthText [50]) that is more complex and have a larger simple size compare with scene digit recognition datasets.

Footnotes

Appendix

Fig. 5

Noise transition matrix

Acknowledgment

This work is supported by ’the Fundamental Research Funds for the Central Universities(2020YJ003)’

URL:, Password: udd9

References

Alex

and Burns

, Estimating and rating the quality of optically character recognised text, In Apostolos Antonacopoulos and Klaus U. Schulz, editors, Digital Access to Textual Cultural Heritage 2014, DATeCH 2014, Madrid, Spain, May 19–20, 2014, pages 97–102, ACM, 2014.

Angluin

and Laird

P.D.

, Learning from noisy examples, Mach Learn 2(4) (1987), 343–370.

Baek

, Kim

, Lee

, Park

, Han

, Yun

, Oh

S.J.

and Lee

, What is wrong with scene text recognition model comparisons? dataset and model analysis, In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 – November 2, 2019, pages 4714–4722, IEEE, 2019.

Beernaerts

, Debever

, Lenoir

, De Baets

and de Weghe

N.V.

, A method based on the levenshtein distance metric for the comparison of multiple movement patterns described by matrix sequences of different length,, Expert Syst Appl 115 (2019), 373–385.

Belinkov

and Bisk

, Synthetic and natural noise both break neural machine translation, In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 – May 3, 2018, Conference Track Proceedings, OpenReview.net, 2018.

Bissacco

, Cummins

, Netzer

and Neven

, Photoocr: Reading text in uncontrolled conditions, In IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1–8, 2013, pages 785–792, IEEE Computer Society, 2013.

Brill

and Moore

R.C.

, An improved error model for noisy channel spelling correction, In 38th Annual Meeting of the Association for Computational Linguistics, Hong Kong, China, October 1–8, 2000, pages 286–293. ACL, 2000.

Spodarev

Bulinski

Cano

J.R.

, Luengo

and García

, Label noise filtering techniques to improve monotonic classification, Neurocomputing 353 (2019), 83–95.

10.

Chen

, Liao

, Chen

and Zhang

, Understanding and utilizing deep neural networks trained with noisy labels, In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9–15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1062–1070. PMLR, 2019.

11.

Cheng

, Bai

, Xu

, Zheng

, Pu

and Zhou

, Focusing attention: Towards accurate text recognition in natural images, In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22–29, 2017, pages 5086–5094. IEEE Computer Society, 2017.

12.

Elkan

, The foundations of cost-sensitive learning, In Bernhard Nebel, editor, Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, IJCAI 2001, Seattle, Washington, USA, August 4–10, 2001, pages 973–978. Morgan Kaufmann, 2001.

13.

Elkan

and Noto

, Learning classifiers from only positive and unlabeled data, In Ying Li, Bing Liu and Sunita Sarawagi, editors, Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24–27, 2008, pages 213–220. ACM, 2008.

14.

Forman

, Counting positives accurately despite inaccurate classification, In João Gama, Rui Camacho, Pavel Brazdil, Alípio Jorge and Luís Torgo, editors, Machine Learning: ECML 2005, 16th European Conference on Machine Learning, Porto, Portugal, October 3–7, 2005, Proceedings, volume 3720 of Lecture Notes in Computer Science, pages 564–575. Springer, 2005.

15.

Frénay

and Verleysen

, Classification in the presence of label noise: A survey, IEEE Trans Neural Networks Learn Syst 25(5) (2014), 845–869.

16.

Ghosh

, Kumar

and Sastry

P.S.

, Robust loss functions under label noise for deep neural networks, In Singh

Satinder P.

and Markovitch

Shaul

, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4–9, 2017, San Francisco, California, USA, pages 1919–1925, AAAI Press, 2017.

17.

Goodfellow

I.J.

, Shlens

and Szegedy

, Explaining and harnessing adversarial examples, In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings, 2015.

18.

Han

, Yao

, Yu

, Niu

, Xu

, Hu

, Tsang

I.W.

and Sugiyama

, Co-teaching: Robust training of deep neural networks with extremely noisy labels, In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December 2018, Montréal, Canada, pages 8536–8546, 2018.

19.

Han

, Luo

and Wang

, Deep self-learning from noisy labels, In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 – November 2, 2019, pages 5137–5146. IEEE, 2019.

20.

, Zhang

, Ren

and Sun

, Deep residual learning for image recognition, In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016, pages 770–778. IEEE Computer Society, 2016.

21.

Heigold

, Varanasi

, Neumann

and van Genabith

, How robust are character-based word embeddings in tagging and MT against word scrambling or random nouse? In Colin Cherry and Graham Neubig, editors, Proceedings of the 13th Conference of the Association for Machine Translation in the Americas, AMTA 2018, Boston, MA, USA, March 17–21, 2018 – Volume 1: Research Papers, pages 68–80. Association for Machine Translation in the Americas, 2018.

22.

Jaderberg

, Simonyan

, Vedaldi

and Zisserman

, Synthetic data and artificial neural networks for natural scene text recognition, CoRR, abs/1406.2227, 2014.

23.

Jaderberg

, Simonyan

, Zisserman

and Kavukcuoglu

, Spatial transformer networks, In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7–12, 2015, Montreal, Quebec, Canada, pages 2017–2025, 2015.

24.

Karpukhin

, Levy

, Eisenstein

and Ghazvininejad

, Training on synthetic noise improves robustness to natural noise in machine translation, In Wei Xu, Alan Ritter, Tim Baldwin, and Afshin Rahimi, editors, Proceedings of the 5th Workshop on Noisy User-generated Text, W-NUT@EMNLP 2019, Hong Kong, China, November 4, 2019, pages 42–47. Association for Computational Linguistics, 2019.

25.

, Wang

, Shen

and Zhang

, Show, attend and read: A simple and strong baseline for irregular text recognition, In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 – February 1, 2019, pages 8610–8617. AAAI Press, 2019.

26.

, Wang

, Shen

and Zhang

27.

, Wei

, Kang

and Wu

, An efficient noise-filtered ensemble model for customer churn analysis in aviation industry, J Intell Fuzzy Syst 37(2) (2019), 2575–2585.

28.

Lipton

Z. C.

, Wang

Y.-X.

and Smola

A.J.

, Detecting and correcting for label shift with black box predictors, In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10–15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3128–3136. PMLR, 2018.

29.

Liu

, Chen

, Wong

K.-Y.K.

, Su

and Han

, Star-net: A spatial attention residue network for scene text recognition, In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19–22, 2016. BMVA Press, 2016.

30.

Malykh

, Robust to noise models in natural language processing tasks, In Fernando Emilio Alva-Manchego, Eunsol Choi, and Daniel Khashabi, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 – August 2, 2019, Volume 2: Student Research Workshop, pages 10–16. Association for Computational Linguistics, 2019.

31.

Namysl

, Behnke

and Köhler

, NAT: noise-aware training for robust neural sequence labeling, In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5–10, 2020, pages 1501–1517. Association for Computational Linguistics, 2020.

32.

Northcutt

C.G.

, Jiang

and Chuang

I.L.

, Confident learning: Estimating uncertainty in dataset labels, CoRR, abs/1911.00068, 2019.

33.

Northcutt

C.G.

, Wu

and Chuang

I.L.

, Learning with confident examples: Rank pruning for robust classification with noisy labels, In Gal Elidan, Kristian Kersting, and Alexander T. Ihler, editors, Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence, UAI 2017, Sydney, Australia, August 11–15, 2017. AUAI Press, 2017.

34.

Shi

, Bai

and Yao

, An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition, IEEE Trans Pattern Anal Mach Intell 39(11) (2017), 2298–2304.

35.

Shi

, Wang

, Lyu

, Yao

and Bai

, Robust scene text recognition with automatic rectification, In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016, pages 4168–4176, IEEE Computer Society, 2016.

36.

Simonyan

and Zisserman

, Very deep convolutional networks for large-scale image recognition, In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings, 2015.

37.

van Rooyen

, Menon

A.K.

and Williamson

R.C.

, Learning with symmetric label noise: The importance of being unhinged, In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7–12, 2015, Montreal, Quebec, Canada, pages 10–18, 2015.

38.

Wang

, Wu

D.J.

, Coates

and Ng

A.Y.

, End-to-end text recognition with convolutional neural networks, In Proceedings of the 21st International Conference on Pattern Recognition, ICPR 2012, Tsukuba, Japan, November 11–15, 2012, pages 3304–3308. IEEE Computer Society, 2012.

39.

Xiao

, Biggio

, Nelson

, Xiao

, Eckert

and Roli

, Support vector machines under adversarial label contamination,, Neurocomputing 160 (2015), 53–62.

40.

, Chen

, Xiao

, Gao

, Shen

and Shen

H.T.

, What machines see is not what they get: Fooling scene text recognition models with adversarial text images, In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13–19, 2020 pages 12301–12311. IEEE, 2020.

41.

Zeiler

M.D.

, ADADELTA: an adaptive learning rate method, CoRR, abs/1212.5701, 2012.

42.

Zeng

, Zhang

and Zheng

, Re-kissme: A robust resampling scheme for distance metric learning in the presence of label noise, Neurocomputing 330 (2019), 138–150.

43.

Zhang

, Bengio

, Hardt

, Recht

and Vinyals

, Understanding deep learning requires rethinking generalization, In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings OpenReview.net, 2017.