Dual-threshold sample selection with latent tendency difference for label-noise-robust pneumoconiosis staging

Abstract

Background

The precise pneumoconiosis staging suffers from progressive pair label noise (PPLN) in chest X-ray datasets, because adjacent stages are confused due to unidentifialble and diffuse opacities in the lung fields. As deep neural networks are employed to aid the disease staging, the performance is degraded under such label noise.

Objective

This study improves the effectiveness of pneumoconiosis staging by mitigating the impact of PPLN through network architecture refinement and sample selection mechanism adjustment.

Methods

We propose a novel multi-branch architecture that incorporates the dual-threshold sample selection. Several auxiliary branches are integrated in a two-phase module to learn and predict the progressive feature tendency. A novel difference-based metric is introduced to iteratively obtained the instance-specific thresholds as a complementary criterion of dynamic sample selection. All the samples are finally partitioned into clean and hard sets according to dual-threshold criteria and treated differently by loss functions with penalty terms.

Results

Compared with the state-of-the-art, the proposed method obtains the best metrics (accuracy: 90.92%, precision: 84.25%, sensitivity: 81.11%, F1-score: 82.06%, and AUC: 94.64%) under real-world PPLN, and is less sensitive to the rise of synthetic PPLN rate. An ablation study validates the respective contributions of critical modules and demonstrates how variations of essential hyperparameters affect model performance.

Conclusions

The proposed method achieves substantial effectiveness and robustness against PPLN in pneumoconiosis dataset, and can further assist physicians in diagnosing the disease with a higher accuracy and confidence.

Keywords

pneumoconiosis staging chest X-ray label noise sample selection

Introduction

Pneumoconiosis is a group of heterogeneous occupational interstitial lung diseases caused by the inhalation of mineral dust, and can lead to chronic pulmonary inflammation and fibrosis.¹ These potentially severe symptoms have been threatening the health of millions of mine workers exposed to dust worldwide.² Discovering its early symptoms and determining the stage precisely are essential for disease prevention and diagnosis. Beyond necessary epidemiological investigation of occupational dust exposure, radiologists follow a specification that grades the disease from Stage 0 (considered to be normal) through Stage 3, and determine the stage by comparing chest X-ray radiographs (CXR) of patients to the standard films (Figure 1). The stages are primarily determined by assessing small opacity agglomerations in the lung fields. The opacities are so tiny and diffuse that even the experienced radiologists may unintentionally confuse them with normal lung tissue, resulting in incorrect diagnostic conclusions. To this end, one possible solution is to introduce deep neural networks (DNNs) for pneumoconiosis stage determination. However, the primary methods introduced only simple techniques^3–7 and did not consider the noisy labels caused during annotation procedure.

Figure 1.

Standard pneumoconiosis CXR films of Stage 0, Stage 1, Stage 2 and Stage 3.

In medical images, label noise originates synthetically from both subjective and objective factors. And CXRs of pneumoconiosis are typical examples among them. One of the key subjective factors is inter/intra-observer variability, which indicates the label uncertainty caused by disagreement of multiple annotators^8–10 or inconsistent determination criteria by a single individual.¹¹ Objectively, the intrinsic similarity and deviation among medical image samples of different classes affect the acquisition of true labels. From a domain-specific perspective, it appears in pneumoconiosis staging that one class may only be confused with its neighbouring classes. This is because, apart from varied opinion of different observers, the radiological feature is nuanced at the decision boundary between neighbouring classes, as illustrated in Figure 2. It remains ambiguous how the underlying semantic feature of CXRs correlates with the stages. In this paper, we denote this special type of label noise as progressive pair label noise (PPLN), because the diffuse opacities deteriorate along with the progression of the patient’s condition.

Figure 2.

The radiological feature of pneumoconiosis in CXRs by and large varies continuously and chronologically along with the patients’ condition progression in principle, as aligned with the grey thick dashed bi-directional arrow. Some typical samples in real-world dataset are spreaded across the arrow. And the bi-directional blue arrow sketches the concept of progressive feature tendency we propose in this paper.

For one thing, PPLN may not simply be ascribed to either class-conditional or instance-dependent factors. In fact, common label noise cases presume that different classes have definite criteria and clear boundaries, whereas pneumoconiosis staging depends on empirical deduction by experienced experts with fuzzy, undetermined standards. For another, some researchers have taken this special label noise type as a basic assumption. Sun et al.¹² proposed a log-normal label distribution learning method to alleviate the ambiguity at the decision boundary between pneumoconiosis stages. However, unlike most label distribution scenarios with densely sequenced labels and corresponding continuous metrics,^11,13 pneumoconiosis staging labels are relatively sparse, and there has not been a sufficiently evident metric that linearly quantifies the density of small opacities. This can introduce profound unreliability into the formulation of the specific label ambiguity.

Therefore, this study is motivated to explore a novel and reasonable paradigm to mitigate the impact of confusion between adjacent classes under the setting of PPLN. An intuitive way is to train several smart virtual experts at the decision boundaries to uncover the potential criteria. Working as a special observer, each of the experts is responsible for distinguishing the confusing samples into two adjacent stages. As in Figure 2, the confusing samples are usually located in the dashed circles, within which the samples are prone to be mistakenly assigned to either of the adjacent stages. Yet, this plausible conduction is difficult to achieve because the potentially genuine criteria for classification may not be easily found. If the standpoint of an expert is shifted from the boundary to a certain stage (e.g., Stage 1), we believe that there should exist an implicit progressive feature tendency (details in Section “Potential neighbouring label mining”) in both the mild and severe directions. That is, standing at Stage 1, observers may find in the lung fields that Stage-0 samples have a milder opacity agglomeration, while Stage-2 and -3 ones appear in a severer condition. Such a tendency inspires us to ascertain if it may assist in making decision at each boundary in a discretized binary manner.

In this paper, we design a novel multi-branch architecture named FTLTD-Net (a Network based on Feature Tendency and Latent Tendency Difference), as illustrated in Figure 3, to tackle PPLN in pneumoconiosis CXR dataset for precise staging. Based on the concept of progressive feature tendency, FTLTD-Net (1) utilizes a two-phase module with multiple auxiliary branches to discern the causal pattern of adjacent-class ambiguity by learning from adjacent-class samples and predicting the progressive feature tendency; (2) incorporates the outcome of tendency prediction to calculate the Latent Tendency Difference (LT-Diff) of each sample for improving dynamic selection of trusted samples. These two aspects correspond to both class-conditional and instance-dependent factors of PPLN. Briefly, each of the auxiliary branches plays the role of tendency learner and predictor as a binary classifier. This structure tackles the confusion originating from a class-conditional perspective. Moreover, as the dynamic instance-specific threshold (DIST) has been shown to be effective in existing sample selection methods,¹⁴ we introduce a difference-based threshold to complement the selection criteria. Instead of iterating for a continuous increase through training epochs like a probability-based confidence threshold, the difference-based one is expected to decrease dynamically to narrow the gap between the bi-directional progressive feature tendencies of potential true-labelled samples. To this end, the model computes LT-Diff from the confidence scores of progressive feature tendency and considers such a difference-based metric in the loss functions as penalty terms. The difference-based threshold works with probability-based one as DIST to constitute a dual-threshold sample selection, which intends to improve the reliability of selected samples against instance-dependent factors of PPLN.

Figure 3.

The overall architecture of FTLTD-Net consists of 3 parts. (a) The main branch serves as a common classifier. (b) The Potential Neighboring Label Disambiguation is embedded in the auxiliary branches. The module finally determines the progressive feature tendency of samples and generates the dynamic latent difference threshold for sample selection. (c) Both probability- and difference-based dynamic thresholds are involved in sample selection to partition the samples into clean and hard sets. Here is a demonstration how the image instance $(X_{m}, y_{m} = 1)$ is actually processed within the model, and the input paths of other possible instances can be simply inferred.

Our main contributions in this paper are concluded as follows.

To adapt PPLN, a two-phase assistant module is integrated into FTLTD-Net as auxiliary branches to learn and predict the progressive feature tendency.

A branch-consistent focal loss function is proposed and incorporated to learn the consistent pattern between the main and auxiliary branches, and to avoid occasional sample imbalance.

A special metric, LT-Diff, is proposed from the progressive feature tendency prediction, and the double layback-ratio trick is employed in the iteration of LT-Diff-based threshold for stricter sample selection criteria.

Penalty terms based on LT-Diff are appended to the sample selection loss functions to enforce the LT-Diff-based thresholds to decrement.

Related work

Noisy labels are referred to as inaccurate labels corrupted from the real ones. Due to their powerful representational capability, DNNs are sensitive to, or can even memorize, noisy labels. Learning from noisy labels aims at establishing robust training paradigms for DNNs against noisy samples. Intensive efforts have been made to address this problem, and the existing approaches are categorized into several orthogonal directions.¹⁵ In this section, we reorganize the taxonomy from the perspective of our motivation.

Noise-type-oriented mechanism

In fact, the noise type is regarded as the hypothesis how the labels are corrupted from ground truth. Generally, the corruption probability is essentially affected by the dependency between data features and class labels. Label noise can be categorized into instance-independent noise (IIN) and instance-dependent noise (IDN).

For IIN, the prototypical approach was to design dedicated architectures by simulating the label transition process, with the primary technique being to add a noise adaptation layer.^16,17 Patrini et al.¹⁸ designed a two-procedure approach that is agnostic to both the application domain and network architecture. Yet IIN is actually an impractical assumption for real-world datasets due to substantial variance among training samples. Recent studies have focused on IDN, which considers more about instance-specific confusing factors. Xia et al.¹⁹ proposed part-dependent label noise to describe IDN by approximating a combined transition matrix from parts of instances, whereas it relies on loss correction by estimating noise rates. Later, Cheng et al.²⁰ designed a confidence-regularized sample sieve to eliminate the estimation of noisy rate. Observing that these methods focused on first-order statistics, Zhu et al.²¹ exploited second-order statistics to transform an IDN problem into a new class-dependent label noise. Garg et al.²² presented a graphical modelling approach named InstanceGM, combining discriminative and generative models. Yang et al.²³ investigated directly modelling the transition from Bayes optimal labels to noisy labels (Bayes-Label Transition Matrix). Observing low search efficiency in instance-dependent transition matrices, Zhang et al.²⁴ introduced a human-cognition-assisted structured transition matrix network (STMN) for better matrix estimation performance.

However, none of these have incorporated into network design a special noise type where there are no definite criteria discriminating adjacent classes and the confusion concentrates at the decision boundary. PPLN in pneumoconiosis CXR dataset is a typical example, which is considered in our proposed FTLTD-Net.

Sample selection

Sample selection attempts to select true-labelled examples and exclude those false-labelled ones from a noisy training dataset. Empirical studies have proved that the deep networks memorize simple, generalized patterns in the early phases of training.^25,26

Initially, Malach et al.²⁷ proposed a meta-algorithm to maintain two networks and update them using the training data selected based on the disagreement of each network. Several pioneering researchers assumed that the clean (i.e., potentially true-labelled) samples tend to have small loss values.^28–31 Han et al.²⁸ proposed a co-teaching method by simultaneously training two networks to filter possible noisy samples according to the training loss of each minibatch. Li et al.³¹ proposed a framework, DivideMix, inspired by semi-supervised learning techniques for learning with noisy labels. DivideMix dynamically divides training samples into a clean set and a corrupted set, and then trains models on both sets with MixMatch³² strategy. Later, some researchers found that small-loss trick does not always work. Lyu et al.³³ proposed adaptively selecting trusted samples for model training by a designed curriculum loss. Lu et al.³⁴ proposed a method called self-ensemble label correction (SELC) to correct noisy labels using model predictions of historical epochs. Inspired by SELC, Wang et al.³⁵ designed a novel stochastic neural ensemble learning (SNEL) with the idea of collecting parameters simultaneously both across model instances and along optimization trajectories.

Despite some approaches not explicitly narrating the targeting noise type, intuitively, sample selection can handle the training dataset with IDN based on confidence scores calculated from samples. Rather than using a fixed confidence value,³⁶ some recent studies employed an instance-specific threshold mechanism dynamically or adaptively. Li et al.¹⁴ proposed dynamic instance-specific selection and correction (DISC) with a dynamic threshold strategy for all the instances, each of which is grouped into subsets marked clean, hard, and purified. By employing semi-supervised learning, Li et al.³⁷ devised a novel adaptive threshold for unlabelled instances, providing a bounded probabilistic guarantee for the correctness of the pseudo-labels it assigns. Inspired by curriculum learning,³³ Wu et al.³⁸ proposed a time-consistency curriculum to select samples with consistent predictions over training epochs. However, most of the methods focused on probability-based confidence scores transformed into adaptive thresholds for sample selection. These methods did not consider the intrinsic feature tendency and difference, as in PPLN. The proposed FTLTD-Net utilizes a latent difference metric to investigate a novel form of dynamic-threshold sample selection.

Method

As shown in Figure 3, the fundamental architecture of FTLTD-Net consists of three modules: (a) the main branch serving as a common classifier; (b) the PNLD (Potential Neighboring Label Disambiguation) module containing several auxiliary branches; and (c) the dual-threshold sample selection leveraging a difference-based measurement. Such an architecture considers both class-conditional and instance-dependent factors of PPLN. The auxiliary branches in PNLD are designed to match the corresponding classes to simulate the potential class-conditional factor of PPLN, and discern the causal pattern that results in adjacent-class ambiguity. This auxiliary module aims at analytically learning from the pattern and making an assertion about which of the two adjacent classes has a more definite possibility of being confused. The assertion is based on the newly-proposed progressive feature tendency (PFT), which is observed from one sample to determine whether it conveys symptomatically milder or severer feature than its adjacent-class samples. Such a process can be interpreted as learning and predicting the underlying PFT in two respective phases. Subsequently, calculated from the assertion based on the second phase of PNLD, a special metric named LT-Diff is incorporated for each sample to obtain one of the dual dynamic thresholds for sample selection. Such dynamically decreasing thresholds supplement criteria according to which the samples are partitioned into clean or hard sets. The dual dynamic thresholds update along with training epochs to incorporate the instance-dependent factor of PPLN.

In this section, we first formulate the problem to be solved, especially the special label noise type PPLN, and demonstrate in detail the modules introduced into this novel architecture.

Problem formulation

Objective

Let the grey-scale image space be denoted as $X \subset R^{H \times W}$ , and the label space as $Y = 0, 1, \dots, C - 1$ , where $H, W, C$ denote the height, width and number of classes, respectively. Assume that the noisy training dataset $D = {(X_{i}, y_{i})}_{i = 1}^{N}$ is drawn independently and identically from a distribution $P_{D}$ over $X \times Y$ , where the noisy labels may exist. Generally, the goal of learning from noisy labels is to search the parameter space $Θ$ for the potential optimal parameter $ϑ$ to obtain a function $M (\cdot; ϑ)$ with an objective $L$ , such that the empirical risk $R$ is minimized. Formally, without the definite ground-truth labels, the optimal parameter is obtained by

ϑ^{*} = \arg min_{ϑ \in Θ} R_{D}^{L} = \arg min_{ϑ \in Θ} E_{(X, y) \sim D} [L (M (X; ϑ), y)]

(1)

Progressive pair label noise

Here we define this special label noise type with intrinsically continuous feature and sparsely discrete classes. For a given label $y \in Y$ , the potential neighbouring label (PNL) can be denoted as

N e i g h b o r_{C} (y) = {\begin{matrix} y \pm 1, & if y \notin {0, C - 1}, \\ y + 1, & if y = 0, \\ y - 1, & if y = C - 1 \end{matrix}

(2)

The neighbouring latent labels can form a set called potential neighbouring label set

Q_{C} (y) = {\tilde{y} | \tilde{y} = N e i g h b o r_{C} (y)}

. For simplicity, we further define the generalized potential neighbouring label (GPNL) set as

{\tilde{Q}}_{C} (y) = {\begin{matrix} Q_{C} (y) \cap {y}, & if y \in {0, C - 1}, \\ Q_{C} (y), & otherwise \end{matrix}

(3)

This setting ensures that

| {\tilde{Q}}_{C} (y) | = 2

, and either

y \in {0, C - 1}

can be regarded as a margin class, treated as a special case (see Section “Potential neighbouring label mining”). Correspondingly, every element in

{\tilde{Q}}_{C} (y)

can be regarded as a generalized potential neighbour (GPN) of y. These newly introduced concepts simply describe how PPLN manifests itself in the dataset. In addition, PFT can be interpreted as the extent to which a label y might be confused with either of its GPNs in

{\tilde{Q}}_{C} (y)

and quantified with a confidence score.

Potential neighbouring label disambiguation

This module is introduced into the architecture to explore the class-conditional factor of PPLN by learning and predicting the PFT. Its main structure is several auxiliary branches, the number of which is equal to that of the classes. Working in parallel to the main branch, the auxiliary branches diverge from the tail of a backbone network (e.g., a ResNet³⁹), while the head remains.

Module input

The backbone head $f (\cdot)$ (e.g., the first 3 layers of a ResNet) is treated as a feature extractor in both the main and auxiliary branches with shared parameters. For a given sample $(X_{m}, y_{m})$ , the head generates
$h_{m} = f (X_{m})$
(4)
And its tail (e.g., the last layer of a ResNet) diverges in multiple branches with respect to the number of classes C, each of which is denoted as $g_{j} (\cdot), j = 0, 1, \dots, C - 1$ . Note that the feature extractors share parameters in both the main branch and the auxiliary branches. For simplicity, the main branch is indexed as C. Then each possible feature of $X_{m}$ at the j-th branch is denoted as
$z_{m}^{(j)} = g_{j} (h_{m}), j = 0, 1, \dots, C - 1, C,$
(5)
where $z_{m}^{(j)} \in R^{D}$ holds, and D is the feature dimension. By possible, we mean that the branch to which a sample is input relies on different phases.

In general, PNLD consists of two phases, each of which starts with a feature split to adopt matching samples in a mini-batch $B$ for Potential Neighboring Label Mining (PNLM) and Progressive Feature Tendency Determination (PFTD). In Figure 3(b), both phases retain the identical structure in the auxiliary branches and each pair of corresponding branches shares identical parameters.

Feature split

The two-phase module performs different processes to learn and predict the PFT. That is, PNLM adopts adjacent-class samples while PFTD adopts the corresponding ones. Formally, for a given sample $(X_{m}, y_{m})$ , all possible features of different branches matching either phase are denoted as follows:
$u_{m}^{(j_{q})} = z_{m}^{(j_{q})}, j_{q} \in {\tilde{Q}}_{C} (y_{m}),$
(6\rm a)
$u_{m}^{ft} = z_{m}^{(y_{m})},$
(6\rm b)
$v_{m} = z_{m}^{(C)},$
(6\rm c)
where (1) $u_{m}^{(j_{q})}$ and $u_{m}^{ft}$ denote the feature matching PNLM and PFTD phases, respectively; (2) $v_{m}$ denotes the features in the main branch.

The corresponding probability vectors of the features are expressed as follows:
$p_{m}^{aux (j_{q})} = softmax (Ψ_{j_{q}} u_{m}^{(j_{q})}), j_{q} \in {\tilde{Q}}_{C} (y_{m}),$
(7\rm a)

$p_{m}^{f t} = softmax (\tilde{Ψ} u_{m}^{f t}),$
(7\rm b)
$p_{m}^{main} = softmax (Φ v_{m}),$
(7\rm c)
where $p_{m}^{aux (j_{q})}$ , $p_{m}^{ft} \in R^{2}$ are the probabilities of the PNLM and PFTD phases, respectively, and $p_{m}^{main} \in R^{C}$ is the probability in the main branch.

Potential neighbouring label mining

PNLM utilizes the auxiliary branches to mitigate the latent ambiguity under PPLN, by inputting each sample to two branches corresponding to its GPNLs based on its given label.

Branch input

The underlying goal of PNLM is to train C high-accuracy binary classifiers (BCFs). Each $B C F_{j}$ recognizes the two potential neighbouring labels in its GPNL set ${\tilde{Q}}_{C} (j)$ by adopting those samples whose given labels belong to ${\tilde{Q}}_{C} (j)$ . For instance, $(X_{m}, y_{m} = 1)$ is adopted into Branches 0 and 2 because these two integers belong to ${\tilde{Q}}_{C} (1)$ (as shown in Figure 3). In turn, Branch 1 takes those samples labelled with 0 and 2 as its input and learns the implicit pattern of these two classes.

PFT mining

In this phase, each branch learns PFT from its adjacent-class samples, as Branch 1 mines the critical discrimination between classes 0 and 2. This is based on the fact that the features of pneumoconiosis CXRs develops along with the progression of the disease. Figure 4 depicts that Branch 1 observes a relatively clear tendency showing whether a sample it adopts may convey mild or severe feature at the standpoint of Stage 1 of pneumoconiosis. By standpoint, we mean that, from a certain perspective, each branch acts as if it were a dedicated and strict expert that discriminates the samples to be assigned the PFT in either direction. After several iterations, all the auxiliary branches may learn specialized class-conditional knowledge.

Figure 4.
Demonstration of progressive feature tendency with standpoint at Stage 1. Correspondingly, Branch 1 is abstracted from a virtual expert observing milder features of Stage-0 samples and severer features of Stage-2 samples.

Ideally, to acquire the specialized knowledge as precisely as possible, all C classifiers should be well-trained to distinguish the samples of two adjacent classes by regarding the output as a confidence score measuring the PFT. And the observed PFT should strictly point to the two adjacent classes. However, several special cases remain to be discussed and addressed under the setting of PPLN, as illustrated in Figure 5.

Figure 5.
Ideal and actual cases that all possible auxiliary branches may meet.

Case 1: PPLN in given labels

The existence of PPLN implies that the given labels in the dataset are probably noisy. Two possible sub-cases are considered as illustrated in Figure 5. Branch $j^{'}$ is a well-trained BCF only if all the given labels of samples are correct. Nonetheless, the fact is that this branch still works from the perspective of PFT. Once either given label is incorrect, say $y_{p} = j^{'} - 1$ , its potential neighbouring labels are $j^{'}$ and $j^{'} - 2$ . For one thing, the samples with $j^{'} - 2$ as the potential true label, despite the given label $y_{p} = j^{'} - 1$ , have overall similarity to those with milder-direction labels than $j^{'}$ (i.e., samples with labels whose index numbers are less than $j^{'}$ ). The overall similarity reflects the PFT in the mild direction. For another, the samples with $j^{'}$ as the potential true label contain such ambiguous features that they are prone to being confused with those labelled as $y_{p} = j^{'} - 1$ . Even though their potential true label should be $j^{'}$ , these samples may have milder features as a result of the confusion. This explains the principle of PFT towards the mildness. A corresponding conclusion of the PFT towards the severity can be drawn when $y_{p} = j^{'} + 1$ holds.

Case 2: Margin classes

Apparently, if the label $y_{m}$ of sample $(X_{m}, y_{m})$ were in ${0, C - 1}$ , it should have had one and only PNL intuitively. We refer to these two classes as margin classes. Based on this perception, we define the GPNL set to handle margin classes. For example, the GPNL set of class 0 is denoted as ${\tilde{Q}}_{C} (0) = {0, 1}$ . Obviously, by regarding class 0 itself as its own GPNL, ${\tilde{Q}}_{C} (0)$ not merely keeps the cardinality equal to 2, but retains the structure of Branch 0 identical to other auxiliary branches as well. In addition, the margin branch can also make sense in terms of feature tendency. As shown in Figure 5, a sample $(X_{p}, y_{p} = 0)$ tends to have the mildest features among all training samples. And a sample $(X_{q}, y_{q} = 1)$ has a tendency to be similar to those samples with severer feature (not the severest but at least medium). Considering the existence of PPLN as in Case 1, the feature tendency towards medium severity holds as well. In Case 2, we denote the tendency as marginal feature tendency (Figure 6).

Figure 6.
Demonstration of special marginal feature tendency with standpoint at Stage 0.

It turns out that all the auxiliary branches can learn PFT at their respective standpoints in spite of the two special cases with the goal of robustness against PPLN. For the model to learn consistent PFT in this phase, we propose Branch-Consistent Focal Loss (BCFL) as
$L_{aux} = \frac{1}{C} \sum_{j = 0}^{C - 1} L_{aux}^{(j)},$
(8\rm a)
$L_{aux}^{(j)} = \frac{1}{N_{j}} \sum_{m = 1}^{N_{j}} \sum_{k \in {\tilde{Q}}_{C} (y_{m})} Focal (p_{m, k}^{main}, p_{m, k}^{aux (j)})$
(8\rm b)
In equation 8a, $L_{aux}^{(j)}$ is the Focal Loss for the j-th auxiliary branch. And in equation 8b, (1) $N_{j}$ is the number of samples adopted into the j-th branch in a mini-batch $B$ ; (2) as k represents the index in label space $Y$ , for a sample $(X_{m}, y_{m})$ , $p_{m, k}^{main}$ is the probability of $X_{m}$ belonging to the k-th class in the main branch, and $p_{m, k}^{{aux}_{j}}$ is the probability of $X_{m}$ belonging to the k-th class predicted by the j-th branch. The specific form of Focal Loss joining two probability distribution is inspired by Lin et al.⁴⁰ and defined as
$\begin{aligned} Focal (p_{a}, p_{b}) \\ = - α \cdot (p_{a} (1 - p_{b})^{γ} \log p_{b} + (1 - p_{a}) p_{b}^{γ} \log (1 - p_{b})), \end{aligned}$
(9)
where $p_{a}, p_{b}$ are two probability distributions, and $α$ and $γ$ are the parameters in Focal Loss.

In addition, BCFL plays an essential role in managing occasional sample imbalance in a mini-batch, which may cause a certain auxiliary branch to learn biased tendency. That is, as in Branch $j^{'}$ illustrated in Figure 7, the number of samples labelled as either class may be much greater than the other class. Thus, BCFL can enable each auxiliary branch to learn the balanced feature tendency, allowing it working as a dedicated PFT evaluator for the next phase.

Figure 7.
Demonstration of occasional sample imbalance in certain mini-batches.

Progressive feature tendency determination

The main task of all auxiliary branches in PNLM is to learn the PFT in either direction at their respective standpoints. In this phase, building upon the knowledge in the previous phase, each branch continues to serve as a BCF to evaluate the latent PFT of the samples.

Branch I/O

The auxiliary branches in PFTD retain the identical structure and parameters to those in PNLM, although they may adopt samples differently. Unlike PNLM, each Branch j in PFTD takes the samples in the form of $(X_{m}, y_{m} = j)$ (as in equation 6b) to determine their latent tendency to either mild or severe with $B C F_{j}$ , as illustrated in Figure 8. Referring to the cases in Figure 5, each Branch j in this phase outputs a two-component prediction vector, as equation 7b suggests.

Figure 8.
Demonstration of progressive feature tendency determination in a certain auxiliary branch.

Based on PFT discussed above, either component represents the confidence score evaluating the tendency in the corresponding direction of a sample. In fact, the direction of tendency is equivalent to the possible direction of PPLN, depicting the probability distribution of the potential neighbouring true labels. The two components directly manifest the degree of PPLN as a direct metric.

Tendency regularization

For its flexibility, PFTD appends one additional operation called Sharpen as regularization following some previous work.^31,41 Sharpen is formulated as
$Sharpen {(p_{m}^{ft}, T)}_{k} = \frac{{(p_{m, k}^{ft})}^{1 / T}}{\sum_{i \in {\tilde{Q}}_{C} (y_{m})} {(p_{m, i}^{ft})}^{1 / T}}, k \in {\tilde{Q}}_{C} (y_{m}),$
(10)
where T is the temperature, a fixed parameter determining to what extent the probability components are adjusted to be sharper or more flatten. And the final latent PFT ${\tilde{p}}_{m}$ is obtained by
${\tilde{p}}_{m, k} = Sharpen {(p_{m}^{ft}, T)}_{k}, k \in {\tilde{Q}}_{C} (y_{m})$
(11)
Subsequently, ${\tilde{p}}_{m}$ will be used in sample selection for calculating LT-Diff.

Sample selection based on dual dynamic thresholds

Sample selection picks potentially true-labelled samples to mitigate the memorization effect of DNNs. To investigate additional evidence for measuring the memorization strength of model, we propose the Latent Tendency Difference (LT-Diff) based on PFT to clarify another potential intrinsic instance-dependent factor related to label noise. With expectation of memorizing the labels with more essential image features, DISC assumes that DNN’s memorization strength for one instance as confidence score of a given class, and that DIST, dynamically incrementing, is utilized to evaluate the strength.¹⁴ To improve the sample selection strategy, we introduce the LT-Diff as a metric whose value is expected to decrement dynamically. Moreover, the probability-based and difference-based dynamic thresholds are combined to form the dual-threshold sample selection.

Dynamic instance-specific latent tendency difference

Latent tendency difference

Formally, the LT-Diff for sample $(X_{m}, y_{m})$ is defined as
${diff}_{m}^{LT} = | {\tilde{p}}_{m, k_{1}} - {\tilde{p}}_{m, k_{2}} |, k_{1}, k_{2} \in {\tilde{Q}}_{C} (y_{m})$
(12)
Apparently, there are only two elements, $k_{1}$ and $k_{2}$ , in ${\tilde{Q}}_{C} (y_{m})$ , representing its two GPNs, respectively. The two corresponding components of ${\tilde{p}}_{m, k_{1}}, {\tilde{p}}_{m, k_{2}}$ of ${\tilde{p}}_{m}$ represent and directly reflect the predicted feature tendencies in both directions. Provided that both are regarded as of equal significance, their absolute difference may indicate PFT as an indirect metric, further reflecting the strength of memorizing PPLN for the auxiliary branch. Intuitively, such memorization of label noise should be minimized as much as possible. Based on this rationale, we introduce the dynamic difference-based threshold.

Dual dynamic thresholds

The probability-based and difference-based thresholds are obtained by epoch-wise iteration as
$τ (t_{e}) = λ τ (t_{e} - 1) + (1 - λ) p (t_{e}), τ (0) = 1 / N,$
(13\rm a)
$\tilde{τ} (t_{e}) = \tilde{λ} \tilde{τ} (t_{e} - 1) + (1 - \tilde{λ}) {diff}^{LT}, \tilde{τ} (0) = {\tilde{τ}}_{0},$
(13\rm b)
where $t_{e}$ is the epoch index, $λ$ and $\tilde{λ}$ are the layback ratios controlling the delaying degree and threshold stability, and $p_{m} (t_{e}) = max_{j} p_{m, j}^{main}$ . The probability-based threshold follows DIST in DISC¹⁴ as in equation 13a. Likewise, we refer to the difference-based threshold in equation 13b as dynamic LT-Diff threshold. Despite its decreasing tendency, its form of iteration is consistent with the probability-based threshold. Transforming equation 13 yields
$τ (t_{e}) - τ (t_{e} - 1) = (1 - λ) (p (t_{e}) - τ (t_{e} - 1)),$
(14\rm a)

$\tilde{τ} (t_{e}) - \tilde{τ} (t_{e} - 1) = (1 - \tilde{λ}) ({diff}^{LT} - \tilde{τ} (t_{e} - 1)),$
(14\rm b)
which suggests that there should be positive correlation between the epoch-wise variation of the thresholds and the corresponding difference by subtracting the last-epoch threshold from the probability or the LT-Diff.

Double layback-ratio trick

Here we introduce a small trick on the layback ratio based on a very common mathematical principle. That is, given two numbers with a fixed sum, their difference changes by 2 units when either increases or decreases by 1 unit. In the calculation of LT-Diff, such a property does naturally exist, which we consider incorporating in the threshold iteration. This can be simply adapted by doubling its variation factor $1 - \tilde{λ}$ related to probability-based threshold. Then, it should hold that $1 - \tilde{λ} = 2 (1 - λ)$ , and $\tilde{λ}$ satisfies
$\tilde{λ} = 2 λ - 1,$
(15)
which is hereby referred to as double layback-ratio trick (DLRT), setting a stricter criterion and helping release the memorization effect to PPLN further.

Sample selection

Additional difference-based criterion

In sample selection, all training samples are assessed and selected into two sets: clean and hard. For sample $(X_{m}, y_{m})$ , the criteria are based on the probability $p_{m}^{main}$ (as defined in equation 7c) from the main branch and the LT-Diff ${diff}_{m}^{LT}$ . If (1) its probability is greater than $τ (t_{e})$ and (2) its LT-Diff is less than $\tilde{τ} (t_{e})$ , the sample can be partitioned into the clean set $C$ , formulated as
$\begin{aligned} C & = {(X_{m}, y_{m}) | {diff}_{m}^{LT} < \tilde{τ} (t_{e})} \\ \cap {(X_{m}, y_{m}) | p_{m, y_{m}}^{main} > τ (t_{e})} \end{aligned}$
(16)
This indicates that the samples selected in $C$ not merely contain essential clean features for the main branch to learn, but restrict the memorization strength of the auxiliary branches to PPLN as well. If either of the two conditions above holds exclusively, the sample can be partitioned into the hard set, formulated as
$\begin{aligned} H & = {(X_{m}, y_{m}) | {diff}_{m}^{LT} < \tilde{τ} (t_{e})} \\ \cup {(X_{m}, y_{m}) | p_{m, y_{m}}^{main} > τ (t_{e})} - C, \end{aligned}$
(17)
which may contain the confusing samples at the decision boundaries as shown in Figure 2. These samples embody the inconsistency between dual criteria and should be tackled in particular to enhance the generalization capability of the model.

Penalty terms for LT-Diff

Subsequently, the samples in $C$ and $H$ are handled differently by loss functions, formulated as
$L_{C} = - \frac{1}{N} \sum_{i = 1}^{N_{c}} (\log p_{i, y_{i}}^{main} + δ_{1} \cdot \log (1 - {diff}_{i}^{LT})),$
(18\rm a)
$L_{H} = \frac{1}{N} \sum_{i = 1}^{N_{h}} (\frac{1 - {(p_{i, y_{i}}^{main})}^{q}}{q} + δ_{2} \cdot \frac{1 - {(1 - {diff}_{i}^{LT})}^{q}}{q}),$
(18\rm b)
where $N_{c}$ , $N_{h}$ ,and N denote the sizes of $C$ , $H$ , and the entire dataset, respectively. Here, N acts as an overall scalar regarded as the weighted learning rate.¹⁴ In equation 18, $δ_{1}$ and $δ_{2}$ denote the coefficients of two LT-Diff-based penalty terms $\log (1 - {diff}_{i}^{LT})$ and $(1 - (1 - {diff}_{i}^{LT})^{q}) / q$ , similar to the terms in Cross Entropy (CE) loss and Generalized Cross Entropy (GCE) loss.⁴² As LT-Diff is supposed to change inversely against probability, $(1 - {diff}_{i}^{LT})$ retains the unified form shared with probability, enforcing the values of LT-Diff to decrease for optimization.

Overall loss function

The previous sections about our proposed method are based on the perspective that PPLN is caused by both class-conditional and instance-dependent factors. Accordingly, the overall objective is adapted as
$L = {\begin{matrix} ω_{1} (t_{e}) L_{aux} & + & ω_{2} (t_{e}) L_{CE} & , t_{e} \leq t_{w a r m}, \\ ω_{1} (t_{e}) L_{aux} & + & ω_{2} (t_{e}) (L_{C} + ζ L_{H}) & , t_{e} > t_{w a r m}, \end{matrix}$
(19)
where (1) $L_{CE}$ is the standard CE loss; (2) $ζ$ is a coefficient for the trade-off between clean and hard sets; (3) $ω_{1}$ and $ω_{2}$ are the weighted ramp functions^41,43 with respect to epoch index $t_{e}$ . Here, the ramp functions are formulated as
$\begin{aligned} ω_{1} (t_{e}) = {\begin{matrix} 1 & , t_{e} \leq t_{w a r m}, \\ \exp (- φ_{1} {(1 - \frac{t_{w a r m}}{t_{e}})}^{2}) & , t_{e} > t_{w a r m}, \end{matrix} \end{aligned}$
(20\rm a)
$\begin{aligned} ω_{2} (t_{e}) = {\begin{matrix} \exp (- φ_{2} {(1 - \frac{t_{e}}{t_{w a r m}})}^{2}) & , t_{e} \leq t_{w a r m}, \\ 1 & , t_{e} > t_{w a r m}, \end{matrix} \end{aligned}$
(20\rm b)
where $φ_{1}$ and $φ_{2}$ are two positive integers and $ω_{1} (t_{e})$ , $ω_{2} (t_{e})$ ramp down and up as Gaussian curves, respectively. In equations 19 and 20, warm-up is introduced in the context of ramp functions and $t_{w a r m}$ is the index threshold for warm-up epochs.

The benefits of the above setup are twofold. (1) Warm-up is a procedure in which all samples in the dataset are trained with the standard Cross-Entropy loss for a few epochs, and it is prone to be influenced by class-conditional label noise. We believe that, working as virtual specialized experts, the auxiliary branches in FTLTD-Net tend to learn the specific class-conditional knowledge in PPLN, which is correlated with the underlying IIN factors. Moreover, unlike the confidence penalty³¹ and the uncertainty assessment,⁸ focusing on the knowledge in the auxiliary branches is a reasonable way of fitting the latent distribution pattern of PPLN. Hence, in warm-up procedure, $ω_{1} (t_{e})$ keeps constant while $ω_{2} (t_{e})$ ramps up till $t_{e} = t_{w a r m}$ . (2) When the model basically acquires the simpler class-conditional pattern of PPLN, the more sophisticated instance-specific knowledge in PPLN should be learned through sample selection based on the dual dynamic thresholds. And the focus may be shifted to loss functions for clean and hard sets. Similarly, in the subsequent epochs, $ω_{1} (t_{e})$ ramps down to the end of training while $ω_{2} (t_{e})$ keeps constant.

Experiment

Dataset

The CXRs of pneumoconiosis used in this study were collected from a hospital in Shanxi Province, China. In practice, training with label noise ensures that the test set must be true-labelled. Therefore, three experienced radiologists, who have been working on pneumoconiosis for more than 10 years, participated in this study to select the clean (true-labelled) CXR samples. To select true-labelled CXRs, the radiologists reviewed the diagnostic conclusions of all related patients through the corresponding radiological reports in detail. The annual physical examination records of every individual patient have been taken into consideration because the critical time points of stage determination are essential for label correctness. Based on the historical diagnostic information, the radiologists re-assess the corresponding CXR films to exclude imprecise staging conclusions. And the CXRs are selected into the test set as agreed upon by all three radiologists. In summary, a total of 2014 CXRs are involved in this study, as listed in Table 1.

Table 1.
Structure of the pneumoconiosis CXR dataset and its split into train and test sets.

Train Test Total

Stage 0 740 183 923

Stage 1 591 146 737

Stage 2 182 44 226

Stage 3 104 24 128

Total 1617 397 2014

Implementation details

Preprocessing

Originally, all the pneumoconiosis CXR data are 16-bit grey-scale images in the shape of irregular rectangles and contain text-based personal information about names, dates of examination and so forth, which is less relevant to radiological features of the disease and may distract the model’s attention. We first conduct anonymization to avoid the model to memorize non-radiological feature, by erasing printed text-based information. Then all images are centre-cropped to regular squares and resized to $512 \times 512$ for model input. Before training, data augmentation is applied to the resized images, including random rotation in the range of $[- 15^{\circ}, 15^{\circ}]$ and random horizontal flipping. We performed only simple augmentation techniques, as complex edits on images can be harmful to medical images. All training samples are normalized to 32-bit floats complying with a mean of 0.5281 and a standard deviation of 0.2294, based on statistical results from the dataset.

Synthetic label noise

We regard that there exists the intrinsic PPLN, denoted as Original in the context, in the real-world pneumoconiosis CXR dataset, with its exact latent noise rate remaining unclear. Besides, to demonstrate the effectiveness of the proposed model, we also synthesize the enhanced PPLN considering both the class-conditional and instance-dependent factors by mixing IDN and asymmetric IIN with a given noise rate $ρ$ . First, based on the part-dependent label noise,¹⁹ the IDN is generated with the rate $ρ$ for each instance following a truncated Gaussian distribution. The noisy label for each sample is chosen randomly according to the calculated probabilities. Next, the IDN-based noisy label is applied with asymmetric label noise, by assigning either of its two GPNs, as defined in Section “Problem formulation”, evenly with a probability of $ρ / 2$ , to obtain the final noisy label.

Experiment setup

All experiments are conducted in the runtime environment of Python 3.8 and PyTorch 1.11.0 on a PC, which is equipped with Ubuntu 20.04 as the operating system, an Intel Xeon Silver 4210R CPU, and two NVIDIA RTX A5000 GPUs, each with 24 GB memory. Here, we list the hyperparameters in general. The whole training process is iterated for 200 epochs, following most methods for learning from noisy labels. We show that sufficient iteration can motivate the dynamical properties of the thresholds (see Section “Ablation study”). The Adam optimizer with a weight decay of 0.0001 is employed with an initial learning rate 0.001, which is divided by 10 at epoch 45 and 80. The batch size is set to 16. Other hyperparameters involved in Section “Method” are as follows. For the Focal Loss in auxiliary branches, $α$ and $γ$ are set to $1.0$ and $2.0$ , respectively. And the Sharpen temperature T is $2.0$ . For initialization of dynamic LT-diff threshold, $\tilde{τ} (0)$ is initialized as ${\tilde{τ}}_{0} = 1.0$ . For double layback-ratio trick in equation 15, we have $λ = 0.99$ and $\tilde{λ} = 2 λ - 1 = 0.98$ . The coefficients $δ_{1}$ and $δ_{2}$ for LT-Diff penalty terms are both set to $1.5$ . In GCE Loss for hard set (equation 18b), q is set to $0.7$ .^14,42 The number of warm-up epochs is determined by the epoch index $t_{w a r m} = 15$ . Finally, $φ_{1}$ and $φ_{2}$ are set to 5 and 1, respectively, to control the rates of ramping up and down.

Comparative study

We compare our proposed method with several state-of-the-art (SOTA) methods mentioned in Section “Related work”. For fair results, all the comparative experiments in this study are equipped with ResNet18³⁹ as backbone. These SOTA methods are implemented using their released code and their key features are briefly described as follows.
Base indicates the backbone model without any extra technique against label noise. Here it represents the vanilla ResNet18.

Co-teaching²⁸ leverages the mutual information from two parallel networks and selects the high-confidence samples by ranking the losses output from the other network. Only those samples with smaller losses will be kept as clean samples for backpropagation.

Co-teaching+³⁰ introduces a progressive advancement from Co-teaching by incorporating the “disagreement update” strategy to effectively sustain disagreement between the two networks.

DivideMix³¹ trains two networks using a co-divide strategy. One network fits to a Gaussian Mixture Model to divide the dataset into clean and hard sets, which are then used to train the other.

SELC³⁴ utilizes the ensemble predictions formed by an exponential moving average of network outputs to progressively enhance the targets in loss function.

DMUE⁴¹ incorporates several extra branches in the network for latent distribution mining of potential labels, originally utilized for facial expression recognition to eliminate the ambiguity. Unlike our proposed method, each of the branches may excavate all other possible classes (not just neighbouring classes as in our FTLTD-Net).

DISC¹⁴ considers two views of data augmentation, and conducts sample selection and correction based on DIST, computed from confidence scores of two views.
Based on the setting of synthetic PPLN, in Table 2 are listed the results of all comparative methods under different PPLN noise rates. In general, our FTLTD-Net outperforms the SOTA methods in almost all metrics, though DISC has trivial efficiency in terms of F1 score and AUC for original PPLN. And all compared methods degrade to varying extents in the involved metrics as the PPLN rate increases. Other methods, not adapted to PPLN, are relatively more sensitive to the rise of PPLN rate. Specifically, these classical methods dealing with label noise show symbolic effectiveness of the basic training strategies. Co-teaching, Co-teaching+ and DivideMix all employ double networks and small-loss trick and outperform the Base. Encountering PPLN, disagreement update in Co-teaching and co-divide in DivideMix works better than Co-teaching even though their efficiency reduces when PPLN rate goes large. And co-divide makes more sense in dealing with PPLN than both Co-teaching and Co-teaching+. Self ensemble in SELC can handle low-rate PPLN, but the performance degrades as PPLN rate reaches 30%. One possible interpretation may be the accumulation of much more confusing probabilities as the training epoch progresses. DMUE has a similar architecture involving multiple auxiliary branches, but performs worse than our FTLTD-Net in all PPLN rates. We believe that the directionality of PFT helps eliminate the ambiguity between neighbouring classes, while DMUE adopts all other possible latent classes in each auxiliary branch, lacking the directional relevance. Unlike facial expression recognition, CXRs have more intensive inter-class similarity, such as similar positions of lung outline and similar range of grey-scale intensity, because all the images convey radiological features in the lung fields owing to the penetrating ability of X-ray. The binary property of bidirectional PFT can release the influence of inter-class similarity to some extent. And we also believe that the dual dynamic thresholds assist in the selection of trusted samples for a robust model against PPLN. As conducting pure dynamic probability-based sample selection, DISC mildly pales in comparison to our FTLTD-Net. It may make sense that the LT-Diff based dynamic thresholds can have a better efficiency in discriminating CXR samples with adjacent-class confusion in terms of PPLN.

Table 2.
Results (%) of comparative study between the state-of-the-art methods and our proposed FTLTD-Net.

PPLN rate Original 10%

Method ACC PRE SEN F1 AUC ACC PRE SEN F1 AUC

Base 85.01 81.75 78.71 79.63 93.53 82.38 75.01 70.55 71.62 91.44

Co-teaching²⁸ 86.47 83.75 80.63 81.58 93.61 84.35 76.86 72.29 73.38 91.51

Co-teaching+³⁰ 87.82 83.84 80.72 81.66 94.01 85.66 78.09 73.46 74.57 92.88

DivideMix³¹ 89.07 82.12 79.06 79.99 92.31 86.83 79.24 74.54 75.67 91.17

SELC³⁴ 89.47 84.02 80.89 81.84 94.32 87.27 79.61 74.89 76.02 92.58

DMUE⁴¹ 86.24 81.17 78.51 79.28 93.95 83.58 76.98 72.33 73.44 90.32

DISC¹⁴ 90.24 83.93 81.05 82.44 95.93 87.95 79.99 74.93 77.28 92.84

FTLTD-Net 90.92 84.25 81.11 82.06 94.64 89.08 81.31 76.49 77.64 93.47

PPLN rate 20% 30%

Method ACC PRE SEN F1 AUC ACC PRE SEN F1 AUC

Base 81.35 72.17 64.96 67.89 89.59 79.03 68.29 64.93 66.19 87.92

Co-teaching²⁸ 83.29 73.90 66.51 69.51 89.63 80.52 69.92 66.48 67.77 87.05

Co-teaching+³⁰ 84.58 75.05 67.54 70.59 91.00 81.89 71.01 67.52 68.82 88.40

DivideMix³¹ 85.79 76.12 68.51 71.60 90.27 83.38 72.02 68.48 69.81 89.66

SELC³⁴ 86.18 76.46 68.82 71.92 90.68 83.75 72.34 68.79 70.12 90.07

DMUE⁴¹ 82.49 73.01 68.27 70.93 89.43 80.09 70.35 67.14 68.96 88.83

DISC¹⁴ 86.03 77.36 69.62 72.49 91.18 83.96 71.87 67.94 69.37 90.82

FTLTD-Net 87.56 78.04 70.24 73.41 92.55 85.49 73.84 70.21 71.57 91.93

The experiments are conducted under different label noise (PPLN) rates. Original means the intrinsic real-world PPLN in pneumoconiosis dataset while other percentages indicate the synthetic PPLN.

Compared with the above methods, our proposed FTLTD-Net has slower degradation as PPLN rate increases. We employ confusion matrices to demonstrate the robustness of FTLTD-Net across various stages of pneumoconiosis, as depicted in Figure 9. Stages 0 and 3 are less susceptible to lower rates of PPLN while the robustness for Stages 1 and 2 decreases relatively more profound as PPLN rate increases. The model may not identify a sample as a non-adjacent class, which avoids unreasonable classification. It turns out that the model maintains stable robustness across stages of pneumoconiosis despite the increasing noise rate.

Figure 9.
Confusion matrices of classification results of our proposed FTLTD-Net under different PPLN rates.

Ablation study

To verify the effectiveness of each key component and critical parameters, we conduct a series of ablation studies, evaluating all possible combinations of key components and analyzing how the model performance changes with varying critical hyperparameters.

Component analysis

We conduct experiments for all possible combinations of essential components proposed in this paper, including BCFL, DLRT, and LT-Diff penalty terms (PTs). The first two are inappropriate to be removed directly from the model but can be replaced with their degraded forms. A degraded form of BCFL can be obtained by making auxiliary branch predictions subjective to given labels, weakening the consistency to the main branch. And the mode of iteration of LT-Diff thresholds can be degraded to be the same as probability-based thresholds, with $\tilde{λ} = λ$ . The statistics of module combinations are summarized in Table 3 under different PPLN rates.

Table 3.
Statistics for component analysis.

PPLN rate Original 10%

BCFL DLRT PT ACC (%) F1 (%) AUC (%) ACC (%) F1 (%) AUC (%)

84.31 75.32 90.79 82.74 73.32 89.60

✓ 86.41 77.43 92.08 83.95 75.83 91.42

✓ 86.59 77.36 92.34 84.34 75.99 91.95

✓ 88.93 78.27 92.69 86.65 76.19 92.66

✓ ✓ 87.03 79.94 92.54 84.73 76.81 92.48

✓ ✓ 88.76 80.45 93.25 86.96 77.41 93.07

✓ ✓ 90.39 81.73 93.68 88.14 78.24 93.23

✓ ✓ ✓ 90.92 82.06 94.64 89.08 77.64 93.47

PPLN rate 20% 30%

BCFL DLRT PT ACC (%) F1 (%) AUC (%) ACC (%) F1 (%) AUC (%)

80.86 68.47 87.22 79.05 65.19 86.25

✓ 81.83 71.26 89.53 80.19 68.27 89.06

✓ 82.24 71.64 90.33 80.32 68.84 90.12

✓ 84.68 72.13 91.39 83.07 69.60 91.54

✓ ✓ 82.92 72.69 91.18 81.23 70.11 91.29

✓ ✓ 85.27 73.15 91.63 83.31 70.46 91.63

✓ ✓ 86.38 73.71 91.53 84.85 70.79 91.27

✓ ✓ ✓ 87.96 73.41 92.55 85.49 71.57 91.93

Degraded BCFL is obtained by making auxiliary branch predictions subjective to given labels, and degraded DLRT is obtained with $\tilde{λ} = λ$ .

The results indicate that all three modules contribute respectively to improving model performance, combined to obtain the best. Besides tackling mini-batch level data imbalance, BCFL helps minimize the deviation between auxiliary and main branches and ensures the features extracted from main and auxiliary branches to preserve consistency. Inspired by how the values of LT-Diff change, DLRT directly affects the strictness of sample selecting criterion about LT-Diff, which makes the difference-based thresholds iterate at a relatively faster rate. In contrast, their degraded forms lack such capabilities. It turns out that branch-wise consistency of parameters and a stricter sample selection criterion account for better statistics even if the effect is less pronounced when either or both of the two modules are integrated in the model. In addition, the decrease of LT-Diff thresholds should be attributed to the PTs, without which the probability-based thresholds are the only ones serving to partition the samples. For one thing, combining PTs with BCFL positively influences probability-based sample selection with stronger branch-wise consistency. For another, DLRT might enhance the effect that PTs conduct to the model with a stricter criterion for difference-based sample selection.

To understand why LT-Diff thresholds are significant to model performance, we depict the variation of LT-Diff threshold distributions across epochs under different PPLN rates using histograms, as shown in Figure 10. In the moving distribution of the histogram group lies typical phenomena that the distribution descends to low values under all PPLN rates and manifests two centres after several epochs, and that the model trained with higher PPLN tends to obtain denser LT-Diff distribution at lower thresholds. From a perspective of LT-Diff, whether a sample is clean depends locally on how its value changes between neighbouring epochs. If LT-Diff is temporarily less than the threshold, it will be regarded as clean at this epoch. Globally, the number of times for which the sample is regard as clean reflects the trend for its threshold to decrease. Hence, through all training epochs, the samples near the low-threshold centre are more likely to be clean while those near the higher-threshold centre are to be hard. The remaining uncertainty arises from the combined selection strategy and the probability-based thresholds.

Figure 10.
Histograms of LT-Diff threshold distribution along with epochs. Three rows corresponds to PPLN rates 10%, 20% and 30%, respectively. The ratios indicate the proportion of samples within corresponding instance-specific threshold intervals.

Hyperparameter analysis

We select several critical hyperparameters to evaluate their impact on model performance. The candidate values are chosen with intervals based on prior research and empirical testing to avoid excessively increasing the computational burden. Since FTLTD-Net is a sample selection method, the chosen hyperparameters are regarding dynamic thresholds involved and loss function values. The ratios of clean and hard samples to the whole dataset reflect model’s determination on sample reliability, and the stability of their epoch-wise variation demonstrates the impact of hyperparameter on model performance and consistency.

Effect of temperature T

The result statistics for several candidate values of T are listed in Table 4. Figure 11 shows how the overall LT-Diff threshold level influences the ratios of clean and hard samples in training set. The Sharpen operation in equation 10 smooths the probabilities of PFT. A higher T reduces the gap between two PFT directions, resulting in lower LT-Diff values. When the thresholds are relatively fixed, the model tends to regard the samples to be clean. In contrast, a lower T results in the likelihood of samples identified as hard ones. An appropriate T helps maintain ratios of clean and hard samples stable in the latter epochs, enabling the model to consistently partition samples with high confidence.

Figure 11.
Line charts show how the average of LT-Diff thresholds, the ratio of clean samples, and the ratio of hard samples change with different Sharpen temperatures as the number of epochs increases.

Table 4.
Accuracy, F1 score, and AUC with different sharpen temperatures.

Temperature T ACC (%) F1 (%) AUC (%)

0.5 87.69 78.69 93.18

1.0 88.43 79.52 93.30

1.5 89.83 80.63 93.26

2.0 90.92 82.06 94.64

2.5 88.76 81.07 93.85

Effect of layback ratios $λ, \tilde{λ}$

The model performance after introducing DLRT is shown in Section “Component analysis”. And Table 5 lists how the evaluation metrics change as $λ$ varies mildly. As equation 14 suggests, the actual iterative factors are $(1 - λ)$ and $(1 - \tilde{λ})$ . Too large iterative factors (with too small layback ratios simultaneously) may accelerate the thresholds to converge, cause non-smooth epoch-wise iterations and resulting in non-stable clean and hard sample ratios at the latter epochs, as depicted in Figure 12. Conversely, if the iterative factors are too small, the model is prone to be misled, as a looser criterion may result in regarding less reliable samples as clean, although the epoch-wise variation of thresholds is more stable. Therefore, moderate layback ratios are beneficial to model performance.

Figure 12.
Line charts show how the average of LT-Diff thresholds, the ratio of clean samples, and the ratio of hard samples change with different layback ratios as the number of epochs increases.

Table 5.
Accuracy, F1 score, and AUC with different layback ratios.

Layback ratios ACC (%) F1 (%) AUC (%)

$λ$ $\tilde{λ}$

0.999 0.998 89.36 81.13 93.71

0.995 0.990 89.65 82.14 94.32

0.990 0.980 90.92 82.06 94.64

0.985 0.970 88.24 80.73 94.52

0.980 0.960 86.55 79.77 94.34

Adversarial effect of temperature T and layback ratios $λ, \tilde{λ}$

The two groups of hyperparameters affect the determination of the model on sample reliability in different respects. Temperature T directly influences the scale of LT-Diff, which is the evaluation results about the class-conditional factor of PPLN. On the other hand, layback ratios $λ, \tilde{λ}$ control the delaying degree and stability of dynamic thresholds, which are the criteria of sample reliability considering the instance-dependent factor of PPLN. The optimal values of these hyperparameters are the outcome of confrontation between evaluation and determination at respective modules.

Effect of PT coefficients $δ_{1}, δ_{2}$

Table 6 presents the statistics and Figure 13 depicts loss curves under different PT coefficients. The results indicate that the model is more optimal when PTs take relatively larger weights ( $δ_{1} = δ_{2} = 1.5$ ) than original CE or GCE loss terms (with a fixed coefficient 1) in equation 18. These coefficients makes the model converge mildly faster after about 75 epochs. These two coefficients directly determine the proportion how LT-Diff values affect the optimization of model parameters. It turns out from the results that a higher proportion of PTs, relative to confidence-based original CE or GCE, can accelerate the model to converge. This provides further evidence that the idea of difference-based dual threshold sample selection makes sense in the specific label noise type, and that the role that PTs play is relatively more substantial comparing to that of normal loss function paradigm.

Figure 13.
Loss curves with different PT coefficients.

Table 6.
Accuracy, F1 score, and AUC with different PT coefficients.

PT coefs $δ_{1}, δ_{2}$ ACC (%) F1 (%) AUC (%)

0.5 84.88 78.04 93.95

0.8 85.98 78.34 94.32

1.0 86.93 79.41 93.92

1.2 88.06 80.91 93.41

1.5 90.92 82.06 94.64

1.8 88.39 81.42 94.77

2.0 87.47 80.68 94.25

Limitations and future work

Despite the mildly superior performance of our proposed FTLTD-Net dealing with PPLN in pneumoconiosis CXR dataset, there are some limitations in terms of generalizability. One main concern is data imbalance that Stage-0 and Stage-1 samples are dominant while the rest are relatively rare, covering both mini-batch and dataset levels. The FTLTD-Net is more susceptible to the impact of data imbalance at the mini-batch level on account of the PFT learning and predicting paradigm in the auxiliary branches, whereas it relatively ignores the biases from the dataset level, which can cause the model to underperform on underrepresented stages. The other is about different data distribution. It is undeniable that the dataset used in this paper may possibly be distributed quite differently compared with real-world ones. We must acknowledge the potential sampling bias, as the data predominantly comes from a specific hospital, which could limit the model’s applicability to other populations with different demographics or clinical settings. In addition, data imbalance may limit generalizability of the model to real-world scenarios where data distribution may vary. Therefore, techniques such as resampling and domain adaptation should be utilized in the future work to ensure data balance and sampling fairness when it comes to observing the model performance on different datasets. Besides, the special label noise formalized in this paper theoretically occurs in other scenarios, where the underlying feature changes continuously but the criteria of classification are not clear enough. A limitation of this study lies in as well the difficulty in identifying suitable scenarios to transfer our proposed method to other applications. Therefore, future work is supposed to focus on exploring other reasonable scenarios where PPLN could be applicable.

Conclusion

In this paper, we propose a novel multi-branch architecture named FTLTD-Net for pneumoconiosis diagnosis, which considers a special label noise type referred to as PPLN. The FTLTD-Net incorporates a two-phase auxiliary module to learn and predict the underlying PFT to mitigate the impact of PPLN. And it employs a special metric LT-Diff to obtain a difference-based threshold for better dynamic sample selection of trusted samples. Extensive experiments demonstrate that, as the PPLN rate increases, our proposed methods outperforms the SOTA methods for its lower sensitivity to PPLN, and that the proposed modules consistently improve performance across all evaluation metrics. However, we acknowledge several limitations in our approach that data imbalance and sampling bias from the dataset’s hospital-specific origin may affect the model’s generalizability. Future work will focus on applying resampling and domain adaptation techniques to address data imbalance, as well as exploring the transferability of our approach to other scenarios involving PPLN.

	Train	Test	Total
Stage 0	740	183	923
Stage 1	591	146	737
Stage 2	182	44	226
Stage 3	104	24	128
Total	1617	397	2014

PPLN rate	Original	10%
Method	ACC	PRE	SEN	F1	AUC	ACC	PRE	SEN	F1	AUC
Base	85.01	81.75	78.71	79.63	93.53	82.38	75.01	70.55	71.62	91.44
Co-teaching²⁸	86.47	83.75	80.63	81.58	93.61	84.35	76.86	72.29	73.38	91.51
Co-teaching+³⁰	87.82	83.84	80.72	81.66	94.01	85.66	78.09	73.46	74.57	92.88
DivideMix³¹	89.07	82.12	79.06	79.99	92.31	86.83	79.24	74.54	75.67	91.17
SELC³⁴	89.47	84.02	80.89	81.84	94.32	87.27	79.61	74.89	76.02	92.58
DMUE⁴¹	86.24	81.17	78.51	79.28	93.95	83.58	76.98	72.33	73.44	90.32
DISC¹⁴	90.24	83.93	81.05	82.44	95.93	87.95	79.99	74.93	77.28	92.84
FTLTD-Net	90.92	84.25	81.11	82.06	94.64	89.08	81.31	76.49	77.64	93.47

PPLN rate	20%	30%
Method	ACC	PRE	SEN	F1	AUC	ACC	PRE	SEN	F1	AUC
Base	81.35	72.17	64.96	67.89	89.59	79.03	68.29	64.93	66.19	87.92
Co-teaching²⁸	83.29	73.90	66.51	69.51	89.63	80.52	69.92	66.48	67.77	87.05
Co-teaching+³⁰	84.58	75.05	67.54	70.59	91.00	81.89	71.01	67.52	68.82	88.40
DivideMix³¹	85.79	76.12	68.51	71.60	90.27	83.38	72.02	68.48	69.81	89.66
SELC³⁴	86.18	76.46	68.82	71.92	90.68	83.75	72.34	68.79	70.12	90.07
DMUE⁴¹	82.49	73.01	68.27	70.93	89.43	80.09	70.35	67.14	68.96	88.83
DISC¹⁴	86.03	77.36	69.62	72.49	91.18	83.96	71.87	67.94	69.37	90.82
FTLTD-Net	87.56	78.04	70.24	73.41	92.55	85.49	73.84	70.21	71.57	91.93

PPLN rate	Original	10%
BCFL	DLRT	PT	ACC (%)	F1 (%)	AUC (%)	ACC (%)	F1 (%)	AUC (%)
			84.31	75.32	90.79	82.74	73.32	89.60
✓			86.41	77.43	92.08	83.95	75.83	91.42
	✓		86.59	77.36	92.34	84.34	75.99	91.95
		✓	88.93	78.27	92.69	86.65	76.19	92.66
✓	✓		87.03	79.94	92.54	84.73	76.81	92.48
✓		✓	88.76	80.45	93.25	86.96	77.41	93.07
	✓	✓	90.39	81.73	93.68	88.14	78.24	93.23
✓	✓	✓	90.92	82.06	94.64	89.08	77.64	93.47

PPLN rate	20%	30%
BCFL	DLRT	PT	ACC (%)	F1 (%)	AUC (%)	ACC (%)	F1 (%)	AUC (%)
			80.86	68.47	87.22	79.05	65.19	86.25
✓			81.83	71.26	89.53	80.19	68.27	89.06
	✓		82.24	71.64	90.33	80.32	68.84	90.12
		✓	84.68	72.13	91.39	83.07	69.60	91.54
✓	✓		82.92	72.69	91.18	81.23	70.11	91.29
✓		✓	85.27	73.15	91.63	83.31	70.46	91.63
	✓	✓	86.38	73.71	91.53	84.85	70.79	91.27
✓	✓	✓	87.96	73.41	92.55	85.49	71.57	91.93

Temperature T	ACC (%)	F1 (%)	AUC (%)
0.5	87.69	78.69	93.18
1.0	88.43	79.52	93.30
1.5	89.83	80.63	93.26
2.0	90.92	82.06	94.64
2.5	88.76	81.07	93.85

Layback ratios	ACC (%)	F1 (%)	AUC (%)
0.999	0.998	89.36	81.13	93.71
0.995	0.990	89.65	82.14	94.32
0.990	0.980	90.92	82.06	94.64
0.985	0.970	88.24	80.73	94.52
0.980	0.960	86.55	79.77	94.34

PT coefs $δ_{1}, δ_{2}$	ACC (%)	F1 (%)	AUC (%)
0.5	84.88	78.04	93.95
0.8	85.98	78.34	94.32
1.0	86.93	79.41	93.92
1.2	88.06	80.91	93.41
1.5	90.92	82.06	94.64
1.8	88.39	81.42	94.77
2.0	87.47	80.68	94.25

Footnotes

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by National Natural Science Foundation of China [grant numbers 62376183, U21A20469]; Funding Project of Local Science and Technology Development Guided by Central Government of China [grant number YDZJSX2022C004]; Open Project of National Health Commission Key Laboratory of Pneumoconiosis, China [grant number YKFKT004]; Special Fund for Science and Technology Innovation Teams of Shanxi Province, China [grant number 202304051001009]; and the Non-profit Central Research Institute Fund of Chinese Academy of Medical Science [No. 2020-PT320-005].

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

X-M

Luo

Song

M-Y

, et al. Pneumoconiosis: current status and future prospects. Chin Med J 2021; 134: 898–907.

Wang

Zhang

, et al. Prevention and treatment of pneumoconiosis in the context of healthy China 2030. China CDC Wkly 2023; 5: 927–932.

Okumura

Kawashita

Ishida

. Computerized classification of pneumoconiosis on digital chest radiography artificial neural network with three stages. J Digit Imaging 2017; 30: 413–426.

Zheng

Deng

Jin

, et al. An improved CNN-based pneumoconiosis diagnosis method on X-ray chest film. In: Milošević

Tang

(eds) Human centered computing. Cham: Springer International Publishing, 2019, pp.647–658.

Devnath

Luo

Summons

, et al. An accurate black lung detection using transfer learning based on deep neural networks. In: 2019 International conference on image and vision computing New Zealand (IVCNZ) , 2019, pp.1–6.

Devnath

Luo

Summons

, et al. Automated detection of pneumoconiosis with multilevel deep features learned from chest X-ray radiographs. Comput Biol Med 2021; 129: 104125.

Wang

Zhu

, et al. Potential of deep learning in assessing pneumoconiosis depicted on digital chest radiography. Occup Environ Med 2020; 77: 597–602.

Wang

, et al. Improving medical images classification with label noise using dual-uncertainty estimation. IEEE Trans Med Imaging 2022; 41: 1533–1546.

Sun

Luo

, et al. expertNet: Defeat noisy labels by deep Expert consultation paradigm for pneumoconiosis staging on chest radiographs. Expert Syst Appl 2023; 232: 120710.

10.

Liao

Xie

, et al. Modeling annotator preference and stochastic annotation error for medical image segmentation. Med Image Anal 2024; 92: 103028.

11.

Liang

Luo

, et al. Ambiguity-aware breast tumor cellularity estimation via self-ensemble label distribution learning. Med Image Anal 2023; 90: 102944.

12.

Sun

Luo

, et al. A fully deep learning paradigm for pneumoconiosis staging on chest radiographs. IEEE J Biomed Health Inform 2022; 26: 5154–5164.

13.

Gao

B-B

Xing

Xie

C-W

, et al. Deep label distribution learning with label ambiguity. IEEE Trans Image Process 2017; 26: 2825–2838.

14.

Han

Shan

, et al. DISC: learning from noisy labels via dynamic instance-specific selection and correction. In: 2023 IEEE/CVF conference on computer vision and pattern recognition (CVPR). Vancouver, BC, Canada: IEEE, 2023, pp.24070–24079.

15.

Song

Kim

Park

, et al. Learning from noisy labels with deep neural networks: A survey. IEEE Trans Neural Netw Learn Syst 2022; 34: 8135–8153.

16.

Xiao

Xia

Yang

, et al. Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp.2691–2699.

17.

Goldberger

Ben-Reuven

. Training deep neural-networks using a noise adaptation layer. In: International conference on learning representations, 2016.

18.

Patrini

Rozza

Krishna Menon

, et al. Making deep neural networks robust to label noise: a loss correction approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp.1944–1952.

19.

Xia

Liu

Han

, et al. Part-dependent label noise: towards instance-dependent label noise. In: Advances in neural information processing systems, volume 33. Curran Associates, Inc., 2020, pp.7597–7610.

20.

Cheng

Zhu

, et al. Learning with instance-dependent label noise: a sample sieve approach, 2021.

21.

Zhu

Liu

. A second-order approach to learning with instance-dependent label noise. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp.10113–10123.

22.

Garg

Nguyen

Felix

, et al. Instance-dependent noisy label learning via graphical modelling. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision , 2023, pp.2288–2298.

23.

Yang

, et al. A parametrical model for instance-dependent label noise. IEEE Trans Pattern Anal Mach Intell 2023; 45: 14055–14068.

24.

Zhang

Cao

Yang

, et al. Cognition-driven structural prior for instance-dependent label transition matrix estimation. IEEE Trans Neural Netw Learn Syst 2024; 36: 3730–3743.

25.

Yao

Yang

Han

, et al. Searching to exploit memorization effect in learning with noisy labels. In: Proceedings of the 37th international conference on machine learning . PMLR, 2020, pp.10789–10798.

26.

Yang

Yao

Han

, et al. Searching to exploit memorization effect in deep learning with noisy labels. IEEE Trans Pattern Anal Mach Intell 2024; 46: 7833–7849.

27.

Malach

Shalev-Shwartz

. Decoupling “When to Update” from “How to Update”. In: Advances in neural information processing systems, volume 30. Curran Associates, Inc., 2017.

28.

Han

Yao

, et al. Co-teaching: robust training of deep neural networks with extremely noisy labels. In: Advances in neural information processing systems, volume 31. Curran Associates, Inc., 2018.

29.

Jiang

Zhou

Leung

, et al. MentorNet: learning data-driven curriculum for very deep neural networks on corrupted labels. In: Proceedings of the 35th international conference on machine learning . PMLR, 2018, pp.2304–2313.

30.

Han

Yao

, et al. How does disagreement help generalization against label corruption? In: Proceedings of the 36th international conference on machine learning . PMLR, 2019, pp.7164–7173.

31.

Socher

Hoi

SCH

. DivideMix: learning with noisy labels as semi-supervised learning, 2020.

32.

Berthelot

Carlini

Goodfellow

, et al. MixMatch: a holistic approach to semi-supervised learning. In: Advances in neural information processing systems, volume 32. Curran Associates, Inc., 2019.

33.

Lyu

Tsang

. Curriculum loss: robust learning and generalization against label corruption, 2020.

34.

. SELC: self-ensemble label correction improves learning with noisy labels. In: Proceedings of the thirty-first international joint conference on artificial intelligence . Vienna, Austria: International Joint Conferences on Artificial Intelligence Organization, 2022, pp.3278–3284.

35.

Wang

Cui

, et al. Robust stochastic neural ensemble learning with noisy labels for thoracic disease classification. IEEE Trans Med Imaging 2024; 43: 2180–2190.

36.

Xiong

Hoi

SCH

. Learning from noisy data with robust representation learning. In: Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp.9485–9494.

37.

Liu

, et al. InstanT: semi-supervised learning with instance-dependent thresholds. Adv Neural Inf Process Syst 2023; 36: 2922–2938.

38.

Zhou

, et al. A time-consistency curriculum for learning from instance-dependent noisy labels. IEEE Trans Pattern Anal Mach Intell 2024; 46: 4830–4842.

39.

Zhang

Ren

, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp.770–778.

40.

Lin

T-Y

Goyal

Girshick

, et al. Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision , 2017, pp.2980–2988.

41.

She

Shi

, et al. Dive into ambiguity: latent distribution mining and pairwise uncertainty estimation for facial expression recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp.6248–6257.

42.

Zhang

Sabuncu

. Generalized cross entropy loss for training deep neural networks with noisy labels. In: Advances in neural information processing systems, volume 31. Curran Associates, Inc., 2018.

43.

Laine

Aila

. Temporal ensembling for semi-supervised learning, 2017.

Layback ratios		ACC (%)	F1 (%)	AUC (%)
$λ$	$\tilde{λ}$	ACC (%)	F1 (%)	AUC (%)
0.999	0.998	89.36	81.13	93.71
0.995	0.990	89.65	82.14	94.32
0.990	0.980	90.92	82.06	94.64
0.985	0.970	88.24	80.73	94.52
0.980	0.960	86.55	79.77	94.34

Dual-threshold sample selection with latent tendency difference for label-noise-robust pneumoconiosis staging

Abstract

Background

Objective

Methods

Results

Conclusions

Keywords

Introduction

Related work

Noise-type-oriented mechanism

Sample selection

Method

Problem formulation

Objective

Progressive pair label noise

Potential neighbouring label disambiguation

Module input

Feature split

Potential neighbouring label mining

Branch input

PFT mining

Case 1: PPLN in given labels

Case 2: Margin classes

Progressive feature tendency determination

Branch I/O

Tendency regularization

Sample selection based on dual dynamic thresholds

Dynamic instance-specific latent tendency difference

Latent tendency difference

Dual dynamic thresholds

Double layback-ratio trick

Sample selection

Additional difference-based criterion

Penalty terms for LT-Diff

Overall loss function

Experiment

Dataset

Implementation details

Preprocessing

Synthetic label noise

Experiment setup

Comparative study

Ablation study

Component analysis

Hyperparameter analysis

Effect of temperature T

Effect of layback ratios λ , λ ~

Adversarial effect of temperature T and layback ratios λ , λ ~

Effect of PT coefficients δ 1 , δ 2

Limitations and future work

Conclusion

Footnotes

Funding

Declaration of conflicting interests

References

Effect of layback ratios $λ, \tilde{λ}$

Adversarial effect of temperature T and layback ratios $λ, \tilde{λ}$

Effect of PT coefficients $δ_{1}, δ_{2}$