A multimodal similarity-aware and knowledge-driven pre-training approach for reliable pneumoconiosis diagnosis

Abstract

Background

Pneumoconiosis staging is challenging due to the low clarity of X-ray images and the small, diffuse nature of the lesions. Additionally, the scarcity of annotated data makes it difficult to develop accurate staging models. Although clinical text reports provide valuable contextual information, existing works primarily focus on designing multimodal image-text contrastive learning tasks, neglecting the high similarity of pneumoconiosis imaging representations. This results in inadequate extraction of fine-grained multimodal information and underutilization of domain knowledge, limiting their application in medical tasks.

Objective

The study aims to address the limitations of current multimodal methods by proposing a new approach that improves the precision of pneumoconiosis diagnosis and staging through enhanced fine-grained learning and better utilization of domain knowledge.

Methods

The proposed Multimodal Similarity-aware and Knowledge-driven Pre-Training (MSK-PT) approach involves two stages. In the first stage, we deeply analyze the similar features of pneumoconiosis images and use a similarity-aware modality alignment strategy to explore the fine-grained representations and associated disturbances of pneumoconiosis lesions between images and texts, guiding the model to match more appropriate feature representations. In the second stage, we utilize data-associated features and pre-stored domain knowledge features as priors and constraints to guide the downstream model in the visual domain without annotations. To address potential erroneous labels generated by model predictions, we further introduce an uncertainty threshold strategy to mitigate the negative impact of imperfect prediction labels and enhance model interpretability.

Results

We collected and created the pneumoconiosis chest X-ray (PneumoCXR) dataset to evaluate our proposed MSK-PT method. The experimental results show that our method achieved a classification accuracy of 81.73%, outperforming the state-of-the-art algorithms by 2.53%.

Conclusions

MSK-PT showed diagnostic performance that matches or exceeds the average radiologist's level, even with limited labeled data, highlighting the method's effectiveness and robustness.

Keywords

pneumoconiosis diagnosis multimodal pre-training chest X-ray similarity-aware learning

Introduction

Pneumoconiosis is currently one of the most serious and common irreversible occupational diseases in China.¹ It is caused by the long-term inhalation of dust, leading primarily to lung fibrosis. As the disease progresses, the lung tissue hardens and calcifies, ultimately resulting in extreme difficulty in breathing, restricted mobility, loss of labor capacity, and eventual death due to respiratory failure.² According to the “Diagnostic Criteria for Occupational Pneumoconiosis” (GBZ 70-2015),³ the staging of pneumoconiosis follows the guidelines of the International Labour Organization (ILO),⁴ where chest X-ray (CXR) images of patients are compared with standard reference images³ to classify the disease into four stages: Stage-0 (normal), Stage-1, Stage-2, and Stage-3, based on the extent and severity of the lesions. This diagnostic process is complex and highly subjective, requiring radiologists to have extensive knowledge and experience, which makes the annotation of pneumoconiosis images costly and time-consuming. Although the use of automated pneumoconiosis assessment methods, such as computer-aided diagnosis (CAD), has significantly increased in recent years, the inherent low clarity of X-ray images and the high inter-class similarity of pneumoconiosis imaging features continue to pose challenges for accurate diagnosis. Therefore, developing an objective and reliable intelligent diagnostic technique is of great clinical significance for the accurate diagnosis of pneumoconiosis.

In recent years, the rapid development of deep learning technology has driven revolutionary progress in the field of computer vision and has been widely applied in medical image analysis.^5–8 However, most existing methods^9,10 primarily focus on using image data for the diagnosis of pneumoconiosis, overlooking the expert textual annotations that accompany the images. In medical image diagnosis, textual reports can provide rich semantic information and serve as important auxiliary signals, especially in accurately identifying complex disease categories. However, the fine-grained relationships between pneumoconiosis images and their corresponding textual reports remain unclear, as illustrated in Figure 1. This may be due to two main reasons: (1) the degree and progression of lesions in pneumoconiosis patients vary at different stages, resulting in minimal differences in lesion characteristics between similar images;³ (2) most descriptions in the textual reports pertain only to specific sub-regions of the corresponding medical images.¹¹ This implies that much of the information may be irrelevant to our analysis, necessitating more granular recognition. Traditional contrastive learning methods^12,13 struggle to accurately capture fine-grained representations between modalities, thus limiting the effective utilization of medical image and text data. Consequently, achieving joint training of images and texts remains a highly challenging task.

Figure 1.

Illustration of the similarity between images and clinical text reports. Examples are from samples in the PneumoCXR dataset.

Considering the integration of image-text semantic dependency relationships, multimodal pre-training models provide a promising solution. These models not only focus on the features of similar data themselves but also explore dependencies between different instances, aiming for improved performance in classification tasks. In general image classification, Li et al.¹⁴ proposed an unsupervised domain hint distillation framework designed to transfer knowledge from a larger teacher model to a lightweight target model by leveraging hints from unlabeled domain images. Bica et al.¹⁵ introduced a method for pre-training fine-grained multimodal representations from image-text pairs, enhancing performance in image-level and region-level tasks. Similarly, in medical image diagnosis, Monajatipoor et al.¹⁶ introduced BERTHop, utilizing a pre-trained language encoder (BlueBERT) to address domain gap issues and capture correlations between two modalities more effectively. Moon et al.¹⁷ employed a BERT-based architecture combined with a novel multimodal attention masking scheme to maximize generalization performance in visual-language understanding tasks such as diagnostic classification, medical image report retrieval, medical visual question answering, and radiology report generation. Liu et al.¹⁸ proposed a method for better contrastive learning in medical visual-language pre-training by categorizing medical image-text pairs into positive, negative, and neutral groups, thereby facilitating the construction of more suitable contrastive losses for continual improvement in cross-modal retrieval and image classification tasks. You et al.¹⁹ investigated a modal-shared contrastive language-image pre-training framework to optimize parameter sharing in the transform model, promoting the transfer of common semantic structures between language and vision.

The aforementioned studies demonstrate that researchers have explored various robust tasks using multimodal models and proposed a series of methods to mitigate classification errors caused by relying solely on individual instances. However, in scenarios with low paired distinctiveness, existing multimodal methods primarily focus on capturing global information from each modality, neglecting the importance of cross-modal perception of local information. Consequently, these models exhibit limited capability in fine-grained classification of similar structures and lack the ability to effectively comprehend fine-grained details of input data, which restricts their performance in tasks that require a more nuanced understanding. To address these limitations, researchers have started exploring techniques that enable finer alignment and comprehension. Peng et al.²⁰ achieved fine-grained alignment and comprehension by generating contextually consistent precise location markers, associating text descriptions with specific visual elements in images. You et al.²¹ adopted a hybrid region representation approach that combines discrete coordinates with continuous features, capturing spatial relationships in images more accurately and facilitating precise alignment between language and visual content. Additionally, Chen et al.²² introduced an extra location-aware module to enhance the understanding of local information. By integrating information at the local level, these models demonstrated superior performance in region- or object-level tasks that require precise multimodal understanding.

While the aforementioned approaches provide insights into fine-grained understanding within general image modalities, they are constrained by benchmark tasks within these modalities. In the field of pneumoconiosis imaging, lesion areas exhibit diffuse interstitial fibrosis features, posing challenges for existing models to achieve superior performance. Therefore, there is a need for further exploration in fine-grained understanding of pneumoconiosis images. Addressing the challenge posed by low discriminative and high similarity characteristics, there is a crucial necessity to construct effective multimodal, multi-granular information understanding models for images and text. This entails achieving deeper analysis and quantification. Moreover, modeling the correlation between visual representations and domain knowledge is essential to enhance the capacity for capturing underlying discriminative features, which presents challenges in model design. Thus, resolving these technical issues is the focal point and challenge of this study.

In this study, we propose a novel Multimodal Similarity-aware and Knowledge-driven Pre-Training (MSK-PT) approach，This method correlates unstructured data from different stages of pneumoconiosis to explore fine-grained representations and associated perturbations of pneumoconiosis lesions across text-image pairs. Specifically, we design a two-stage training strategy for MSK-PT. In the first stage, employing a multi-layer alignment strategy, our goal is to enable the model to grasp fine-grained information. Beyond capturing global information like other multimodal models, we align local information within input modalities to precisely identify and locate specific regions in images. In the second stage, we use data-correlated features and domain-specific medical knowledge as guiding signals for the model to fine-tune downstream tasks, completing inference under a fully visual modality. This approach provides more accurate and robust support for clinical pneumoconiosis diagnosis.

In brief, this work contributes to the field by:

We proposed a multi-modal similarity-aware learning and knowledge-driven pre-training method called MSK-PT, designed for robust pneumoconiosis staging diagnosis.

We employed a multi-level text-image alignment strategy to understand both coarse and fine-grained multi-modal representations, while developing a similarity-aware modality alignment module to capture the consistency and differences between similar imaging features, thereby exploring local representations more effectively.

We incorporated data-associated features and domain knowledge as priors and constraints in the model to enhance the training and inference of downstream visual tasks. Additionally, by introducing an uncertainty threshold, we effectively mitigated the impact of erroneous labels on model performance and provided reliable visual cues for clinical decision-making through Gradient-weighted Class Activation Mapping (Grad-CAM).

We collected and constructed a pneumoconiosis chest X-ray (PneumoCXR) dataset for method development and extensive evaluation. Experimental results demonstrate that even with limited labeled data, the MSK-PT method outperforms existing learning methods.

The remainder of this paper is organized as follows: Section 2 summarizes and discusses the existing research on related technologies. Section 3 provides detailed information about the MSK-PT model. Section 4 describes the experimental setup. Section 5 presents the quantitative and qualitative evaluation of MSK-PT's performance compared to existing methods on internal test datasets and conducts extensive ablation studies. Section 6 discusses our limitations and future research work. Section 7 concludes the paper.

Related work

Image-text contrastive learning

Contrastive learning (CL), a self-supervised learning technique, has achieved significant advancements in the field of deep learning.^23–25 It focuses on the similarities and differences between data pairs, enhancing network representation learning by maximizing the mutual information between the input and its representation. In recent years, extensive research^26–28 has provided a diverse foundation for the development of contrastive learning, covering various models, loss functions, and pretext tasks. In computer vision, He et al.²³ treated contrastive learning as a dictionary look-up, using a dynamic dictionary with a queue and a momentum encoder. Chen et al.²⁴ employed random data augmentation, viewing multiple augmented versions of the same image as positive samples and other images as negative samples to optimize the contrastive loss. Grill et al.²⁹ introduced a slowly moving average of the target network output from the online network, effectively eliminating the need for negative samples. Chen et al.³⁰ dispensed with negative samples and the momentum encoder, relying on the same encoder and a stop-gradient mechanism to prevent output collapse. Caron et al.³¹ used vision transformers as the foundation for self-supervised learning, implementing self-distillation without any labels.

Additionally, the principles of contrastive learning are applied to multimodal representation learning, defining robust loss functions by contrasting positive and negative multimodal sample pairs.^32–34 Liu et al.³⁵ used contrastive loss for multimodal data representation learning, capturing complementary and synergistic interactions between modalities. Huang et al.³⁶ introduced a similarity aggregation strategy that utilizes signals from global and local representations for retrieval, jointly learning multimodal global and local representations of medical images by contrasting attention-weighted image regions with words in paired reports. Han et al.³⁷ preserved task-relevant information by maximizing mutual information between unimodal inputs. Song et al.³⁸ combined a distance metric function between label categories with curriculum learning, using a supervised prototype contrastive learning (SPCL) loss to address classification imbalance. Yang et al.³⁹ proposed a supervised cluster-level contrastive learning (SCCL) method, leveraging variance adaptive density (VAD) to overcome the inefficiency of high-dimensional supervised contrastive learning (SCL). Tu et al.⁴⁰ introduced a contrastive learning scenario between context and knowledge (CKCL), more efficiently utilizing contextual information and external knowledge. These studies use contrastive loss to interpret relationships between modalities, based on the principle of pulling an anchor and a positive sample together in the multimodal embedding space while pushing them away from many negative samples. However, the performance of downstream tasks is often limited by the constraints of self-supervised learning tasks due to the absence of data labels. To leverage label information, Khosla et al.⁴¹ proposed a supervised contrastive loss, extending self-supervised contrastive learning methods to fully supervised learning. This approach achieves success in visual representation learning⁴² and few-shot learning.⁴³

Therefore, researchers have utilized contrastive learning algorithms to explore a variety of tasks, effectively leveraging global and local multimodal information during training. However, in scenarios with low pairwise discriminability, existing contrastive learning methods exhibit limited capabilities in fine-grained classification of similar structures. This limitation is particularly evident in pneumoconiosis imaging, where diffuse interstitial fibrosis in lesion areas presents a significant challenge. In this study, we specifically designed a local similarity-aware modal alignment module to enhance the model's ability to comprehend fine-grained details in both images and text. By adopting the principles of contrastive learning, we establish effective connections between images and text, identify discriminative features of similar categories, and thereby enhance the accuracy and reliability of classification.

Knowledge-based medical pre-training

In recent years, advancements in artificial intelligence have yielded promising results across various applications. Through representation learning on image-text datasets,^44,45 a series of visual-language pretraining models^14,15 have been developed, with significant focus in the field of medical visual-language pretraining (Med-VLP).^16–19 Recent research has emphasized enhancing representation learning by integrating domain-specific medical knowledge. Zhang et al.⁴⁶ introduced knowledge-augmented diagnosis (KAD), leveraging existing medical domain knowledge to guide the use of paired CXR images and radiology reports. Chen et al.⁴⁷ proposed a knowledge-based learning framework for identifying unlabeled biomedical microscopy images through self-supervised pretraining, emphasizing optimizations between encoders and decoders critical for initializing high-quality segmentation decoders. Chen et al.⁴⁸ introduced the knowledge-enhanced contrastive visual-language pretraining (KoBo) framework, integrating clinical knowledge to enhance semantic consistency learning in visual-language tasks, addressing challenges of semantic overlap and transfer. Chen et al.⁴⁹ proposed a systematic approach to strengthen medical visual-language pretraining by integrating structured medical knowledge, aligned representations, knowledge-injected fusion models, and designing knowledge-induced pseudo tasks. Pan et al.⁵⁰ fused knowledge graph-based contrastive pretraining methods to enhance alignment and reasoning capabilities of visual and language representations. These studies underscore the potential of integrating domain-specific knowledge into medical contrastive visual-language pretraining to improve performance on downstream tasks, addressing challenges such as limited medical data and the need for more effective representation learning.

The learning of these models revolves around encoding images and text into crucial representations necessary for downstream tasks, enabling them to perform various zero-shot prediction tasks not specifically trained on by the models. These studies collectively highlight the potential of visual-language pretraining in enhancing performance across various tasks. They have demonstrated outstanding performance in diverse downstream tasks such as medical object detection,⁵¹ image classification,⁵² and semantic segmentation.⁵³ However, they have yet to effectively explore different granularities of visual representations and often rely on partial semantic information. To address these limitations, Li et al.⁵⁴ employed a knowledge-guided contrastive framework to capture multi-granular semantic information. It accurately aligns the pathology of each image with corresponding medical terminologies, thereby enhancing the model's performance in downstream tasks.

In addition to the aforementioned efforts, maintaining consistency between the knowledge features of training classes and the image features is crucial. During training, these knowledge features should progressively align with the image features. However, if the representations of a few images from specific classes diverge from those of the entire dataset, it may lead to overfitting on the semantic labels of specific samples. For example, in the context of pneumoconiosis image embeddings, there is a risk of overfitting to comorbidity information with similar clinical manifestations, even if such information is irrelevant to pneumoconiosis staging. To mitigate the overfitting of knowledge features for each class, our work focuses on translating data-associated features and domain knowledge into priors and constraints for the model. We conduct a deep analysis of similar pneumoconiosis images, employing a similarity-aware modal alignment strategy to capture both consistency and diversity among similar images. This approach enhances the model's ability to understand fine-grained details across different modalities, establishes semantic correlations between images and text, promotes feature diversity, and effectively trains the model to discern between similar samples. These efforts aim to serve downstream tasks in fully visual modalities effectively.

Method

Overall architecture

Due to the varying severity of pneumoconiosis lesions across different lung lobes and the diverse and complex nature of lesion characteristics, radiology reports can convey more comprehensive semantic information compared to discrete label values. This capability enhances diagnostic performance in downstream tasks, particularly where labeled data is limited. Therefore, this paper proposes a two-stage multimodal pre-training framework, MSK-PT as shown in Figure 2. In the first stage, we employ a multi-layer alignment strategy to construct an integrated model. Using a similarity-aware modal alignment module, we learn joint representations of domain knowledge and image features. This helps establish semantic correlations between textual and visual features, thereby enhancing the model's predictive capabilities. In the second stage, we utilize data-associated features and pre-trained knowledge to guide visual representation learning. By introducing an uncertainty threshold strategy to improve the model's generalization ability and robustness. Next, we will first describe the problem scenario considered, followed by detailed introductions to the training specifics of each stage.

Problem scenario

Recent literature⁵⁴ indicates that the performance of multimodal learning methods largely depends on the availability of large annotated datasets. However, obtaining accurately labeled medical images is both expensive and time-consuming in clinical practice. Therefore, an important research direction is how to effectively learn labeled text-image representations to achieve competitive performance. In this study, we first define a training set $B = {(x_{d, i}, x_{p, i})}_{i = 1, 2, \dots, T}$ of T text-image pairs using labeled domain data. We utilize two encoders: a text encoder $℮_{d}$ and an image encoder $℮_{p}$ . These encoders map $B$ to a text feature representation space $δ = {d_{i} | ℮_{d} (x_{d, i})}_{i = 1, 2, \dots, T}$ , an image feature representation space $ε = {p_{i} | ℮_{p} (x_{p, i})}_{i = 1, 2, \dots, T}$ , and a local text-image feature representation space $M = {(D_{i}, G_{i})}$ $_{i = 1, 2, \dots, T}$ . For the $i$ -th text-image pair $(x_{d, i}, x_{p, i})$ , the text encoder generates a global text representation $d_{i}$ and a series of encoded text tokens $D_{i} = {n_{i}^{1}, n_{i}^{2}, \dots, n_{i}^{L}}$ $\in R^{L \times d}$ , while the image encoder generates a global image representation $p_{i}$ and a series of encoded visual tokens $G_{i} = {m_{i}^{1}, m_{i}^{2}, \dots, m_{i}^{S}} \in R^{S \times d}$ ， $L$ denotes the length of the text tokens, and $S$ represents the total number of image tokens.

Knowledge Distillation. Inspired by literature⁵⁵, we transfer pre-trained knowledge features to downstream unlabeled visual tasks. Through knowledge distillation, the downstream tasks can learn and grasp domain-specific expertise, thereby performing better during deployment. Specifically, we employ the Kullback-Leibler (KL) divergence loss function to match the output distributions between the two stages, facilitating effective knowledge transfer and adaptation. The application of KL divergence loss is demonstrated as follows:

L_{k d} (ℓ^{p}, ℓ^{t_{w}}, ℓ^{t_{s}}, τ) = τ^{3} K L (σ (ℓ^{p} / τ), σ (ℓ^{t_{w}} / τ), σ (ℓ^{t_{s}} / τ))

(1)

Where

ℓ^{p}

represents the logits predictions from the pre-training stage,

ℓ^{t_{w}}

and

ℓ^{t_{s}}

denote the logits predictions from two branches in the downstream task learning stage,

σ (\cdot)

denotes the softmax function, and

τ

is the temperature parameter. By incorporating these domain-specific knowledge features, the downstream task model's understanding and reasoning capabilities are enhanced, enabling more precise alignment and representation learning of unlabeled medical images. It is noteworthy that the KL divergence loss function smooths the probability distribution by adjusting the temperature parameter

τ

, allowing the model to better capture the structured knowledge from the pre-training stage. This approach not only improves the performance of downstream tasks but also enhances the model's robustness and generalization ability.

Global representation alignment (GRA)

To learn global text-image information, MSK-PT employs a global contrastive loss²⁴ operating at the level of global text $(d_{i})$ and global image $(p_{i}) .$ Specifically, we first use two projection layers ℶ_d and ℶ_p to map features $d_{i}$ and $p_{i}$ into a normalized shared feature space, obtaining $d_{i} * \in R^{d}$ and $p_{i} * \in R^{d}$ , respectively. Then, we compute the similarity between $d_{i} *$ and $p_{i} *$ using dot product. To enhance the effectiveness of these features, we apply self-attention mechanism (SA)⁵⁶ and layer normalization (LN)⁵⁷ on the features, defined by the following equations:

d_{i}^{*} = L N (S A {ℶ_{d} (d_{i})})

(2a)

p_{i}^{*} = L N (S A {ℶ_{p} (p_{i})})

(2b)

s i m {d_{i}^{*}, p_{i}^{*}} = d_{i}^{T} p_{i}^{*}

(2c)

We employ a text-image contrastive loss based on the InfoNCE loss to optimize this process, aiming to maximize the mutual information between correctly paired text-image pairs in the latent space. The InfoNCE loss function maximizes the similarity between the embeddings of corresponding images and texts while minimizing the similarity with other images and texts within the batch. This approach facilitates effective learning of text and image embeddings.

L_{T 2 I} (d_{i}, p_{i}) = - l o g (\frac{e x p (s i m (d_{i}^{*}, p_{i}^{*}) / τ_{1}}{\sum_{k = 1}^{O} e x p (s i m (d_{i}^{*}, p_{k}^{*}) / τ_{1}})

(3a)

L_{I 2 T} (p_{i}, d_{i}) = - l o g (\frac{e x p (s i m (p_{i}^{*}, d_{i}^{*}) / τ_{1}}{\sum_{k = 1}^{O} e x p (s i m (p_{i}^{*}, d_{k}^{*}) / τ_{1}})

(3b)

Where

O

represents the batch size, and

τ_{1}

denotes the temperature parameter. Finally, the overall loss for the GRA component is formulated as follows:

L_{G R A} = \frac{1}{2 T} \sum_{i = 1}^{T} (L_{T 2 I} + L_{I 2 T})

(4)

Similarity-aware modality alignment (SMA)

In pneumoconiosis images, variations in individual necessitate consideration of both intra-class differences within similar lesions and inter-class differences between dissimilar lesions. These variations are disease-specific but constitute a small proportion of the overall image. Furthermore, clinical text reports only accurately describe a few lesions related to the disease. Given these challenges in existing pneumoconiosis data, unlike traditional similarity alignment in contrastive learning, we introduce a Similarity-aware Modality Alignment (SMA) module. This module focuses on category-level rather than individual-level contrastive learning to accurately differentiate lesion types and identify subtle features of pneumoconiosis lesions. The module comprises two parts: Token-wise image-text alignment and Prototype-wise disease alignment, as illustrated in Figure 3.

Figure 3.

Similarity-aware modality alignment module. In this module, (a) denotes the cross-correlation matrix between text-image in TA; (b) and (c) stand for the representation space within text and image in TA; (d) denotes the prototype(disease)-level representation on text-image positive pairs in PA; (e) demonstrates the final aggregation similarity representation space on text and image produces by a dual alignment learning.

Token-wise Alignment (TA). We employ a token-wise alignment module based on a bidirectional cross-attention mechanism to explicitly match and align local text and image features using TA. Similar to global feature alignment, for the $i$ -th text-image pair, we first project the generated text and image token embeddings into a normalized low-dimensional embedding space, obtaining ${\tilde{D}}_{i} = {{\tilde{n}}_{i}^{1}, {\tilde{n}}_{i}^{2}, \dots, {\tilde{n}}_{i}^{L}} \in R^{L \times d}$ and ${\tilde{G}}_{i} = {{\tilde{m}}_{i}^{1}, {\tilde{m}}_{i}^{2}, \dots, {\tilde{m}}_{i}^{S}}$ $\in R^{S \times d}$ respectively. Then, we apply self-attention mechanism (SA) and layer normalization (LN) to each feature, as detailed below:

{\tilde{n}}_{i} = L N (S A {ℶ_{d} (n_{i})})

(5a)

{\tilde{m}}_{i} = L N (S A {ℶ_{p} (m_{i})})

(5b)

To achieve token alignment, we utilize a cross-attention mechanism⁵⁴ to compute soft matches between generated text tokens and image tokens. Specifically, for the

j

-th text token

{\tilde{n}}_{i}^{j}

in the i -th text-image pair, we interact

{\tilde{n}}_{i}^{j}

with all corresponding image tokens

{\tilde{m}}_{i}^{k}

to compute its cross-modal image embedding

r_{i}^{j}

. This allows text token

{\tilde{n}}_{i}^{j}

to better identify and align local lesion features, obtaining a more precise and comprehensive cross-modal representation. The process is represented by the following equation:

r_{i}^{j} = \sum_{k = 1}^{T} R (s o f t m a x (\frac{{(Q {\tilde{n}}_{i}^{j})}^{T} (K {\tilde{m}}_{i}^{k})}{\sqrt{d}}) (V {\tilde{m}}_{i}^{k}))

(6)

Where

Q \in R^{d \times d}, K \in R^{d \times d}, V \in R^{d \times d}

. To learn fine-grained information and accurately capture subtle differences between text and image, we propose a method based on token-wise fine-grained contrastive loss, operating at the level of each text-image pair. Specifically, we use a loss function

L_{T 2 I}^{ζ^{'}}

to minimize the distance between text token

{\tilde{n}}_{i}^{j}

and its cross-modal image embedding

r_{i}^{j}

, while maximizing the distance between

{\tilde{n}}_{i}^{j}

and other cross-modal image embeddings. Additionally, considering the varying importance of different text tokens, we assign a weight

w_{i}^{j}

to each token when calculating the loss. This approach not only effectively reduces computational and memory costs but also significantly enhances the model's ability to represent fine-grained details.

L_{T 2 I}^{ζ^{'}} = - \frac{1}{2 T} \sum_{i = 1}^{T} (\frac{1}{L} \sum_{j = 1}^{L} w_{i}^{j} (l o g \frac{e x p (s i m ({\tilde{n}}_{i}^{j}, r_{i}^{j}) / τ_{2})}{\sum_{k = 1}^{L} e x p (s i m ({\tilde{n}}_{i}^{j}, r_{i}^{k}) / τ_{2})} + l o g \frac{e x p (s i m (r_{i}^{j}, {\tilde{n}}_{i}^{j}) / τ_{2})}{\sum_{k = 1}^{L} e x p (s i m (r_{i}^{j}, {\tilde{n}}_{i}^{k}) / τ_{2})}))

(7)

Where

τ_{2}

is the token-wise temperature parameter. We utilize two symmetric InfoNCE losses similar to GRA to optimize the training process. Specifically, for the j -th image token

{\tilde{m}}_{i}^{j}

in the i -th text-image pair, we compute

{\tilde{m}}_{i}^{j}

with all corresponding text tokens

{\tilde{n}}_{i}^{k}

to obtain its cross-modal text embedding

{\ddot{r}}_{i}^{j}

. By comparing the similarity between

{\tilde{m}}_{i}^{j}

and

{\ddot{r}}_{i}^{j}

, we calculate the token-level image-text alignment loss

L_{I 2 T}^{ζ^{'}}

. Ultimately, the TA loss combines local text-to-image (T2I) and image-to-text (I2T) alignment losses jointly, as shown below:

L_{T A} = \frac{1}{2} (L_{T 2 I}^{ζ^{'}} + L_{I 2 T}^{ζ^{'}})

(8)

Prototype-wise Alignment (PA). In traditional alignment methods, samples from different instances are often treated as negative pairs. However, this approach can lead to pushing apart many samples with similar semantics in the embedding space, particularly problematic when dealing with closely related inter-class similarities in diseases like pneumoconiosis images. To address this issue and further enhance the model's fine-grained alignment capabilities in processing multimodal data, we introduce a local prototype-wise alignment module. This module establishes prototypes at the disease level to align similar disease samples, thereby accurately capturing disease pairs with subtle pathological features.

Specifically, for each text-image embedding pair $({\tilde{d}}_{i}, {\tilde{p}}_{i})$ , we first apply the iterative clustering algorithm Sinkhorn-Knopp.⁵⁸ This algorithm clusters the text and image embeddings separately, resulting in their respective clustering predictions $u_{d, i} \in R^{K}$ and $u_{p, i} \in R^{K}$ . Additionally, we introduce $K$ learnable prototype cluster centers $T = {t_{1}, \dots, t_{k}}$ to directly calculate the softmax probabilities of the text or image embeddings with each cluster center. This process helps establish the association between text-image embeddings and the cluster centers, thereby more accurately expressing the semantic relationships between text and image. The process is represented by the following equation:

g_{d, i}^{(k)} = \frac{e x p ({\tilde{d}}_{i}^{T} t_{k} / τ_{3})}{\sum_{k^{'}} e x p ({\tilde{d}}_{i}^{T} t_{k}^{^{'}} / τ_{3})}

(9a)

g_{p, i}^{(k)} = \frac{e x p ({\tilde{p}}_{i}^{T} t_{k} / τ_{3})}{\sum_{k^{'}} e x p ({\tilde{p}}_{i}^{T} t_{k}^{^{'}} / τ_{3})}

(9b)

Where

τ_{3}

is the prototype-wise temperature parameter, and

k

represents the

k

-th element of the vector. Cross-modal prototype-wise alignment is achieved through the following two cross-entropy losses for cross-modal prediction and optimization. We use

u_{p, i}

as “pseudo-labels” to train the text representation

d_{i}

, and

u_{d, i}

as “pseudo-labels” to train the image representation

p_{i}

, as detailed below:

h ({\tilde{d}}_{i}, u_{p, i}) = \sum_{k = 1}^{K} u_{p, i}^{(k)} l o g g_{d, i}^{(k)}

(10a)

h ({\tilde{p}}_{i}, u_{d, i}) = \sum_{k = 1}^{K} u_{d, i}^{(k)} l o g g_{p, i}^{(k)}

(10b)

Finally, to comprehensively consider the relationships between all texts and images, we define the PA loss by averaging the two prediction losses. This ensures that the model's training effectively optimizes multimodal alignment capabilities across the entire dataset. The PA loss is defined as follows:

L_{P A} = \frac{1}{2 T} \sum_{i = 1}^{T} (h ({\tilde{d}}_{i}, u_{p, i}) + h ({\tilde{p}}_{i}, u_{d, i}))

(11)

In summary, the primary objective of SMA is to optimize the model through two levels of loss functions: the token-level TA objective and the disease-level PA objective. These objectives aim to effectively capture local fine-grained feature embeddings between images and texts, enhancing their associations through a dual contrastive learning approach. As illustrated in Figure 3, in practical applications, we adjust the hyperparameters

α

and

β

and jointly optimize these loss functions to improve the model's performance and robustness in pneumoconiosis image recognition. The process is detailed as follows:

L_{S M A} = α L_{T A} + β L_{P A}

(12)

Training process

Our proposed MSK-PT method consists of two main stages: multimodal pre-training and downstream task learning, as shown in Figure 2. In this section, we will explain the training process for each stage in detail.

Figure 2.

MSK-PT approach overview. The approach is divided into two stages: (a) Pre-training stage, which utilizes weakly augmented labeled domain data to train the base task model through multi-layer alignment strategies. (b) Downstream task learning stage, where data-related features and pre-trained knowledge features are propagated to fully unlabeled visual samples, and diagnostics are performed by predicting their reliability. When weakly augmented unlabeled pneumoconiosis images are inputted, the model may assign a prediction score higher than the threshold θ, indicating unreliable predictions, prompting radiologists to reevaluate them. Conversely, for strongly augmented unlabeled pneumoconiosis images, if the predicted result falls below the threshold θ, it suggests a reliable diagnosis.

Stage I: Multi-modal pre-training learning. In the initial stage, our objective is to learn general medical image representations from clinical text reports to improve the performance of downstream pneumoconiosis image recognition tasks, particularly when labeled data is limited, as shown in Figure 2(a). We utilize a training set $B = {(x_{d, i}, x_{p, i})}_{i = 1, 2, \dots, T}$ containing T text-image pairs, and map $B$ to the global feature representation spaces $δ$ and $ε$ , representing text features and image features, respectively, using a text encoder ℮_d (e.g., BERT⁵⁹) and an image encoder ℮_p (e.g., ResNet-50⁶⁰). At the same time, we generate a local feature set in the text-image feature representation space $M$ .

Furthermore, during the pre-training stage, we update the model parameters by minimizing the cross-entropy loss between the predicted probability distribution $P (y | x)$ and the true labels $y$ . Once the text encoder is trained, its output features are fixed and no longer updated. In this setup, we preserve the trained text features $W = [d_{1}, d_{2}, \dots, d_{T}] \in R^{T \times d}$ as shared vectors for subsequent stages. This approach not only removes the reliance on the text branch in downstream tasks but also significantly reduces computational costs during training. At the same time, these shared vectors provide consistent text feature representations for downstream tasks, ensuring the model's stability and performance across different stages. This strategy effectively optimizes resource utilization, enabling more efficient training and inference in subsequent stages.

Stage II: Downstream task learning. In this stage, our goal is to leverage instance-related features and pre-trained knowledge to aid unsupervised visual representation learning for downstream tasks while maintaining consistency with pre-training results, as depicted in Figure 2(b). To achieve this, we adopt a strategy of reusing pre-trained features, avoiding additional computations for text features. Specifically, we employ trainable visual cues and feature projectors to train downstream weakly augmented image encoder $℮_{p}^{w}$ and strongly augmented image encoder $℮_{p}^{s}$ .When processing unlabeled datasets, images are initially inputted into both weakly and strongly augmented image branches to obtain normalized image features $p_{i}^{w} = M (℮_{p}^{w} (x)) / ‖ M (℮_{p}^{w} (x)) ‖_{2} \in R^{d}$ and $p_{i}^{s} = M$ $(℮_{p}^{s} (x)) / ‖ M (℮_{p}^{s} (x)) ‖_{2} \in R^{d}$ . To ensure precise alignment of features, we introduce a trainable mapping layer $M (\cdot)$ in the downstream image encoder branch, achieving feature size matching at minimal cost. Subsequently, we multiply the pre-stored text features $W \in R^{T \times d}$ with the downstream image features. Finally, using a uncertainty-based classifier, we compute prediction logits and evaluate prediction confidence based on threshold $θ$ .

Uncertainty threshold. Inspired by,⁶¹ we introduce uncertainty scores $U_{i}$ to evaluate the prediction confidence of each sample during both model training and evaluation. Specifically, the uncertainty score $U_{i}$ is computed based on the model's prediction probability distribution for each sample $i$ . For a given $i$ , the model generates a prediction probability vector $[p_{i 1}, p_{i 2}, \dots, p_{i k}]$ , where $p_{i k}$ represents the probabilities of sample $i$ belonging to each class $k$ . These probabilities are computed by the model's final layer (e.g., softmax layer), reflecting the model's confidence in assigning sample $i$ to different classes.

To quantify uncertainty, we employ a method based on the maximum predicted probability to calculate uncertainty scores $U_{i}$ :

U_{i} = 1 - m a x_{k} (p_{i k})

(13)

Where $m a x_{k} (p_{i k})$ is the maximum predicted probability among all classes for sample $i$ .If the model is confident in its prediction for sample $i$ (i.e., high maximum predicted probability), the uncertainty score $U_{i}$ will be low. Conversely, if the model's prediction confidence is low (i.e., low maximum predicted probability), the uncertainty score $U_{i}$ will be high. Furthermore, we compute the true error value $G_{i}$ of samples $i$ using the following formula:

G_{i} = 1 - 1 {P_{i}, y_{i}}

(14)

Where

P_{i}

represents the final predicted class by the model for sample

i

y_{i}

denotes the true class label of the sample, and

1 {P_{i}, y_{i}}

is an indicator function. When

P_{i}

equals

y_{i}

, its value is 1, indicating a correct prediction; otherwise, its value is 0, indicating an incorrect prediction.

By combining the uncertainty score $U_{i}$ and the true error value $G_{i}$ , we can assess the prediction confidence and accuracy of each sample and determine the optimal threshold on the validation dataset. Inspired by the Youden index,⁶² we compute the ROC curve for each validation set and analyze the true positive rate (TPR) and false positive rate (FPR) at different thresholds. By averaging the results from five-fold cross-validation (detailed in Section 4.2), we ensure the selected threshold maintains robustness and reliability across various data distributions. The objective function based on TPRs, FPRs, and threshold parameters $θ$ on the validation dataset is as follows:

J (θ) = 2 \times T P R s (θ) - F P R s (θ)

(15)

The final optimal threshold

θ

is calculated through

θ = a r g m a x_{θ} J (θ)

to mitigate potential negative impacts from imperfect predictions in downstream tasks. Predictions below

θ

are deemed potentially correct, indicating high confidence in the diagnostic outcome. Conversely, predictions above

θ

are considered less reliable and may require assessment by a professional radiologist. Ultimately, we obtained the optimal threshold

θ = 0.0981

and the confidence level of the model's predictions:

C (P) = {\begin{matrix} U < θ high \to confidence \\ U \geq θ low \to confidence \end{matrix}

(16)

Overall training objective

The overall objective of MSK-PT encompasses three main loss functions: weighted cross-entropy loss to minimize the difference between predicted probabilities and true labels; global text-image contrastive loss to enhance learning of global features; and fine-grained similarity-aware modal alignment loss to accurately capture local similarities between text and images. Specifically, these are detailed as follows:

L_{C E} = - \frac{1}{T} \sum_{i = 1}^{T} \sum_{j = 1}^{k} y_{i j} l o g (p_{i j})

(17a)

L = L_{C E} + λ_{1} L_{G R A} + λ_{2} L_{S M A}

(17b)

Where

T

represents the number of samples from the training data text-image pairs.

y_{i j}

is the true label of the

i

-th sample belonging to the

j

-th category, and

p_{i j}

is the model's predicted probability that the i -th sample belongs to the

j

-th category.

λ_{1}

and

λ_{2}

are hyperparameters balancing the two losses. We optimize the model parameters by minimizing the overall loss

L

Experiment setup

In this section, we introduce the pneumoconiosis dataset, provide detailed implementation specifics of the model training, discuss the evaluation metrics used, and outline the baseline methods for comparison.

Datasets

We collected the PneumoCXR dataset, comprising data from independent patients between 2018 and 2021, at an occupational disease prevention and control institute in Shanxi Province, China. In addition, clinical data and diagnostic reports of each patient were collected, and a total of 2014 patients’ image-text pairs were obtained. To minimize assessment errors, predictions were independently diagnosed and reported by at least four radiologists with over 10 years of clinical experience each. Final results were determined based on consensus among the radiologists, categorizing each image into one of four labels: ‘stage-0’, ‘stage-1’, ‘stage-2’, and ‘stage-3’, as detailed in Table 1. All clinical data collection was conducted with informed consent from the patients and approval from the hospital's ethics committee.

Table 1.

PneumoCXR dataset split detailed information.

Types	Train set	Test set	Total
Stage-0	740	183	923
Stage-1	591	146	737
Stage-2	182	44	226
Stage-3	104	24	128
Total	1617	397	2014

Training details

The training process for the MSK-PT approach involves two stages: multimodal pre-training and downstream task fine-tuning.

Stage I: Multi-modal Pre-training. During this stage, our focus is to enable the model to understand multimodal inputs and perform fine-grained alignment adjustments through pre-training. However, it is challenging to obtain multimodal pneumoconiosis data with fine-grained features. Therefore, during the pre-training process, in order to avoid the overfitting problem caused by the complexity of the model, we adopt the “five-fold cross-validation” training strategy. Specifically, we randomly divide the dataset into five subsets, select one of the parts as the validation set each time for training, and the rest as the training set. This process is repeated five times to ensure that each subset is evaluated as a validation set, so as to more comprehensively evaluate the performance of the model.

Prior to inputting into the network, all data dimensions are uniformly resized to 224 × 224 pixels for computational convenience. We utilize BERT⁵⁹ as the text encoder module and ResNet-50⁶⁰ as the image encoder module. The training process iterates for 200 epochs, using the Adam optimizer, with parameters set according to recommendations from:⁶³ exponential decay rates $β_{1} = 0.90$ and $β_{2} = 0.999$ , initial learning rate $1.0 \times e^{- 3}$ , and a ‘cos’ learning rate scheduling gradually reducing to $1.0 \times e^{- 4}$ . The batch size is set to 16, with the current iteration's learning rate dynamically adjusted by formula $l r_{i t e r} = l r_{i n i t} \times \frac{1}{2} [c o s (\frac{i t e r \times π}{m a x_i t e r}) + 1]$ . All experiments were conducted in a Python 3.8 and PyTorch 1.11.0 environment.

Stage II: Downstream Task Tuning. In the second stage, we conducted evaluations of downstream tasks on the PneumoCXR test dataset comprising 397 CXR images. The ResNet-50⁶⁰ architecture continued to serve as the backbone for the downstream image encoder. To further enhance the model, we leveraged data-associated features and pre-trained domain knowledge to propagate label information from labeled data to unlabeled samples in the downstream fully visual modality. We continued fine-tuning the model using the Adam optimizer with a learning rate of $1.0 \times e^{- 3}$ , following the same learning rate and decay strategy employed during the pre-training phase. The training batch size remained at 16, and the training process iterated for 200 epochs.

Evaluation index

To comprehensively evaluate our model's performance in pneumoconiosis diagnosis, we utilized multiple evaluation metrics including Accuracy, Sensitivity, Specificity, F1 Score (F1), and Area Under the Curve (AUC).

Accuracy = \frac{T P + T N}{T P + F P + F N + T N}

(18)

Sensitivity = \frac{T P}{T P + F N}

(19)

Specificity = \frac{T N}{T N + F P}

(20)

F_{1} = 2 \cdot \frac{precision \cdot recall}{precision + recall}

(21)

In our evaluation metrics, True Positive (TP), False Negative (FN), and True Negative (TN) represent the model's performance in different prediction scenarios:

True Positive (TP): The model predicts Positive, and the actual value is also Positive. This indicates the model correctly identifies true positive cases.

False Negative (FN): The model predicts Negative, but the actual value is Positive, indicating the model fails to identify positive cases, resulting in incorrect negative predictions.

True Negative (TN): The model predicts Negative, and the actual value is also Negative. This indicates the model correctly identifies true negative cases.

Additionally, AUC refers to the area under the Receiver Operating Characteristic (ROC) curve. The ROC curve plots the False Positive Rate (FPR) on the x-axis against the True Positive Rate (TPR) on the y-axis. A higher AUC value indicates better classification performance of the model. The formulas for calculating FPR and TPR are as follows:

F P R = \frac{F P}{F P + T N}

(22)

T P R = \frac{T P}{T P + F N}

(23)

The value of AUC is always less than 1. Since the ROC curve typically lies above the diagonal line ( $y = x$ ), the AUC value usually ranges between [0.5, 1]. A higher AUC value closer to 1 indicates better classification performance of the model.

To facilitate comparison and analysis of different models, we computed the average values of these metrics as a comprehensive evaluation of overall performance. This approach not only helps identify models that excel in specific metrics but also ensures that the chosen model performs well across multiple performance dimensions, thereby enhancing the reliability and accuracy of dust lung disease diagnosis.

Baseline methods

To validate the effectiveness of the proposed MSK-PT model, we compared its performance from multiple perspectives against current state-of-the-art baseline models. Considering significant domain differences between natural image-text pairs and medical image-text pairs, directly transferring pre-trained natural image-text models to medical tasks may result in performance degradation. Therefore, we chose state-of-the-art pre-trained models specifically designed for medical domains as comparison baselines.To ensure fairness in experiments and minimize the impact of network architectures on final performance, all experiments utilized the same BERT⁵⁹ and ResNet-50⁶⁰ as foundational text and image encoder modules, consistent with MSK-PT. Below are the selected comparison baselines and their highlights:

Baseline: Uses plain BERT and ResNet-50 as the backbone model without additional techniques.

MedViLL¹⁷: Employs a BERT⁵⁹-based architecture with a multimodal attention masking scheme to maximize performance on visual-language understanding tasks.

MedKLIP⁶⁴: Enhances visual-language pretraining by leveraging domain-specific medical knowledge descriptions through paired image-text data augmentation.

KAD⁴⁶: Trains a knowledge encoder using a comprehensive medical knowledge graph, applying it to guide visual representation learning with images and radiology reports.

ConVIRT⁴⁴: Improves visual representation by maximizing consistency between real image-text pairs and random pairs through bidirectional contrastive objectives between image and text modalities.

MGCA⁵³: Facilitates generalized medical visual representation learning via multi -scale semantic correspondence between medical images and radiology reports.

MLIP⁵⁴: Integrates language information into the visual domain using domain -specific medical knowledge as guiding signals through image-text contrastive learning.

Evaluation and analysis

In this section, we compare MSK-PT with other state-of-the-art models and conduct visual evaluations. We performed ablation experiments to validate the effectiveness of each module and component of MSK-PT. All baseline models underwent diagnostic performance evaluation on the PneumoCXR test dataset to verify the effectiveness of MSK-PT.

Comparative results with other methods

Table 2 presents the experimental results comparing MSK-PT with several representative visual-linguistic models, demonstrating the effectiveness of their training strategies and outperforming baseline models. Unlike typical medical image classification tasks, pneumoconiosis CXR images exhibit strong inter-class similarities, such as similar lung contour positions and grayscale intensity ranges among lesions, due to the fact that all images reflect the radiological characteristics of the lungs after X-ray penetration. Our similarity perception module mitigates these inter-class similarities to some extent. On the PneumoCXR test dataset, MSK-PT demonstrates outstanding performance across most metrics and averages, surpassing all reference baseline models and achieving state-of-the-art performance. This result indicates that MSK-PT significantly enhances the reliability of detection in unlabeled cases by leveraging data-associated features and domain knowledge acquired during pre-training, even in the absence of annotated medical information or clinical interpretation. Additionally, we believe that uncertainty thresholds contribute to enhancing the robustness of MSK-PT, ensuring reliable predictive outcomes.

Table 2.

Comparative results with the SOTA baselines method on testing sets. The best predictive performance is highlighted with boldface numbers.

Method	Acc. (%)	Sens. (%)	Spec. (%)	F1 (%)	AUC (%)	AVG (%)
Baseline	65.45	53.18	62.52	63.12	63.96	61.37
MedViLL¹⁷	74.12	65.84	72.19	74.60	75.28	70.59
MedKLIP⁶⁴	75.46	64.29	72.04	73.18	74.03	71.36
KAD⁴⁶	79.20	68.04	75.15	75.34	78.72	76.15
ConVIRT⁴⁴	73.25	62.19	70.99	72.27	73.88	69.61
MGCA⁵³	76.37	67.18	74.29	74.45	76.68	73.48
MLIP⁵⁴	77.66	66.96	73.35	75.43	77.73	72.73
MSK-PT	81.73	68.92	76.96	76.73	79.63	77.28

ROC curve and confusion matrix analysis

Figure 4 illustrates the ROC curves and confusion matrix of the MSK-PT method in the pneumoconiosis image classification task. Analyzing the ROC curves on the left side of Figure 4, it is evident that the model achieves excellent AUC values across different lesion categories. Specifically, the ROC curves of MSK-PT approach the upper-left corner for each category, indicating high sensitivity and specificity at various thresholds. The confusion matrix on the right side of Figure 4 displays the model's classification results across different categories. Our method demonstrates a substantial number of correct predictions in each category. However, between “Normal” and “Stage 1”, the model incorrectly confuses some pneumoconiosis images. This could be attributed to the early-stage lesions of pneumoconiosis being less pronounced on X-ray chest radiographs, sharing similar radiographic features, and being susceptible to interference from other respiratory diseases. Quantitative analysis further validates the exceptional performance of the MSK-PT model in identifying similar lesion features in pneumoconiosis images.

Figure 4.

ROC curve and confusion matrix of MSK-PT for all four pneumoconiosis types.

Reliability analysis

Figure 5 illustrates four pneumoconiosis image samples detected using our MSK-PT model. Previous studies typically relied on selecting the class with the highest probability as the final diagnostic basis. In contrast, the MSK-PT model not only provides the final prediction but also includes uncertainty scores to explicitly indicate the reliability of the diagnosis. Lower uncertainty scores indicate higher confidence in the model's final decision (as shown in the left column of Figure 5). In cases where the final diagnosis is incorrect (as depicted in the right column of Figure 5), assigning high uncertainty scores helps identify the unreliability of these predictions. High uncertainty scores prompt radiologists to reassess the images to avoid misdiagnosis or missed diagnosis. By introducing uncertainty thresholds, the MSK-PT model enhances both diagnostic accuracy and the assessment of diagnostic result reliability.

Figure 5.

Four samples of pneumoconiosis images were detected using MSK-PT. The left column is samples that the model predicts correctly, and on the contrary, the right column is samples that the model predicts incorrectly. In addition, our MSK-PT not only provides probability scores, but also provides corresponding uncertainty scores to reflect the reliability of the predictions. If the uncertainty score is less than the threshold θ, it means that the model's prediction is reliable. On the contrary, if the uncertainty score exceeds the threshold θ, it means that the result is unreliable and requires re-evaluation by a radiologist.

Visualization analysis

Grad-CAM images are commonly used to localize discriminative regions for object detection and classification tasks. Here, we use Grad-CAM visual heatmaps to support the outstanding performance of MSK-PT in extracting fine-grained features, highlighting improvements over baseline methods. Figure 6 presents a visualization where MSK-PT focuses more on regions of maximum interest related to data-associated features and pre-trained knowledge, significantly enhancing the model's robustness against irrelevant background information and better showcasing features associated with lesions. In contrast, baseline methods struggle to accurately predict subtle differential features of specific diseases, such as basilar reticular opacities in the posterior bases, centrilobular nodules, and branching shadows with blurred boundaries and low contrast against normal tissues, which are low-resolution lesions diffusely throughout the lungs. Therefore, our approach effectively distinguishes disease-specific low-resolution features from other general attributes in CXR images, thereby accurately differentiating characteristic micro-differences among various stages of pneumoconiosis, which is crucial to ensure the interpretability and credibility of the model.

Figure 6.

Visualization of heatmaps of four representative pneumoconiosis samples. On the top, we mark the pathology regions annotated by certified radiologists (the circular markers); on the bottom, we visualize the visual features from MSK-PT (brighter colors means higher feature values). We can see that the baseline method has difficulty in accurately obtaining the location of the disease region, and MSK-PT can successfully highlight the abnormal regions identified by radiologists.

Failure analysis

Additionally, we conducted a visual analysis of the four worst-performing cases in the PneumoCXR test set to further assess the limitations of the model. As shown in Figure 7, since the PneumoCXR dataset provides the exact locations of pneumoconiosis lesions, we compared the lesion areas annotated by radiologists with the high-response areas generated by Grad-CAM. The results indicate that although our method can generate large attention regions covering multiple feature locations in the samples, when dealing with extremely difficult cases, these attention regions still fail to accurately cover the actual lesion areas, ultimately leading to incorrect predictions.

Figure 7.

Visualization of representative cases where our method fails to focus on disease regions. The circular markers indicate the ground truth locations.

Ablation studies

We conducted a series of ablation experiments to examine the effects of different structures on the performance of the proposed MSK-PT model.

Effectiveness of Different Components: To analyze the importance of each component within MSK-PT, we conducted comprehensive ablation experiments, systematically altering one variable at a time while following the same training setup as MSK-PT. The specific experiments include:

“Baseline” refers to using only BERT⁵⁹ and ResNet-50⁶⁰ as the base algorithms.

“MSK-PT w/o SMA” means removing the local fine-grained similarity-aware understanding and only using global text and image representations for pre-training.

“MSK-PT w/o Pre-training Data Feature” signifies using the downstream task's own image encoder and pre-trained knowledge prompts for training the visual task.

“MSK-PT w/o Shared Text Feature” implies using the downstream task's own text encoder and learnable prompts to generate text features.

“MSK-PT w/o Thresholding” indicates the removal of the uncertainty thresholding module deployed in the downstream learning task.

Additionally, we ablated components within the SMA module:

“SMA w/o TA” refers to performing only disease-level semantic correspondence.

“SMA w/o PA” means matching and aligning only the local fine-grained representations between the text and image.

The quantitative results in Table 3 demonstrate that the design of MSK-PT plays a crucial role in its final performance, particularly highlighting the critical role of the SMA module in handling fine-grained similarity and multimodal alignment. Through these experiments, we further validated the importance of each component to the overall performance of the model, proving the rationality and effectiveness of the design.

Effectiveness of SMA Module: During the pre-training phase, the SMA module is designed to enhance the model's ability to understand fine-grained representations between images and text, enabling it to more accurately identify similarities and differences in features. To further validate the contribution of the SMA module within the MSK-PT method, we applied t-SNE visualization⁶⁵ to analyze the features learned by the model on the PneumoCXR dataset. This demonstrates how similar features affect the deep learning model's recognition of pneumoconiosis and how the SMA module improves this recognition capability. As shown in Figure 8, with the SMA module (Figure 8(a)), the features learned during training exhibit more distinct clustering patterns, with significantly reduced overlap between samples. This indicates that the SMA module effectively guides the model in learning more discriminative features, greatly improving its ability to distinguish between different categories, thereby benefiting downstream tasks. In contrast, without the SMA module (Figure 8(b)), samples from different categories show substantial overlap, and due to the low variability in lesion features across images, overfitting occurs, leading to blurred class boundaries. The experimental results demonstrate that the SMA module plays a crucial role in feature learning by precisely capturing and distinguishing between features of different categories, helping the model create clearer inter-class boundaries in the feature space. This significantly enhances the model's ability to extract discriminative features, ultimately improving its accuracy and robustness in diagnostic tasks.

Effectiveness of Knowledge Distillation Strategy: Furthermore, we conducted a detailed comparison of the performance of different distillation methods, with all experiments following the same training settings as MSK-PT. The results indicate that the baseline method showed significant overfitting after just a few epochs of training. By analyzing the accuracy and loss convergence curves in Figure 9, it is evident that compared to the “w/o Shared Data Feature” and “w/o Shared Text Feature” methods, the innovative design of MSK-PT, which includes data-associated features from the pre-training phase and domain knowledge-based distillation strategies, played a crucial role in enhancing model performance. This approach not only effectively mitigated the overfitting issue but also significantly improved the model's generalization ability.

Effect of Different Image Backbone：In Table 4, we used BERT⁵⁹ as the base text encoder module and performed pre-training with various visual encoders and image resolutions, including ViT-B/16⁶⁶ and ResNet-50.⁶⁰ We implemented this using the officially released code. The results indicate that MSK-PT improves downstream task classification performance across different image backbone networks. Although ResNet-50 performed best at a resolution of 1024 × 1024, high computational resources and time costs are significant considerations in practical applications. Therefore, we ultimately chose ResNet-50 at a resolution of 224 × 224 as the base visual encoder, as it balances performance while significantly reducing computational overhead, thereby enhancing training and inference speed.

Effect of Training Sample Numbers: To explore the classification performance of MSK-PT with varying amounts of training samples, we fine-tuned the model based on different proportions of training samples to analyze its robustness to the number of training samples. Figure 10 shows the accuracy, specificity, F1score, and sensitivity under different amounts of training samples. The horizontal axis represents the percentage of training samples, using 25%, 50%, 75%, and 100% proportions, respectively. The vertical axis corresponds to the values of different metrics. The results indicate that as the percentage of training samples increases, the classification performance of the MSK-PT method significantly improves.

We can observe that although the reduction in training samples affects the performance of the classification task, the impact is relatively minor. This is primarily due to MSK-PT's detailed extraction and effective alignment of the underlying multimodal features of pneumoconiosis, as well as the establishment of robust text-image associations, enabling the model to maintain high classification performance even with fewer training samples. This performance not only demonstrates the robustness of the MSK-PT model in handling different amounts of training samples but also highlights its reduced dependency on training data in practical applications.

Effect of Different Loss Functions: To emphasize the significance of global and local loss functions within the model, we conducted ablation experiments using various loss combinations. In MSK-PT's loss function, global contrastive learning serves as the performance baseline, supplemented by token-wise and prototype (disease)-wise parts. We focused on the token-wise alignment (TA) and prototype-wise alignment (PA) within the SMA module. As shown in Table 5, both TA and PA contribute to enhanced classification performance, indicating that alignment at both the token and prototype levels aids the text and image encoders in learning representations more suitable for downstream tasks. Notably, when TA and PA are combined, further performance improvements are observed on the PneumoCXR dataset, suggesting the complementary nature of TA and PA. Interestingly, omitting PA loss results in a more significant performance drop compared to omitting TA loss, highlighting the role of prototype-wise contrastive learning in reducing false negatives and better adapting the model to the similarities in pneumoconiosis images. This finding validates the critical role of this loss function in model training. By combining both global and local losses, MSK-PT achieves optimal performance.

Figure 8.

T-SNE visualizations of features with and without SMA, where different colors represent their true labels.

Figure 9.

The accuracy and loss convergence curves for different knowledge distillation strategies on the test set.

Figure 10.

Avg index value for different percentages of training samples. It shows that MSK-PT remains effective under different dataset sizes.

Table 3.

Ablation study of MSK-PT by removing individual modules.

Method	Acc. (%)	Sens. (%)	Spec. (%)	F1 (%)	AUC (%)	AVG (%)
Method	Ablation on MSK-PT modules
MSK-PT	81.73	68.92	76.96	76.73	79.63	77.28
Baseline	56.63	45.13	51.02	52.92	54.74	53.27
MSK-PT w/o SMA	60.42	50.62	55.86	55.83	57.14	56.13
MSK-PT w/o Pre-training Data Feature	63.68	52.25	56.36	59.02	61.66	58.34
MSK-PT w/o Shared Text Feature	64.76	53.91	56.70	58.79	62.18	57.68
MSK-PT w/o Thresholding	75.20	64.68	66.17	67.07	72.34	68.15
Method	Ablation on SMA modules
SMA w/o TA	71.09	60.80	62.45	65.12	68.51	65.23
SMA w/o PA	66.14	53.67	55.41	58.76	62.20	57.58

Table 4.

Ablation study of MSK-PT with different image encoders and different image resolutions.

Image Backbone	Size	Acc. (%)	Sens. (%)	Spec. (%)	F1 (%)	AUC (%)	AVG (%)
ViT-B/16⁶⁶	224 × 224	80.78	67.51	76.15	75.82	78.91	76.26
ResNet-50⁶⁰	224 × 224	81.73	68.92	76.96	76.73	79.63	77.28
ResNet-50⁶⁰	512 × 512	81.16	68.34	76.52	76.21	79.48	77.13
ResNet-50⁶⁰	1024 × 1024	82.92	69.27	77.23	76.95	80.59	78.54

Table 5.

Ablation study of MSK-PT on different loss task setting.

Tasks Setting			Evaluation Index
$L_{G R A}$	$L_{T A}$	$L_{P A}$	Acc. (%)	Sens.(%)	Spec.(%)	F1 (%)	AUC(%)	AVG (%)
	$\sqrt$	$\sqrt$	79.56	66.52	74.33	75.10	77.96	73.60
$\sqrt$	$\sqrt$		80.20	67.19	74.74	75.63	78.21	74.28
$\sqrt$		$\sqrt$	80.79	67.58	75.57	76.03	78.67	74.65
$\sqrt$	$\sqrt$	$\sqrt$	81.73	68.92	76.96	76.73	79.63	77.28

Limitations and future works

While MSK-PT has shown remarkable performance in multi-modal medical image classification tasks, acquiring and developing high-quality multi-modal medical data presents greater challenges compared to single-modal data. This not only increases the complexity and cost of data acquisition but also restricts the scale and diversity of available datasets. In addition, text-image pretraining models inevitably encounter challenges when predicting information not explicitly mentioned in the text or when handling data representations absent from the training set, thereby limiting the model's ability to generalize effectively across specific domains. In future research, we aim to explore more effective domain adaptation techniques or leverage regularization methods to enhance the model's adaptability to novel data scenarios.

Furthermore, Grad-CAM has been instrumental in visualizing and interpreting the pneumoconiosis staging model in this study. By highlighting the critical regions where the model focuses its attention, this approach not only enhances the model's robustness against irrelevant background information but also strengthens its interpretability and clinical applicability. Moving forward, we aim to refine the Grad-CAM method by integrating additional domain knowledge and multi-modal data to further enhance the model's diagnostic accuracy and interpretability in complex scenarios.

Conclusion

This paper proposes a novel Multimodal Similarity-aware and Knowledge-driven Pre-Training (MSK-PT) approach, investigating the association between pneumoconiosis lesions at different stages. It focuses on discerning differences between similar images and innovatively incorporates associated features from cross-modal pneumoconiosis data to better perceive multimodal inputs. This approach demonstrates superior performance in pneumoconiosis classification tasks requiring high granularity of understanding. Furthermore, to address the scarcity of labeled data, we utilize learned data association features and domain knowledge constraints from the pre-training phase as priors and constraints for the model, transferring them to downstream fullly visual learning tasks for training and inference on unlabeled data. Additionally, we introduce an uncertainty threshold strategy to mitigate potential impacts of label errors, effectively enhancing the model's robustness and interpretability in pneumoconiosis classification.

In summary, in the field of medical research where it is difficult to obtain a large amount of labeled data, our method not only enhances the diagnostic capabilities of multimodal text-image models facing limited labeled image datasets but also facilitates learning and understanding of correlations between text and images to better interpret medical images, thereby improving diagnostic accuracy and reliability. Evaluation on the PneumoCXR dataset demonstrates the effectiveness of the MSK-PT model, outperforming several state-of-the-art multimodal techniques across multiple evaluation metrics. Furthermore, through ablation studies, we provide empirical evidence of the unique contributions of each component within the method.

Footnotes

Author contributions

Xueting Ren: Conceptualization, Methodology, Data collection, Experimental deployment, Writing - original draft. Guohua Ji: Data collection & Experimental deployment. Surong Chu: Methodology & Data collection. Shinichi YOSHIDA: Supervision, review & editing. Juanjuan Zhao: Supervision, Project administration, review & editing. Baoping Jia: Data collection & annotation & curation, Clinical support. Yan Qiang: Methodology, Project administration, review & editing.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is supported by National Natural Science Foundation of China [grant numbers 62376183, U21A20469]; National Health Commission Key Laboratory of Pneumoconiosis open project [grant numbers YKFKT004]; NHC Key Laboratory of Pneumoconiosis Shanxi China [grant numbers 2020-PT320-005]; Shanxi Provincial Science and Technology Innovation Talent Team Special Plan [grant numbers 202304051001009]; The Central Government Guides Local Science and Technology Development Funding Projects [grant numbers YDZJSX2022C004]; China Scholarship Council [grant numbers 202306930020].

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability

Pneumoconiosis datasets supporting the findings of this study were not publicly available due to the confidentiality policy of the Chinese National Health Council and institutional patient privacy regulations. However, they were available from the corresponding authors upon request.*

References

Luo

Song

, et al. Pneumoconiosis: current status and future prospects. Chin Med J (Engl) 2021; 134: 898–907.

Ehrlich

Myers

te Water Naude

, et al. Lung function loss in relation to silica dust exposure in South African gold miners. Occup Environ Med 2011; 68: 96–101.

Cai

. Comprehension of GBZ 70-2015 diagnosis of occupational pneumoconiosis. Zhonghua Lao Dong Wei Sheng Zhi Ye Bing Za Zhi 2016; 34: 866–867.

Muszyńska-Graca

Dąbkowska

Brewczyński

. Guidelines for the use of the international classification of radiographs of pneumoconioses of the International Labour Office (ILO): substantial changes in the currrent edition. Med Pr 2016; 67: 833–837.

Wang

Zhang

Shu

, et al. Intra-class consistency and inter-class discrimination feature learning for automatic skin lesion classification. Med Image Anal 2023; 85: 102746.

Guan

Zhou

, et al. Parameter-free similarity-aware attention module for medical image classification and segmentation. IEEE Trans Emerg Top Comput Intell I 2022; 7: 845–857.

Tian

Wang

Chen

, et al. Diagnose like doctors: weakly supervised fine-grained classification of breast cancer. ACM Trans Intell Syst Technol 2023; 14: 1–17.

Togo

Ogawa

, et al. Self-knowledge distillation based self-supervised learning for COVID-19 detection from chest x-ray images. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp.1371–1375.

Dong

Zhu

Zhang

, et al. Use data augmentation for a deep learning classification model with chest X-ray clinical imaging featuring coal workers’ pneumoconiosis. BMC Pulm Med 2022; 22: 271.

10.

Zhang

Rong

, et al. A deep learning-based model for screening and staging pneumoconiosis. Sci Rep 2021; 11: 2201.

11.

Müller

Kaissis

Zou

, et al. Joint learning of localized representations from medical images and reports. In: European Conference on Computer Vision, Cham, 2022, pp.685–701: Springer Nature Switzerland.

12.

Rizvi

Tang

Jiang

, et al. Local contrastive learning for medical image recognition. AMIA Annu Symp Proc 2023; 2023: 1236–1245.

13.

Wang

Agarwal

, et al. MedCLIP: contrastive learning from unpaired medical images and text. In: Proc Conf Empir Methods Nat Lang Process, 2022, pp.3876–3887.

14.

, et al. Promptkd: unsupervised prompt distillation for vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp.26617–26626.

15.

Bica

Ilić

Bauer

, et al. Improving fine-grained understanding in image-text pre-training, arXiv preprint arXiv:2401.09865 (2024).

16.

Monajatipoor

Rouhsedaghat

, et al. Berthop: an effective vision-and-language model for chest x-ray disease diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Cham, 2022, pp.725–734: Springer Nature Switzerland.

17.

Moon

Lee

Shin

, et al. Multi-modal understanding and generation for medical images and text via vision-language pre-training. IEEE J Biomed Health Inform 2022; 26: 6070–6080.

18.

Liu

Wei

, et al. Improving medical vision-language contrastive pretraining with semantics-aware triage. IEEE Trans Med Imaging 2023; 42: 3579–3589.

19.

You

Zhou

Xiao

, et al. Learning visual representation from modality-shared contrastive language-image pre-training. In: European Conference on Computer Vision, Cham, 2022, pp.69–87: Springer Nature Switzerland.

20.

Peng

Wang

Dong

, et al. Kosmos-2: Grounding multimodal large language models to the world, arXiv preprint arXiv:2306.14824. 2023.

21.

You

Zhang

Gan

, et al. Ferret: Refer and ground anything anywhere at any granularity, arXiv preprint arXiv:2310.07704. 2023.

22.

Chen

Zhang

Zeng

, et al. Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic, arXiv preprint arXiv:2306.15195. 2023.

23.

Fan

, et al. Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp.9729–9738.

24.

Chen

Kornblith

Norouzi

, et al. A simple framework for contrastive learning of visual representations. In: International conference on machine learning, 2020, pp.1597–1607: PMLR.

25.

Zheng

Salakhutdinov

Eysenbach

. Contrastive difference predictive coding, arXiv preprint arXiv:2310.20141. 2023.

26.

Tian

Krishnan

Isola

. Contrastive multiview coding. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, 2020, pp.776–794: Springer International Publishing.

27.

Wang

Liu

. Unsupervised feature learning by cross-level instance-group discrimination. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp.12586–12595.

28.

Guo

Zhang

Chen

, et al. Unsupervised feature learning with emergent data-driven prototypicality. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp.23199–23208.

29.

Grill

Strub

Altché

, et al. Bootstrap your own latent-a new approach to self-supervised learning. Adv Neural Inf Process Syst 2020; 33: 21271–21284.

30.

Chen

. Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp.15750–15758.

31.

Caron

Touvron

Misra

, et al. Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp.9650–9660.

32.

, et al. Decoding visual neural representations by multimodal learning of brain-visual-linguistic features. IEEE Trans Pattern Anal Mach Intell 2023; 45: 10760–10777.

33.

Chakladar

Kumar

Roy

, et al. A multimodal-Siamese Neural Network (mSNN) for person verification using signatures and EEG. Inform Fusion 2021; 71: 17–27.

34.

Yang

Cui

Wang

, et al. Cross-modal contrastive learning for multimodal sentiment recognition. Appl Intell 2024; 54: 4260–4276.

35.

Liu

Fan

Zhang

, et al. Contrastive multimodal fusion with tupleinfonce. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp.754–763.

36.

Huang

Shen

Lungren

, et al. Gloria: a multimodal global-local representation learning framework for label-efficient medical image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp.3942–3951.

37.

Han

Chen

Poria

. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis, arXiv preprint arXiv:2109.00412. 2021.

38.

Song

Huang

Xue

, et al. Supervised prototypical contrastive learning for emotion recognition in conversation, arXiv preprint arXiv:2210.08713. 2022.

39.

Yang

Zhang

Alhuzali

, et al. Cluster-level contrastive learning for emotion recognition in conversations. IEEE Trans Affect Comput 2023; 14: 3269–3280.

40.

Liang

Mao

, et al. Context or knowledge is not always necessary: a contrastive learning framework for emotion recognition in conversations. In: Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp.14054–14067.

41.

Khosla

Teterwak

Wang

, et al. Supervised contrastive learning. Adv Neural Inf Process Syst 2020; 33: 18661–18673.

42.

Wang

Tong

, et al. Soft neighbors are positive supporters in contrastive visual representation learning, arXiv preprint arXiv:2303.17142. 2023.

43.

Yang

Wang

Zhu

. Few-shot classification with contrastive learning. In: European conference on computer vision, Cham, 2022, pp.293–309: Springer Nature Switzerland.

44.

Zhang

Jiang

Miura

, et al. Contrastive learning of medical visual representations from paired images and text. In: Machine Learning for Healthcare Conference, 2022, pp.2–25: PMLR.

45.

Zhang

Nie

Cao

, et al. Self-supervised fusion for multi-modal medical images via contrastive auto-encoding and convolutional information exchange. IEEE Comput Intell Mag 2023; 18: 68–80.

46.

Zhang

, et al. Knowledge-enhanced visual-language pre-training on chest radiology images. Nat Commun 2023; 14: 4542.

47.

Chen

, et al. A knowledge-based learning framework for self-supervised pre-training towards enhanced recognition of biomedical microscopy images. Neural Netw 2023; 167: 810–826.

48.

Chen

Xue

, et al. Knowledge boosting: rethinking medical contrastive vision-language pre-training. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Cham, 2023, pp.405–415: Springer Nature Switzerland.

49.

Chen

Wan

. Align, reason and learn: enhancing medical vision-and-language pre-training with knowledge. In: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp.5152–5161.

50.

Pan

Han

, et al. Contrastive language-image pre-training with knowledge graphs. Adv Neural Inf Process Syst 2022; 35: 22895–22910.

51.

Baumgartner

Jäger

Isensee

, et al. Nndetection: a self-configuring method for medical object detection. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24, 2021, pp.530–539: Springer International Publishing.

52.

Cheng

Lin

Lyu

, et al. PRIOR: prototype representation joint learning from medical images and reports. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp.21304–21314.

53.

Wang

Zhou

Wang

, et al. Multi-granularity cross-modal alignment for generalized medical visual representation learning. Adv Neural Inf Process Syst 2022; 35: 33536–33549.

54.

Yang

Ren

, et al. MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning, arXiv preprint arXiv:2402.02045. 2024.

55.

Chen

Liu

Zhao

, et al. Distilling knowledge via knowledge review. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5008–5017.

56.

Vaswani

Shazeer

Parmar

, et al. Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems 2017, pp. 6000–6010.

57.

Kiros

Hinton

. Layer normalization, arXiv preprint arXiv:1607.06450. 2016.

58.

Cuturi

. Sinkhorn distances: Lightspeed computation of optimal transport. In: Proceedings of the 26th International Conference on Neural Information Processing Systems 2013, pp. 2292–2300.

59.

Devlin

Chang

Lee

, et al. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805. 2018.

60.

Zhang

Ren

, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp.770–778.

61.

Wang

Lin

Wang

, et al. Uncertainty-inspired open set learning for retinal anomaly identification. Nat Commun 2023; 14: 6757.

62.

Shan

. Improved confidence intervals for the Youden index. PloS one 2015; 10: e0127272.

63.

Kingma

. Adam: A method for stochastic optimization. In: International Conference on Learning Representations, 2014.

64.

Tiu

Talius

Patel

, et al. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat Biomed Eng 2022; 6: 1399–1406.

65.

Van der Maaten

Hinton

. Visualizing data using t-SNE. J Mach Learn Res 2008; 9: 2579–2605.

66.

Dosovitskiy

Beyer

Kolesnikov

, et al. An image is worth 16 × 16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations, 2020.