Automated Lesion Segmentation in Medical Imaging via Integration of nnU-Net Optimization and SAM Approach

Abstract

Background:

Deep learning has transformed medical imaging by enabling earlier and more accurate disease diagnosis. Lesion and tumor segmentation, essential for analyzing and tracking morphological changes, is commonly done with U-Net variants, though these often lack cross-domain generalization and do not fully leverage foundation models like the Segment Anything Model (SAM), which still requires manual intervention to define the region of interest (ROI).

Objectives:

To enhance generalization and reduce manual intervention by combining the automatic optimization of nnU-Net with the precision of SAM.

Design:

Experimental evaluation of a hybrid segmentation framework for lung nodule analysis.

Methods:

We propose a novel approach integrating the automatic optimization capabilities of nnU-Net for lesion detection with the high-precision segmentation of SAM, eliminating the need for manual intervention by the clinician. The method was evaluated on the LIDC-IDRI dataset, a widely recognized benchmark for lung nodule segmentation.

Results:

Our approach produces more anatomically coherent segmentations than nnU-Net alone. In many cases, the resulting boundaries more closely reflect true nodule morphology than individual expert annotations, despite high inter-expert variability.

Conclusion:

The proposed integration of nnU-Net with SAM enables fully automated lesion segmentation without manual intervention. The method improves generalization and accuracy across medical imaging domains, achieving expert-level performance in pulmonary nodule segmentation.

Keywords

deep learning lung nodule medical imaging semantic segmentation nnU-Net SAM

Introduction

Image segmentation involves dividing an image into regions with the goal of extracting the areas of interest. It is considered a classic problem in computer vision. To extract regions of interest, characteristics such as grayscale levels, color, and geometric shapes are considered. Segmentation has been applied in various fields, including satellite imagery, autonomous driving, and medical image analysis. Segmentation is of particular importance in medical imaging because it helps locate regions most affected by a disease and serves as a support tool for medical professionals. In the construction of a computer-aided diagnosis (CAD) system, segmentation is one of the initial stages of image processing and acts as a precursor to the disease classification model. Semantic segmentation is commonly applied in medical image analysis to label each pixel in the image, providing precise information about the shape and size of the lesion.

Regarding segmentation methods, we find traditional methods, such as thresholding, region-based, and edge-region-based approaches. These methods use image characteristics and mathematical concepts to achieve fast segmentation. However, traditional techniques generally offer less precision than deep learning methods because deep learning models can automatically extract features from images. Among the deep learning methods we find Fully Convolutional Networks (FCN)-based methods, such as U-Net,¹ an autoencoder architecture for medical imaging segmentation. More recent advancements have been made using Vision Transformers,^2-4 which employ attention mechanisms to focus on the most important features in the image.

One of the main drawbacks of many proposed deep learning architectures is their lack of generality when applied to new datasets with different medical imaging modalities. In the literature, we found specific U-Net modifications that perform well for particular problems; however, they lack versatility. Foundation models address this issue by training transformer-based architectures on large-scale datasets with diverse image types. In recent years, foundation models have revolutionized the field of Artificial Intelligence (AI). These types of models can effectively and efficiently solve several tasks across different domains, excelling in Natural Language Processing (NLP) and Computer Vision (CV) problems. The most prominent foundation models are large language models like GPT-4⁵ and multimodal large language models like Gemini,⁶ which accept images and audio. In contrast, diffusion models, such as Stable Diffusion,⁷ can generate all types of images from a prompt.

The emergence of these foundation models has enabled the development of new general-purpose segmentation models that allow the segmentation of various objects across any domain. These models often incorporate a prompt, such as a user-provided click or bounding box, to precisely indicate the segmented area. This has led to the development of a new field within segmentation known as interactive segmentation.

A prominent example of this approach is the SAM model, which was released in 2023. For medical imaging, MedSAM⁸ was introduced in 2024, a model based on SAM and trained on a large dataset of > 1.5 million medical images. Although these models achieve high segmentation accuracy, they require manual user intervention through a prompt. This prompt can be a mouse click, bounding box, or text input. Another advantage of SAM lies in its zero-shot approach, where it is not necessary to train a model with data from a specific domain to perform segmentation. This approach has been applied to various imaging modalities,⁹ such as CT,¹⁰ Magnetic Resonance Imaging (MRI),¹¹ and pathological images.¹² Although SAM results approach state-of-the-art performance, segmentation becomes challenging in more specific problems, such as weak boundaries, very small lesions, and complex, irregular shapes. This finding is particularly evident in pulmonary nodule segmentation. Therefore, it is necessary to train or fine-tune the SAM model with such images to achieve optimal results.

The training pipeline is another important aspect to highlight. In most segmentation problems, it is necessary to manually optimize and configure the entire process, including preprocessing, normalization, hyperparameters, and architecture configuration. This process must be performed for each dataset. To address these issues, the nnU-Net framework¹³ was published in 2021 by Isensee et al With nnU-Net, it is not necessary to manually configure and adjust the entire pipeline. Different components of the pipeline, such as hyperparameters and architecture configuration, adapt according to the properties of the dataset and image modality.

The nnU-Net framework automates configuring and training CNNs for medical image segmentation via a systematic approach. It begins with dataset feature extraction, known as data fingerprint, where the image modality and intensity value distribution are analyzed. Based on this data fingerprint, rule-based parameters are established, such as normalization tailored to each image modality. In Computed Tomography (CT) scans, global z-score normalization is applied. The framework then automatically determines the network topology and batch size based on the available GPU memory. The loss function and key hyperparameters, such as the learning rate, number of epochs, and optimizer, are fixed. Finally, the network is trained using the previous configuration.

In this paper, we propose a novel approach that combines optimization of the deep learning pipeline within the nnU-Net framework with the SAM model’s precise segmentation capabilities. Using nnU-Net, we automatically and efficiently adapt the deep learning pipeline to detect regions of interest in the image, where lesions are located. These relevant areas detected by nnU-Net are then used as input for the SAM model, eliminating the need for manual intervention and saving time. Moreover, with our proposal to extract regions of interest from the images locally, we avoid the previously mentioned problem of dealing with weak boundaries and small and complex lesions without the need for specific training or fine-tuning. This new approach was applied for lung nodule segmentation.

The remainder of the paper is organized as follows: Section 2 reviews the current state-of-the-art techniques for medical image and pulmonary nodule segmentation. Section 3 details the methodology used in this study, including the dataset, preprocessing techniques, and architectures employed. Section 4 presents the obtained results and compares them with other state-of-the-art methods. Finally, Section 5 offers conclusions and outlines future research directions.

Lung Cancer Overview

Lung cancer is a significant health problem and the leading cause of cancer-related death worldwide. The latest GLOBOCAN 2022¹⁴ estimates the incidence and mortality of different types of cancer and is produced by the International Agency for Research on Cancer (IARC). In addition, lung cancer is the leading cause of cancer-related death, accounting for more than 1.8 million deaths (18.7% of all cancer types). In 2024, 611 720 people died of cancer, of whom 125 070 (20.44%) died of lung cancer in the United States.¹⁵ In Europe, the situation does not improve.¹⁶ Lung cancer remains the leading cause of cancer-related deaths among men, with 153 032 predicted deaths. For women, the predicted mortality is 84 402 compared with the 76 041 deaths observed in 2018. The prediction for 2050 indicates that the number of cases diagnosed will increase. Therefore, early diagnostic methods are crucial to reduce disease prognosis and improve patients’ quality of life.

Lung cancer is diagnosed through physical examination, biopsy, or imaging using magnetic resonance imaging (MRI) and computed tomography (CT).¹⁷ A pulmonary nodule is an abnormal lung area. Pulmonary nodules are common findings, detected in approximately 30% of chest CT scans and 1.6 million patients annually in the United States. They are categorized as small solid (<8 mm), larger solid (≥8 mm), and subsolid, including ground-glass and part-solid nodules.¹⁸ At least 95% of all pulmonary nodules identified are benign, but the risk of malignant tumors increases with nodule size, from < 1% for nodules < 6 mm to 64%–82% for nodules >20 mm.^18,19 Nodules with >10 mm are considered large nodules, while nodules <3 mm are micronodules.²⁰ Other risk factors include patient age, smoking history, and nodule characteristics such as irregular borders and growth rate^21,22

Lung nodule segmentation solutions can be divided into 2 categories: traditional segmentation and deep learning methods. Morphological operations, active contours, and region-growing are common traditional segmentation methods.^23-25 Although traditional methods are resource-efficient, Deep Learning (DL) techniques provide superior results. Thanks to deep learning techniques, it is possible to extract relevant features that allow pixel-by-pixel classification in an image, thereby allowing for more precise segmentation. These techniques can be applied to different types of images, such as CT and histopathological images.^26,27 In recent years, autoencoder architectures, such as U-Net, have been used to solve this problem, specifically in medical imaging. Using these segmentation models, a radiologist can detect nodules that might otherwise go unnoticed, assist in the final diagnosis, and even study changes in nodule size over time. Lung nodule segmentation in computed tomography (CT) images remains a challenging task due to the variety of nodule shapes, sizes, and densities, as well as their similarity to surrounding structures. First, large databases with nodules with different characteristics are required. Due to the sensitive nature of such images, public databases are scarce. The most commonly used public databases are LIDC-IDRI²⁸ and LUNA16.²⁹ The segmentation model requires precise annotations from experts, which is time-consuming. These segmentation masks must be labeled correctly to indicate the nodule’s exact shape for generalization to the learning process. Recent innovations include multi-crop CNNs,³⁰ dual-branch networks,^31,32 and region-based fast marching methods.³³ Although these techniques have promising results, challenges remain in achieving high sensitivity with low false-positive rates, managing different types of nodules, and developing robust models applicable to diverse patient databases.³⁴

Related Works

Currently, lung nodule segmentation is challenging due to its complexity. There are 2 clear divisions when solving this problem: traditional segmentation methods and deep learning-based methods. Among the traditional segmentation methods, we found morphological operations, active contours, and region-growing.^23-25 However, although traditional methods are efficient and do not require as many computing resources, their results are considerably inferior to those of deep learning methods. Deep learning-based methods offer better results than traditional techniques due to the automatic extraction of image features. Deep learning methods are divided into 2D and 3D architectures. Most proposals in the literature work with 3D architectures to extract spatial information from voxels. Although in many cases these architectures perform better than 2D architectures, the results are similar. Additionally, 3D architectures involve much more design complexity and require more computational resources.

Among the most used architectures, U-Net¹ stands out, which achieves good results in medical image segmentation problems thanks to the use of skip connections. However, the base U-Net model fails to extract enough features for precise segmentation. To address these issues, several modifications to the architecture and the use of different preprocessing techniques have been proposed. Chaudhry et al³⁵ propose a 2D base model of U-Net and used transfer learning through pretraining on the LIDC-IDRI dataset²⁸ to segment nodules in the UniToChest dataset, achieving good results. We also observe the use of residual blocks in the literature to prevent relevant information loss and the use of Atrous convolution^36,37 to obtain multiscale features. In this way, nodules of different sizes can be detected.^38,39 Using these 2 techniques, Halder and Dey³⁸ achieve one of the highest segmentation results in the state of the art, with 97.15% of Dice score in cropped images from the LIDC-IDRI dataset. Zhou et al⁴⁰ introduced U-Net++, an extension of U-Net that incorporates additional depth levels, inter-level skip connections, and deep supervision to enhance multi-scale learning. Isensee et al¹³ proposed the nnU-Net framework, which focuses on optimizing preprocessing and hyperparameters instead of modifying the architecture. Therefore, they propose several 2D and 3D U-Net base architectures for segmentation problems of all kinds of medical images. Building upon this framework, our statistical analysis using the UniToChest dataset concluded that the inclusion of residual connections and the application of windowing as a preprocessing technique are significant factors in enhancing nnU-Net segmentation performance for pulmonary nodules.⁴¹

A hybrid approach is also common when addressing this problem. Nguyen et al⁴² propose a new backbone based on U-Net that allows nodule detection and segmentation, thanks to a multi-branch attention auxiliary learning mechanism. On the other hand, Jiang et al³¹ fused features from original images and images with features into a dual-branch network and a multidimensional fusion module. Wang et al⁴³ manage to extract features simultaneously in 2D and 3D CT images. Some studies combine traditional techniques with deep learning methods, as Yang et al⁴⁴ did. In this proposal, they first train a 3D U-Net architecture for segmentation. Then, the model is refined using active contours, achieving 90% of Dice score in the LIDC-IDRI dataset.

New techniques based on transformer networks have been proposed in recent years. The main idea of this approach is to use attention mechanisms to focus on relevant information in the image, improving the segmentation model’s accuracy. Hou et al⁴ propose SMR-UNet, which integrates self-attention mechanisms that extract multi-scale features and replaces U-Net’s convolutional blocks with residuals to avoid gradient explosion. Wang et al⁴⁵ use a swin transformer block to extract global information from the image by leveraging self-attention. Similarly, Cao et al⁴⁶ proposed Swin-Unet, a pure transformer-based U-shaped architecture that replaces convolutional blocks with shifted windows to capture long-range semantic information.

Preprocessing is necessary when working with CT scans before using a model. In many works, cropping is performed to extract the ROI of the nodule as preprocessing.^{4,31,37,43,45} By extracting the nodule’s ROI, better results are often obtained than using the complete CT image, as unnecessary information is eliminated. CT scans use Hounsfield units (HU), and many authors normalize these values by removing irrelevant information from the lung area.^3,35 Another relevant aspect of preprocessing is the prior segmentation of the lung parenchyma, the area where the nodules are located.^3,25,39 To segment the lung parenchyma, there are traditional algorithms like watershed⁴⁷ and accurate deep learning models like U-Net R231 in the lungmask package⁴⁸ with over 90% Dice score.

Among the proposals that leverage SAM for medical image processing, several approaches have been developed. Fine-tuned architectures, such as MedSAM⁸ extend the foundation model by incorporating diverse medical image datasets. To address the domain shift between natural and medical images, adapter-based methods have gained popularity; for instance, Wu et al⁴⁹ proposed a medical SAM adapter to integrate domain-specific knowledge without fine-tuning the entire heavy backbone. Similarly, recent works like DeSAM⁵⁰ focus on decoupling prompt learning to enhance generalization across different modalities. Other approaches eliminate manual user intervention by modifying the prompt encoder. Among these architectures is AutoSAM,⁵¹ which removes the prompt encoder of SAM and replaces it with a custom encoder trained on the same input image, thereby avoiding the need for specific fine-tuning. However, neither of the abovementioned methods optimizes the training pipeline for medical image segmentation as effectively as nnU-Net does.

Among the approaches that integrate nnU-Net with SAM, nnSAM⁵² stands out. This model combines both architectures by concatenating the embedding of the input image in SAM with the latent representation learned by the nnU-Net encoder. This integration leverages the knowledge encoded in the foundation model and the optimization capabilities of nnU-Net, allowing the combination of both models to exploit their respective advantages.

Few datasets are publicly available due to the sensitive nature of the data. LIDC-IDRI²⁸ is the most used database in the literature for segmentation, followed by LUNA16.²⁹ In 2022, new databases were launched such as UniToChest,³⁵ which contains more images than previous databases.

Methodology

Dataset

In this study, we used the database “The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI)”.²⁸ This dataset is one of the most widely used in the literature and is a reference database for lung nodule segmentation and classification. The LIDC-IDRI dataset contains 1018 CT scans obtained from 7 different academic institutions. Images were taken from various CT scanner models from different companies.

The dataset contains the following 3 main categories of nodules and annotations:

Nodules ≥ 3 mm: Lesions considered as nodules with the greatest in-plane dimension between 3 and 30 mm.

Nodules < 3 mm: Lesions considered as nodules with the greatest in-plane dimension less than 3 mm.

Non-nodule ≥ 3 mm: Other pulmonary lesions with the greatest in-plane dimension ≥ 3 mm were not considered nodules.

To annotate the images, 12 thoracic radiologists from 5 LIDC-IDRI institutions (Weill Cornell Medical College, University of California, Los Angeles, University of Chicago, University of Iowa, and University of Michigan) participated in the project. Each scan was independently reviewed by 4 of them in 2 phases. Each CT scan contains 1 or more nodules. Each nodule may be annotated by one, two, three, or 4 radiologists, as not all of them may agree on whether it should be considered a nodule. The first phase was a masked review, and the second was an unmasked review, in which each radiologist reviewed their mask along with the masks of the other radiologists. In this final phase, the radiologist could modify the mask and make a final decision.

The dataset contains 7371 lesions marked as nodules by at least 1 radiologist. Of these lesions, 2669 were nodules larger than 3 mm. Finally, 928 of the nodules larger than 3 mm, were marked by all 4 radiologists.

Preprocessing

For image processing, we used the pylidc library.⁵³ This library facilitates access to CT images, sized $512 \times 512$ , stored in DICOM format. In this study, we only considered lesions marked as nodules larger than 3 mm because these lesions are the most clinically relevant and have the highest probability of being malignant.

Regarding expert annotations, a consensus mask was generated, representing 50% agreement among all annotations (see Figure 1). In this way, we consider the opinions of all experts for model training. Additionally, since we work with 2D images and DICOM images are 3D, we extracted the volume of the annotation and obtained the central slice to avoid redundancy of nodules during training. In addition, the DICOM pixel data pixel values are raw values obtained from the scanner manufacturer. These values must be converted to HU for a correct representation of lung tissue. In this scale, lung parenchyma values range from −700 to −600, while water is 0. To convert the scanner values to HU, we used the following equation:

H U = P i x e l D a t a \times R e s c a l e S l o p e + R e s c a l e I n t e r c e p t

where PixelData represents the pixel values from the scanner, and RescaleSlope and RescaleIntercept are scaling and offset values required to convert the raw values to HU. In all images in the dataset, the RescaleSlope value is 1, while the RescaleIntercept value ranges between 0, −1000, or −1024, depending on the scanner manufacturer. These values are stored in the DICOM image metadata. The HU values were then normalized to the range [0, 1] to facilitate training.

Figure 1.

Consensus annotation masks representing the agreement among radiologists for the region of the nodule.

Because the LIDC-IDRI dataset excludes defined training, validation, and test splits, a random and stratified split was performed according to each nodule’s class. To ensure a representative evaluation, we utilized a fixed stratified split rather than k-fold cross-validation. This strategy strictly retains the underlying class distribution, ensuring that evaluation metrics are not artificially inflated by frequent or simple cases. Therefore, the stratified test set serves as a reliable and robust benchmark for assessing the model’s performance across diverse anatomical structures. The dataset of 2642 nodules was divided into 1902 for training, 265 for validation, and 475 for testing. The database contains pathological information for a limited number of patients. However, the DICOM metadata include information on the nodule’s malignancy, with 5 classes (from least to most malignant):

1: Highly unlikely

2: Moderately unlikely

3: Indeterminate

4: Moderately suspicious

5: Highly suspicious

Each expert labels each nodule with the corresponding malignancy grade. To study the reliability of inter-rater agreement, we calculated Cohen’s kappa for 2 radiologists and Fleiss’ kappa for 3 and 4 radiologists. The results were 0.1757 for 2 radiologists, 0.1721 for 3 radiologists, and 0.1940 for 4 radiologists, indicating very low agreement. Low concordance is common in challenging interpretation tasks, such as nodule classification, where ambiguity is high. In our experiments, the majority vote determines the final class of nodules. In some cases, a significant difference in opinion was observed, with the same nodule being marked as both class 1 and class 5. To address this issue, while creating the classification model for validation, a new class 0 was created to account for such cases.

Method Overview

Figure 2 describes the design of the proposed method, which is divided into 3 phases. Figure 3 illustrates the complete pipeline using an example. In the first phase, the proposed method inputs raw medical images. This image is fed into nnU-Net, which extracts the dataset characteristics, such as image modality and intensity distribution, to automatically configure the training pipeline, automating data normalization, U-Net architecture configuration, and some hyperparameters. In the nnU-Net architecture, the localization of medical anomalies is extracted, which generates a mask.

Figure 2.

Proposed method for lung nodule segmentation integrating nnU-Net and SAM. Given a CT lung scan, in the first phase, the lung nodules’ localization is obtained by training a nnU-Net. Next, in the second phase, the ROI of the nodules is extracted and the original image is cropped. Bounding box coordinates are extracted from the mask using an expansion factor. In the third phase, the bounding box coordinates and the cropped image are passed as input to the prompt and image encoders of the SAM, respectively. The SAM mask decoder returns a fine-tuned mask, which is reconstructed into a new mask to match the original image. The architecture design is based on Isensee et al¹³ and Kirillov et al.⁵⁴

Figure 3.

Complete end-to-end pipeline. Workflow of the proposed method illustrated with a concrete nodule example.

Next, the mask accuracy was improved using the foundation SAM model, which was trained on several images from different domains. This model requires manual user intervention by introducing a point bounding box or describing what is to be segmented. However, if an image with very small regions to segment compared with the total image is introduced, as is the case with lung nodules, precise segmentation is not achieved. To solve this problem, a local enhancement algorithm is proposed in the second phase of the proposed method. First, we extracted the region of interest (ROI) of the lesion, centering it in the image with a margin. A margin is left to add context to the lesion’s surroundings, allowing SAM to easily detect edge and shape differences.

Next, the binary mask generated by nnU-Net serves as a region proposal generator to automate the prompting process. Specifically, the OpenCV contour detection algorithm is employed to identify the lesion boundaries within the mask. From these contours, the spatial bounding box coordinates $(x_{m i n}, y_{m i n}, x_{m a x}, y_{m a x})$ are extracted, adjusted using an expansion factor of 0.1 (empirically determined), and fed into the SAM prompt encoder. This effectively translates the initial semantic segmentation into a precise spatial prompt without requiring manual intervention.

Finally, in the third phase, the SAM model receives the ROI image as input. The image embedding contains a representation of the image’s most relevant features. This embedding serves as the input to the SAM mask decoder, which generates a refined and precise mask. The refined mask is smaller than the original mask; thus, the mask is reconstructed by expanding its dimensions to match the original CT scan resolution ( $512 \times 512$ ) upon a zero-initialized canvas. In this way, the radiologist can obtain a complete image of the lesions. Through this method, radiologists do not need to input a prompt, saving time and effort. Post-processing was strictly limited to the spatial reconstruction of the full-resolution mask; no morphological operations or ensemble strategies were employed. This approach intentionally excludes heuristic refinements to ensure the evaluation reflects the segmentation performance of the proposed architecture.

Architectures

The nnU-Net Architecture

This method, specifically developed to work on semantic segmentation problems for medical images, stands out for its automatic configuration of the entire process of building the model. Preprocessing, architecture configuration, training hyperparameters, and postprocessing are all done automatically based on dataset features and heuristic rules. The main idea of this method is that the impact on performance is more significant in the pipeline configuration than in the modifications that can be made to an architecture.

Figure 2 shows the design of nnU-Net applied to our proposal. First, the dataset features (data fingerprint) are extracted. During this process, the image modality and intensity value distribution are checked. Next, rule-based parameters are established, such as normalization applied to each of the images. When working with CT images, a global z-score normalization is performed across the entire dataset, calculating the mean of the foreground and the standard deviation. The batch size, patch size and network topology are also determined based on the available GPU memory. The hyperparameters, such as the learning rate, optimizer, loss function, and data augmentation techniques are fixed.

The nnU-Net framework uses 3 architectures depending on the image type. All 3 architectures are based on the original U-Net configuration.¹ First, training is performed on a U-Net architecture for 2D images, followed by a 3D architecture with full image resolution. Finally, training is conducted with a cascaded 3D architecture for high-resolution images, performing segmentations at low resolution and refining them with the previous full-resolution model. The results of the 3 models were combined into an ensemble, and the final prediction was made after the postprocessing stage. In this study, we used 2D image only.

SAM

The SAM is a state-of-the-art foundation model for segmenting different shapes and objects. As a foundation model, it requires manual user intervention in the form of bounding boxes, mouse clicks, and text to extract the segmented region, which is its main disadvantage. SAM is trained on an enormous database comprising 11 million images from various domains and 1 billion masks. In addition to its accuracy, its main advantage is that it enables zero-shot learning. This allows the model to segment images that have not been seen before accurately. In the medical imaging context, the proposed approach can be generalized to different types of medical images.

Training Procedure

The experiments were conducted using a system equipped with an NVIDIA GeForce RTX 4070 GPU with 12 GB of memory, 128 GB of RAM, and a 12th Gen Intel^® Core™ i9-12900K CPU. For image configuration, the preprocessing explained in the Dataset and Preprocessing subsections has been carried out. Regarding the loss function and hyperparameters, default values are chosen in nnU-Net (see Table 1). The loss function is a combination of Dice and cross-entropy. The optimizer used is Stochastic Gradient Descent (SGD) with Nesterov momentum of 0.9. We have modified the number of epochs to 200. In this study, we used the ViT-B (Vision Transformer Base) model checkpoint. This variant was selected to maintain an optimal balance between segmentation accuracy and computational efficiency.

Table 1.

nnU-Net Configuration in the Experiments.

nnU-Net Configuration
Input size	$512 \times 512$
Loss function	Dice loss + Cross-entropy
Optimizer	SGD with Nesterov momentum of 0.9
Initial learning rate	0.01
Epochs	200
Expansion factor	0.1
Data augmentation	Rotation Scaling
	Gaussian noise
	Gaussian blur
	Brightness
	Simulation of low resolution
	Gamma augmentation
	Mirroring

Results and Discussion

To evaluate the performance of the proposed method, we used standard segmentation metrics, such as the Dice Similarity Coefficient (DSC) and Intersection over Union (IoU). To validate the contribution of SAM to segmentation accuracy, we conducted a comparative study with nnU-Net by removing the SAM refinement component from our proposal.

In Table 2, a comparison is made between the proposed method and the nnU-Net and YOLO11 instance segmentation models. In our evaluation, the nnU-Net model and the proposed approach achieved comparable results, with a difference of approximately 5% in segmentation metrics. Segmentation metrics DSC and IoU of our model were compared to nnU-Net using the Wilcoxon signed-rank test for paired samples, with each image as an observation. Results show that nnU-Net significantly outperforms our model (P < .01), indicating greater agreement with expert reference masks. Although nnU-Net demonstrated numerically higher metric scores, this can be attributed to its closer alignment with expert annotations, which may inherently contain errors or inconsistencies. In contrast, our proposed model prioritizes the refinement of nodule shapes, particularly at the edges, which better preserves nodule morphology. This focus on shape integrity ensures that the segmentation results are not only precise but also more representative of the true nodule morphology, even if it occasionally results in lower metric scores when compared to imperfect ground truth (GT) annotations. As shown in Table 2, although our method yields lower DSC scores due to boundary expansion, it achieves a higher Average Recall (0.6903) compared to nnU-Net (0.6841). This indicates that the proposed pipeline prioritizes minimizing False Negatives, ensuring that the entire tumor volume, including peripheral irregularities, is captured, which is a clinical priority for oncological diagnosis. We conducted a qualitative analysis by visualizing various nodules to further illustrate this advantage, as shown in Figures 4 and 5. These visual comparisons highlight how our approach produces smoother and more consistent boundaries, which are crucial for clinical applications where accurate shape representation is paramount. Compared to YOLO11, we evaluated the segmentation performance across various variants, with YOLO11m-seg yielding the best results. However, our proposal and nnU-Net significantly outperform YOLO11 in terms of segmentation metrics. This is primarily because YOLO failed to detect 260 of test images, substantially lowering its metrics. YOLO struggles to detect small or complexly shaped nodules, highlighting the advantage of our 2-stage method, which first segments and then classifies, particularly excelling with small or less visible nodules.

Table 2.

Results of Segmentation Model.

Metric	Our method	nnU-Net	YOLO11m-seg
mIoU	0.5346	0.5865	0.2136
mDSC	0.6150	0.6604	0.2678
Average Precision	0.5948	0.6745	0.3432
Average Recall	0.6903	0.6841	0.2311
Average F1-score	0.6138	0.6590	0.2678

Figure 4.

A visual comparison between models and GT. Lowest DSC and IoU values.

Figure 5.

A visual comparison between models and GT. The highest DSC and IoU values.

Figures 4 and 5 display the visualization of nodules with varying sizes and shapes. For a more detailed pixel-level analysis, Figures 6 and 7 present a comparison between the segmentation outputs of our proposed method, GT masks, and nnU-Net. Different colors are used to highlight specific differences: red pixels indicate regions detected exclusively by the GT mask, green pixels represent areas identified by nnU-Net, and yellow pixels correspond to regions detected by our proposed method that are not captured by either the GT or nnU-Net masks. The comparison was conducted by analyzing the lowest and highest metric values. The segmentation masks generated by nnU-Net and our model exhibit similarities. However, the proposed model demonstrates superior adaptation to nodule morphology, particularly in labeling border pixels that nnU-Net fails to detect. Compared with the ground truth, our model occasionally enhances expert annotations by completing pixels that belong to the nodule. These discrepancies with expert annotations lead to lower metric scores because the model’s output does not fully align with the manually annotated ground truth.

Figure 6.

Pixel-level comparison of the proposed method, GT annotations, and nnU-Net. Lowest DSC and IoU values.

Figure 7.

Pixel-level comparison of the proposed method, GT annotations, and nnU-Net. The highest DSC and IoU values.

Because segmentation is often a prerequisite for classification in CAD systems, we evaluated the impact of our segmentation model on the classification of nodules based on malignancy and compared its performance with nnU-Net and YOLO. To achieve this, we cropped the resulting masks using the SAM model to the corresponding cropped CT images. Then, we evaluated these images using several classification models to observe their performance. The number of classes was reduced to 3 by merging class 3 (indeterminate) with class 0 (created for nodules without a consensus classification). Nodules previously categorized as classes 1 and 2 were reclassified as class 1 (benign), whereas those in classes 4 and 5 were reclassified as class 2 (malignant) due to their high malignancy probability. Table 3 presents the results of the different models for expert-annotated masks, the nnU-Net model without SAM, and our proposed approach.

Table 3.

Classification Model Results with Applied Segmentation Masks.

Method	Metrics	Our method	Expert annotations	nnU-Net
ResNet50	AUC	0.8187	0.8412	0.8002
	F1	0.6261	0.6902	0.6296
	Precision	0.7291	0.7603	0.7334
	Recall	0.6046	0.6677	0.6004
	Accuracy	0.7287	0.7609	0.7311
VGG16	AUC	0.8110	0.8503	0.8054
	F1	0.6488	0.6912	0.6542
	Precision	0.7237	0.7430	0.7391
	Recall	0.6263	0.6821	0.6276
	Accuracy	0.7334	0.7554	0.7404
GoogleNet	AUC	0.8151	0.8475	0.8114
	F1	0.6569	0.6877	0.6528
	Precision	0.7338	0.7540	0.7343
	Recall	0.6372	0.6598	0.6330
	Accuracy	0.7405	0.7554	0.7400
EfficientNet B0	AUC	0.8223	0.8485	0.8105
	F1	0.6302	0.6854	0.6752
	Precision	0.7514	0.7612	0.7574
	Recall	0.6067	0.6690	0.6581
	Accuracy	0.7358	0.7584	0.7547

As shown in Table 3, while the overall Accuracy and F1-scores are comparable between the proposed method and the nnU-Net baseline, a consistent trend is observed in the Area Under the Curve (AUC) metric. Our method achieves higher AUC scores across all tested classification backbones, with EfficientNet B0 reaching the highest AUC of 0.8223 compared to 0.8105 for nnU-Net. This consistent improvement suggests that the segmentation masks generated by our pipeline preserve subtle boundary features that are diagnostically relevant. Although these features may penalize the geometric DSC and IoU scores due to mismatch with smoothed ground truth, they appear to provide valuable morphological cues that facilitate the distinction between benign and malignant nodules. Although Cohen’s and Fleiss’ kappa coefficients were low, the classification model achieved strong performance, indicating robustness to inconsistencies in expert annotations. Finally, Table 4 shows that YOLO11m-seg achieved an AUC of 0.7210, lower than the proposed pipeline. This performance difference is largely driven by the missed detections and segmentation challenges observed in the single-stage model for this specific dataset. In this context, the proposed hybrid approach demonstrated greater robustness for small lesion analysis, ensuring that difficult nodules were correctly segmented and classified where the general-purpose model struggled.

Table 4.

Classification Results on Detected Regions Using YOLO11m-seg.

Model	AUC	F1	Precision	Recall	Accuracy
YOLO11m-seg	0.7210	0.5910	0.6030	0.6060	0.637

The proposed methodology is general and can be easily adapted to other imaging modalities and tasks, such as skin lesion or organ segmentation in MRI. Adapting the approach would only require retraining nnU-Net with task-specific annotated data to learn the characteristics of the new imaging modality. This flexibility suggests that our framework could serve as a valuable tool across clinical automatic segmentation applications. Regarding the comparison with state-of-the-art models, unlike standard MedSAM workflows that rely on manual prompts, our proposal automates the process while retaining another clinical advantage: radiologists can intervene and manually adjust the bounding box if the automatic detection fails by nnU-Net. We maintained nnU-Net as the primary baseline due to its automation standards. While Transformer-based models such as Swin-Unet are valid alternatives, our hybrid pipeline is architecture-agnostic and allows for their potential integration as backbones in future work.

Finally, we acknowledge certain limitations in this study. A significant constraint in this research domain is the general scarcity of public datasets available for lung nodule segmentation. Furthermore, datasets that provide both pixel-level segmentation masks and malignancy classification attributes are exceptionally rare, which restricted our experimental validation to the LIDC-IDRI dataset. Additionally, the pipeline relies on the initial detector, meaning prompt variability can impact refinement. Future work will aim to evaluate the proposed framework on emerging datasets as they become available.

Conclusions

In this paper, we have introduced a novel approach for integrating the nnU-Net architecture with the SAM model. This proposal not only enables the automatic segmentation of lesions in medical images without requiring manual prompt by radiologists but also can be applied across medical imaging domains. Furthermore, by leveraging the nnU-Net architecture, there is no need for manual configuration of the architecture or training pipeline, thereby generalizing the problem and saving time.

The proposed approach was applied to pulmonary nodule segmentation. Pulmonary nodules are particularly significant because their early detection can improve the prognosis of the disease. In the proposed architecture, nodules of varying sizes can be detected for subsequent analysis.

The proposed model offers several design advantages. First, it is independent of the base architecture used to generate the initial segmentation mask. This means that if a modified version of nnU-Net or another segmentation architecture is employed, it will still work with SAM because bounding boxes for the region of interest are automatically extracted. Thus, training an encoder and integrating it into the architecture would not be necessary, thereby avoiding dependency. Second, the proposal focuses on the local-level region of interest. Instead of extracting bounding boxes for small lesions from images with a large amount of information, bounding boxes are extracted from the cropped image of the ROI. This allows the SAM model to achieve a more precise segmentation of the affected area.

The proposed segmentation model is particularly useful for the development of a complete CAD system for segmentation and classification using the generated masks. The results obtained from the various classification models indicate that the proposed model achieves performance comparable to that of an expert in classifying the disease.

Finally, in future work, we will continue to explore the application of this approach to different types of medical images, considering their various sizes and shapes. Other architectures will also be used to evaluate segmentation performance.

Footnotes

ORCID iDs

Alejandro Jerónimo

Ignacio Rojas

Olga Valenzuela

Author Contributions

Alejandro Jerónimo contributed to the investigation, conceptualization, methodology, data curation, visualization, software development, original draft writing, and review and editing of the manuscript. Ignacio Rojas was involved in conceptualization, formal analysis, funding acquisition, project administration, provision of resources, validation, supervision, investigation, and manuscript review and editing. Olga Valenzuela contributed to conceptualization, formal analysis, funding acquisition, project administration, resources, validation, supervision, and investigation.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Grants PID2024-160318OB-I00 and PID2021-128317OB-I00 funded by MICIU/AEI/ 10.13039/501100011033 and by ERDF, EU, and PCI2023-146016-2 funded by MICIU/AEI/ 10.13039/501100011033 and by the European Union

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

This paper uses LIDC-IDRI dataset²⁸ ().

References

Ronneberger

Fischer

Brox

U-net: convolutional networks for biomedical image segmentation. In: Navab

Hornegger

Wells

, et al., eds. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Springer International Publishing; 2015:234-241.

Shamshad

Khan

Zamir

, et al. Transformers in medical imaging: a survey. Med Image Anal. 2023;88:102802.

Chen

Wang

Tian

Yao

CT lung nodule segmentation: a comparative study of data preprocessing and deep learning models. IEEE Access. 2023;11:34925-34931.

Hou

Yan

Huang

Fan

Lin

Lung nodule segmentation algorithm with SMR-UNet. IEEE Access. 2023; 11:34319-34331.

OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774. 2024. DOI:10.48550/arXiv.2303.08774.

Team

Gemini: a family of highly capable multimodal models. 2024. DOI:10.48550/arXiv.2312.11805

Esser

Kulal

Blattmann

, et al. Scaling rectified flow transformers for high-resolution image synthesis. 2024. DOI:10.48550/arXiv.2403.03206

Han

You

Wang

Segment anything in medical images. Nat Commun. 2024;15(1):654.

Zhou

Zhang

Chang

, et al. Image segmentation in foundation model era: a survey. 2024. DOI:10.48550/arXiv.2408.12957

10.

Roy

Wald

Koehler

, et al. 2023, SAM.MD: zero-shot medical image segmentation capabilities of the segment anything model. 2023. DOI:10.48550/arXiv.2304.05396

11.

Putz

Grigo

Weissmann

, et al. The segment anything foundation model achieves favorable brain tumor autosegmentation accuracy on MRI to support radiotherapy treatment planning. 2023. DOI:10.48550/arXiv.2304.07875

12.

Deng

Cui

Liu

, et al. Segment anything model (SAM) for digital pathology: assess zero-shot segmentation on whole slide imaging. 2023. DOI:10.48550/arXiv.2304.04155

13.

Isensee

Jaeger

Kohl

SAA

Petersen

Maier-Hein

KH.

NnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods. 2021; 18(2):203-211.

14.

Bray

Laversanne

Sung

, et al. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2024;74:229-263.

15.

Siegel

Giaquinto

Jemal

Cancer statistics, 2024. CA Cancer J Clin. 2024;74(1):12-49.

16.

Santucci

Mignozzi

Malvezzi

, et al. European cancer mortality predictions for the year 2024 with focus on colorectal cancer. Ann Oncol. 2024;35(3):308-316.

17.

Loverdos

Fotiadis

Kontogianni

Iliopoulou

Gaga

Lung nodules: a comprehensive review on current approach and management. Ann Thorac Med. 2019;14(4):226-238.

18.

Mazzone

Lam

Evaluating the patient with a pulmonary nodule: a review. JAMA. 2022;327(3):264-273.

19.

Wahidi

Govert

Goudar

Gould

McCrory

DC.

Evidence for the treatment of patients with pulmonary nodules: when is it lung cancer?

Chest. 2007;132(3): 94S-107S.

20.

Sánchez

Benegas

Vollmer

Management of incidental lung nodules <8 mm in diameter. J Thorac Dis. 2018; 10(Suppl 22):S2611-S2627.

21.

Cruickshank

Stieler

Ameer

Evaluation of the solitary pulmonary nodule. Intern Med J. 2019;49(3):306-315.

22.

Shi

Zhao

Luo

, et al. Size of solitary pulmonary nodule was the risk factor of malignancy. J Thorac Dis. 2014; 6(6):668–676. DOI:10.3978/j.issn.2072-1439.2014.06.22

23.

Kostis

Reeves

Yankelevitz

Henschke

CI.

Three-dimensional segmentation and growth-rate estimation of small pulmonary nodules in helical CT images. IEEE Trans Med Imaging. 2003;22(10):1259-1274.

24.

Kubota

Jerebko

Dewan

Salganicoff

Krishnan

Segmentation of pulmonary nodules of various densities with morphological approaches and convexity models. Med Image Anal. 2011;15(1):133-154.

25.

Nithila

Kumar

SS.

Segmentation of lung nodule in CT data using active contour model and Fuzzy C-mean clustering. Alex Eng J. 2016;55(3):2583-2588.

26.

Allioui

Mohammed

Benameur

, et al. A multi-agent deep reinforcement learning approach for enhancement of COVID-19 CT Image segmentation. J Pers Med. 2022; 12(2):309.

27.

Mahmood

Owais

Noh

, et al. Accurate segmentation of nuclear regions with multi-organ histopathology images using artificial intelligence for cancer diagnosis in personalized medicine. J Pers Med. 2021;11(6):515.

28.

Armato

McLennan

Bidaut

, et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys. 2011;38(2):915-931.

29.

Setio

AAA

Traverso

de Bel

, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge. Med Image Anal. 2017;42:1-13.

30.

Sweetline

Vijayakumaran

Samydurai

Overcoming the challenge of accurate segmentation of lung nodules: a multi-crop CNN approach. J Imaging Inform Med. 2024; 37(3):988-1007.

31.

Jiang

Zhi

Zhang

Zhou

A dual-branch framework with prior knowledge for precise segmentation of lung nodules in challenging CT scans. IEEE J Biomed Health Inform. 2024;28:1540-1551.

32.

Cao

Liu

Song

, et al. Dual-branch residual network for lung nodule segmentation. 2019. DOI:10.48550/arXiv.1905.08413

33.

Savic

Ramponi

Peng

Lung nodule segmentation with a region-based fast marching method. Sensors. 2021;21(5):1908.

34.

Zhang

Jiang

Yang

, et al. Automatic nodule detection for lung cancer in CT images: a review. Comput Biol Med. 2018;103:287-300.

35.

Chaudhry

HAH

Renzulli

Perlo

, et al. UniToChest: a lung image dataset for segmentation of cancerous nodules on CT scans. In: Sclaroff

Distante

Leo

, et al., eds. Image Analysis and Processing – ICIAP 2022. Springer International Publishing; 2022:185-196.

36.

Koltun

Multi-scale context aggregation by dilated convolutions. 2016. DOI:10.48550/arXiv.1511.07122

37.

Selvadass

Bruntha

Sagayam

Günerhan

SAtUNet: series atrous convolution enhanced U-Net for lung nodule segmentation. Int J Imaging Syst Technol. 2024; 34(1):e22964.

38.

Halder

Dey

Atrous convolution aided integrated framework for lung nodule segmentation and classification. Biomed Signal Process Control. 2023;82:104527.

39.

Maqsood

Yasmin

Mehmood

Bukhari

Kim

An efficient DA-Net architecture for lung nodule segmentation. Mathematics. 2021;9(13):1457.

40.

Zhou

Siddiquee

MMR

Tajbakhsh

Liang

UNet++: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging. 2020; 39(6):1856-1867.

41.

Jerónimo

Valenzuela

Rojas

Statistical analysis of nnU-net models for lung nodule segmentation. J Pers Med. 2024;14(10):1016.

42.

Nguyen

Cao

, et al. MANet: multi-branch attention auxiliary learning for lung nodule detection and segmentation. Comput Methods Programs Biomed. 2023;241:107748.

43.

Wang

Zhou

Liu

, et al. Central focused convolutional neural networks: developing a data-driven model for lung nodule segmentation. Med Image Anal. 2017;40:172-183.

44.

Yang

Shao

Huang

, et al. Few-shot segmentation framework for lung nodules via an optimized active contour model. Med Phys. 2024;51(4):2788-2805.

45.

Wang

CA-UNet: convolution and attention fusion for lung nodule segmentation. Int J Imaging Syst Technol. 2023;33(5):1469-1479.

46.

Cao

Wang

Chen

, et al. Swin-unet: Unet-like pure transformer for medical image segmentation. 2021. DOI:10.48550/arXiv.2105.05537

47.

Nandeesh

Nagabushanam

Pramodkumar

Nandini

Lung parenchyma segmentation and nodule detection using deep learning. J Opt. 2024;53(1):635-642.

48.

Hofmanninger

Prayer

Pan

Röhrich

Prosch

Langs

Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem. Eur Radiol Exp. 2020;4(1):50.

49.

Liu

, et al. Medical sam adapter: adapting segment anything model for medical image segmentation. 2023. DOI:10.48550/arXiv.2304.12620

50.

Gao

Xia

, et al. DeSAM: decoupled segment anything model for generalizable medical image segmentation. 2024. DOI:10.48550/arXiv.2306.00499. ArXiv:2306.00499.

51.

Shaharabany

Dahan

Giryes

, et al. AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder, 2023. DOI:10.48550/ arXiv.2306.06370

52.

Jing

, et al. nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance, 2024. DOI:10.48550/arXiv.2309.16967. ArXiv:2309.16967 [cs].

53.

Hancock

Magnan

JF.

Lung nodule malignancy classification using only radiologist-quantified image features as inputs to statistical learning algorithms: probing the Lung Image Database Consortium dataset with two statistical learning methods. J Med Imaging. 2016;3(4):044504.

54.

Kirillov

Mintun

Ravi

, et al. Segment Anything. arXivorg 2023; DOI:10.48550/arXiv.2304.02643.