Combining Ultrasound Imaging and Molecular Testing in a Multimodal Deep Learning Model for Risk Stratification of Indeterminate Thyroid Nodules

Methods

Results

There were 333 patients with indeterminate thyroid nodules who met this study’s inclusion criteria (259 benign and 74 malignant). A majority of patients had Bethesda III pathology (57.4% vs. 42.6% with Bethesda IV). Papillary thyroid cancer (PTC) was the most common malignancy (47% of patients), while NIFTP was present in 35% and follicular thyroid cancer in 15% (Table 1). The performance of the baseline and different model configurations is in Supplementary Data. The models’ predictions aggregated over all five folds were used to plot the receiver operating characteristic (ROC) curves shown in Figure 1a. The baseline MT + BE model achieved an area under the receiver operating characteristic curve (AUROC) of 0.728 [0.68, 0.78] with a sensitivity of 0.946 [0.88, 1.00] and specificity of 0.664 [0.60, 0.73], consistent with prior studies. ⁶ We tested multiple configurations, including whole-frame and patch-based models, with detailed results provided in Supplementary Data. The ensemble model outperformed all others, achieving the highest AUROC of 0.831 [0.77, 0.89], PPV of 0.477 [0.46, 0.50], and specificity of 0.703 [0.66, 0.75], while maintaining the same sensitivity as the clinical baseline. These improvements were statistically significant (p = 0.0008). Attention maps from the patch 256 model are shown in Figure 1b and c.

OPEN IN VIEWER

FIG. 1.

Results of the AMIL framework. (a) ROC curves comparing different model configurations. (b, c) Patch 256 attention maps of representative US scans. In (b), there is a nodule in the right lobe of the thyroid with the attention map including the majority of the nodule. The deep border has a notable rim of hyperechogenicity that is partially contained within the attention map. In (c), there is a large thyroid nodule in the center of the frame, containing a central solid component with hypoechoic peripheral areas. Notably, the attention map is located directly over an area containing isoechoic and hypoechoic nodule components at the periphery as well as the hyperechoic interface between the nodule and the deep surrounding tissue. AMIL, attention multiple instance learning; BE, Bethesda score; ROC, receiver operating characteristic; US, ultrasound; WF, whole-frame images.

Discussion

We developed an AMIL model combining US images and MT to classify indeterminate thyroid nodule malignancy, retaining MT’s high sensitivity while reducing false positives. Minimizing false negatives is critical to avoid delayed cancer diagnoses, while reducing false positives prevents unnecessary surgeries and associated morbidity. By matching MT’s sensitivity, our model aimed to reduce misclassified malignancies, which would lead to fewer false positives associated with MT in clinical practice. Although the PPV improvement is modest (0.477 [0.46, 0.50] vs. 0.448 [0.41, 0.48]), with over 120,000 indeterminate biopsies annually in the United States, ^16,17 even small reductions in false positives could greatly benefit patients and health care systems.

Several studies have utilized deep learning to predict malignancy in indeterminate thyroid nodules, focusing on either pathology slides or using only US studies. ^{18 –20} To our knowledge, this is the first deep learning framework designed specifically to augment MT performance by reducing false positives. MT platforms report varying sensitivity and PPV depending on the test type and patient characteristics. In our cohort, the PPV of MT alone was relatively low (0.448 [0.41, 0.48]), but our framework improved PPV on the same group of patients. Given MT’s widespread clinical use, enhancing its performance represents a more achievable and clinically relevant goal than replacing it entirely with deep learning models. Combining the Bethesda score with MT increased the AUROC from 0.627 (MT alone) to 0.728 [0.68, 0.78], indicating that biopsy results complement MT. However, other metrics showed no significant improvement as the optimal decision point primarily relied on MT. While digitized cytology slides could offer more information that the Bethesda classification might overlook, they are not routinely digitized due to time and cost constraints.

This study has several key limitations that warrant discussion. As a single-center study without external validation, results may reflect practices specific to our institution and lack generalizability. Additionally, we were restricted to analyzing patients for whom US images were available in our database, excluding those with outside imaging or imaging missing for other reasons. We also assumed that patients who did not undergo surgery had benign pathology, an assumption consistent with prior literature studying MT. However, this assumption introduces potential bias, as institutional differences in Bethesda class rate of malignancy may affect the classification of indeterminate nodules. Some of these patients may have had malignant pathology but did not undergo surgery due to factors such as advanced age, medical comorbidities, or sub-centimeter PTCs. This is particularly relevant given that 34% of our “benign” patients had a suspicious MT result. Additionally, NIFTP nodules, which accounted for 35% of the nodules in our cohort, differ from other cancers in terms of presentation, clinical importance, and treatment, making it valuable to distinguish them separately in future analyses. Finally, our dataset was limited to a binarized version of MT output (suspicious or benign). In clinical practice, recent MT reports often include probability ranges of malignancy risk that guide diagnostic decisions. However, reports from different companies use varying methods to compute and present these probabilities, and older reports lack this information entirely. Incorporating such granular data could enhance our model’s ability to build upon MT results and could be explored in future efforts using deep learning approaches.