Revolutionizing tumor detection and classification in multimodality imaging based on deep learning approaches: Methods,applications and limitations

As previously said, medical imaging is crucial in diagnosing, prognosis, therapy, and monitoring numerous diseases and medical conditions. While traditional medical image analysis methods have demonstrated some effectiveness, they often heavily rely on the laborious and subjective process of manually extracting features and implementing manual algorithms. Unluckily, this approach is prone to time-consuming work, subjectivity, and human error. Nonetheless, with the rise of DL, medical image analysis has experienced significant advances, leading to the birth of automated algorithms capable of more accurate interpretation of complicated medical images [61]. The MMI approach acquires complementary and comprehensive insights into a patient’s condition. However, due to these data sets’ inherent complexity, heterogeneity, and high dimensionality, effectively harnessing the potential of MMI data presents a substantial challenge [62–64].

DL techniques have demonstrated immense potential in effectively tackling these challenges and have exhibited exceptional performance in many tasks in analysing medical images [65]. These groundbreaking methods can extract intricate patterns and establish intricate relationships from multimodal imaging data, ultimately enhancing accuracy in diagnosis, treatment planning, and patient prognosis. The primary objective of composing this all-encompassing review article is to provide readers with a comprehensive and exhaustive overview of recent advancements in DL-based MMI analysis methods. By conducting thorough investigations into integrating and amalgamating diverse imaging models, this review seeks to meticulously demonstrate and elucidate how DL algorithms can efficiently harness copious amounts of supplementary information from various modes, thereby augmenting the overall accuracy and resilience of medical image analysis.

Moreover, it is important to note that this comprehensive review article aims to delve deeply into the multifaceted challenges that arise in MMI analysis. The aim is to meticulously examine and deliberate upon these challenges to fully comprehend their intricacy and multifaceted nature. One of the main obstacles to MMI analysis lies in the inherent heterogeneity of the data, which presents a significant obstacle in integrating and interpreting the data. The wide-ranging diversity and variability within the data makes it arduous to effectively analyze and draw meaningful conclusions. Furthermore, the explainability of DL models presents another puzzling issue in MMI analysis. Understanding the underlying mechanisms and decision-making processes of DL models is crucial for their effective and responsible deployment in medical imaging. Overall, this review article aims to shed light on these problems faced in tumor detection and classification in the MMI field and give insight into potential remedies and possibilities for the future.

2.2 Objectives

The primary objectives of this review article on DL-based approaches for MMI analysis are twofold. The initial goal of this essay is to offer a thorough review of the cutting-edge DL methods utilized in MMI analysis. This paper will go deeply into the subtle aspects involving the amalgamation and fusion of several imaging modalities, including, but not limited to, MRI, CT, PET, and ultrasound imaging, to broaden our awareness and expertise in this field. The article aims to present a detailed understanding of how DL can effectively analyze MMI by examining the advancements in DL architectures, algorithms, and training strategies.

Secondly, the review article addresses the challenges and limitations of MMI analysis specifically around tumor detection and classification. It will go through the intrinsic heterogeneity and unpredictability of multimodal imaging data, which presents difficulties for data collection, fusion, and analysis. By highlighting these challenges, the article aims to inspire further research and innovation in developing solutions to overcome them and to encourage the creation of standardized benchmarks and datasets for MMI analysis.

Furthermore, the article outlines potential future directions and emerging trends in DL-based MMI analysis and improving patient outcomes. It will explore avenues for incorporating domain knowledge into DL models, leveraging transfer learning techniques to adapt models to domains and diseases, and integrating clinical data for a holistic analysis. By presenting these future directions, the article aims to provide researchers and practitioners with insights and guidance to advance the field further and drive improvements in MMI analysis.

2.3 DL-based MMI analysis

DL-based multimodal imaging has shown incredible promise in various clinical applications across medical specialities. This game-changing technology effectively uses deep neural networks (DNNs) great potential to completely examine and seamlessly integrate data from various imaging modes, substantially improving diagnosis accuracy and providing critical insights into the area of medical practice. Implementing this unique technique might benefit health sectors such as neuroimaging, cancer imaging, and cardiovascular imaging.

It has greatly impacted many medical domains in molecular imaging, musculoskeletal imaging, gastrointestinal imaging, lung imaging, ophthalmology, dentistry, maxillofacial imaging, obstetrics, and gynecology. This technology can transform how many diseases and disorders are diagnosed and tracked, ultimately improving patient outcomes and worldwide healthcare services. Researchers and practitioners are encouraged to use the meticulously compiled and voluminous evaluation literature displayed in Table 2 to get a comprehensive overview of the existing body of literature and research relevant to DL techniques employed in multimodal imaging.

Multimodal imaging has substantially improved the evaluation of heart function, notably by combining data from echocardiography, MRI, and CT scans. This comprehensive approach gives doctors a full view of the patient’s heart condition and allows them to make better-educated treatment decisions [66]. Moreover, DL technology in cancer imaging has provided new opportunities for researchers to examine PET-CT and PET-MRI data, allowing for observing molecular processes and malignant tumors. A recent study conducted by Laquan Li et al. (2020) proposed a novel method that utilizes full convolutional networks (FCN) to accurately segment tumors from PET-CT images, further showcasing the potential of DL in this field [55].

Neurotransmitter imaging: Utilizing multimodal molecular imaging can facilitate examining neurotransmitter activity and its correlation with neurological disorders. Orthopedics: DL models integrate MRI and CT data to enhance the evaluation of musculoskeletal injury, skeletal health, and surgical planning. Mouchess Maria et al. (2006) employed multimodal imaging techniques to evaluate tumor progression and bone resorption in mouse models of neuroblastoma. They could monitor tumor growth and bone loss by implanting luciferase-expressing human neuroblastoma cells in the femur using radiography, bioluminescence, micro-CT, and MRI. The administration of zoledronic acid inhibits tumor growth and prevents bone loss, while high-resolution MRI can detect distant metastases [67]. Liver disease assessment: The application of multi-modal medical imaging, based on DL, in liver disease assessment employs intricate algorithms. These algorithms can integrate and analyze data from diverse imaging techniques, enhancing accuracy and diagnostic capabilities. Xue Zhongliang et al. (2021) solved the predicament of liver lesion segmentation by utilizing a deep CNN in combination with multi-modal PET and CT scans. They proposed a model that could improve the interaction between features in different models, merge feature maps with varying resolutions, and introduce a similarity loss function to ensure consistency. This model surpassed the performance of baseline techniques in liver tumor segmentation and exhibited greater accuracy [68]. The detection of lung cancer is significantly enhanced through the analysis of combined PET and MRI scans using DL algorithms [56]. Monitoring fetal development is effectively carried out by utilizing ML-based and DL-based multimodal ultrasound and MRI data, thus providing comprehensive insights into fetal development, and detecting abnormalities during pregnancy [69]. Clinical practice transforms by implementing DL-based multimodal imaging, improving diagnostic accuracy, treatment planning, and patient care across various medical specialities. Anticipated advancements in DL algorithms and imaging technology are poised to augment its clinical application further.

DL has a significant role in tumor detection and classification across diverse multimodal imaging modalities. Analyzing multimodal data with DL models enhances diagnostic accuracy, facilitates personalized treatment plans, and enables early detection, thereby revolutionizing cancer management and improving patient outcomes. M. Attique Khan et al. (2020) [70] studied automated brain tumor classification using DL, which addresses radiologists’ challenges. The method incorporates contrast stretching, DL feature extraction, joint learning, and feature fusion. They validated their study on BraTS datasets and achieved high accuracies: 97.8%, 96.9%, and 92.5% for BraTS2015, BraTS2017, and BraTS2018, respectively.

G. Murtaza et al. (2020) [71] performed a study on DL-based breast cancer classification. Their review focuses on the DL-based classification of breast cancer using various medical imaging modalities. It systematically analyzes 49 studies, covering imaging modalities, datasets, preprocessing techniques, neural network architectures, and performance metrics. The review highlights challenges and future research directions in this domain, serving as a valuable resource for both beginners and advanced researchers in multimodality medical imaging. A study by EliasHossain et al. (2022) [72] proposes a strategy for brain tumor segmentation using 3D Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) scans with 3DU-Net Design and ResNet50. The ResNet50 achieved 98.96% accuracy, and the 3DU-Net scored 97.99%. Additionally, image fusion and a specific loss function improved segmentation accuracy. The models were validated using various metrics and integrated into a web server for practical deployment in healthcare settings.

4.1 Image classification

In recent years, there has been outstanding progress in DL-based image classification methods, signifying significant advancements in this domain. These methods have been effectively implemented in image classification tasks in multimodal medical imaging, yielding promising outcomes. One particularly noteworthy contribution in this field is the TransMed model, as proposed by Yin Dai et al. (2021), which specifically focuses on the classification of multimodal medical images. By capitalizing on the capabilities of CNNs and transformers, the TransMed model successfully addresses the existing limitations within this domain. The remarkable front of this model is its performance, which has far surpassed that of the CNN-based model, as demonstrated by two distinct data sets. TransMed’s achievements have significantly improved 10.1% and 1.9%, respectively. This exceptional accomplishment not only underscores the potential of the TransMed model but also instils optimism regarding its applicability to various tasks involving the analysis of medical images [73].

M Xiao et al. (2020) [74] addresses the challenges in diagnosing and classifying gliomas, the most common brain tumors. It introduces two convolutional neural network models: a 2D ResNet-based model for pathology image classification and a 3D DenseNet-based model for MRI image classification. These models achieved first place in the CPM-RadPath-2019 challenge for classifying different grades of gliomas. K Takahashi et al. (2022) [75] explore the utility of DL models in improving the accuracy of PET- CT image classification for breast cancer (BC). Using images with multiple degrees of PET maximum-intensity projection (MIP), DL models trained on 400 images showed promising results, outperforming radiologists in sensitivity and specificity.

Jiapeng Zhang et al. (2022) [76] introduces a DL-based method for classifying multiple organ-specific cancers using PET/CT images, aiming to assist radiologists in cancer screening. The proposed method incorporates a modality fusion module to fuse PET and CT images with segmented multi-organ information. Grayscale transformation and a double-level V-net are utilized for organ segmentation in low-dose CT images, addressing data annotation challenges and enhancing image context information. The classifier achieves an F-score of 82.3%, demonstrating its potential to aid radiologists in cancer screening across six classes.

4.2 Image segmentation

DL-based approaches have shown significant promise in improving the precision and efficiency of segmentation, marking an important milestone in the MMI field. It is worth noting that attention mechanisms have emerged as a notable breakthrough in this sector, promising to improve segmentation accuracy by selectively concentrating on pertinent characteristics. This promising aspect was underscored by the research conducted by Guo Zhe et al. (2019), who made a substantial contribution by introducing a supervisory MMI analysis method based on DL. Their study specifically concentrated on segmenting soft tissue sarcoma lesions using MRI, CT, and PET. Their investigation revealed that fusing different network levels can yield superior outcomes compared to solely analyzing single-modal images. This finding assumes paramount significance as it furnishes invaluable guidance for developing multi-modal image analysis techniques [45].

Multimodal medical image segmentation (MMIS) is a vital challenge in obtaining relevant information from several imaging data sources. Advances in this subject help to enhance clinical decision-making and provide a better knowledge of complicated biological and environmental events. MMIS pertains to the intricate and convoluted process of characterizing and delineating distinct anatomical structures or regions of interest that are visible in medical images produced using a variety of imaging modes. This complex endeavor necessitates the utilization of sophisticated image processing and analysis methodologies that aim to extract vast amounts of information-rich data from each pattern, followed by the harmonious integration of these data sets to generate precise and all-encompassing segmentation. The primary objective of MMIS is to enhance the accuracy and comprehensiveness of medical image analysis by harnessing the complementary advantages provided by different imaging models.

The endeavour of MMIS presents a formidable challenge due to the variations in image characteristics, intensity distribution, and spatial resolution observed across different modes. However, it also holds immense potential for harnessing each modality’s diverse and complementary information, thereby opening avenues for enhanced precision and adaptability in the generated segmentation outcomes. This perspective is fortified by the findings of earlier investigations [40, 77], which support the assertion mentioned earlier.

Several approaches have been developed for MMIS. One common approach is to perform modality-specific segmentations independently and then fuse the results using registration techniques to ensure spatial alignment. Another approach is to directly incorporate the information from multiple modalities into a unified segmentation framework by exploiting the similarities and differences in intensity patterns.

ML and DL techniques have also been effectively utilized in the field of MMIS, whereby they have demonstrated remarkable efficacy. These techniques leverage diverse and comprehensive information derived from various models to construct highly accurate models capable of classifying structures of interest with precision and accuracy. In particular, CNNs and other intricate architectures are frequently employed in these methods, as they can establish intricate nonlinear associations between input images and their corresponding segments [77–79]. The employment of ML and DL methodologies in MMIS has yielded significant achievements due to their ability to exploit the vast array of information accessible in different models. By virtue of these techniques, models can be trained to precisely and accurately segment structures of interest present in medical images. Furthermore, the performance of these models can be further augmented through the utilization of CNN or other advanced architectures, as they effectively assimilate the intricate nonlinear relationships between input images and their corresponding segments [77–79].

The segmentation outcomes derived from MMIS possess many practical and scholarly applications within clinical practice and research. These applications span a broad spectrum of disciplines, encompassing treatment planning, image-guided intervention, surgical navigation, disease diagnosis, and treatment response monitoring. In order to facilitate the process of quantitative analysis, accurate volumetric measurement, tracking of disease progression, and the effective utilization of computer-aided diagnostic systems, it becomes imperative to acquire precise and dependable segmentation results.

There is no doubt that the task of MMIS presents a formidable challenge, yet it undeniably holds paramount importance within medical imaging. This task capitalizes on the advantages of various imaging modalities to construct all-encompassing and exceptionally precise segments, which in turn provide invaluable aid in diagnosis, treatment planning, and patient management. The continuous advancement of sophisticated algorithms and technological innovations within this specific field has, without a doubt, played a significant role in propelling noteworthy progress in medical imaging, ultimately leading to enhanced levels of patient care and prognostic capabilities.

5.1 2D & 3D CNNs and YOLO based models

In this section, we will discuss the segmentation approaches that employ a DL model, specifically CNN and FCN, SegNet, RNet, U-Net, VNet, WNet and YOLO based models for tumor detection and classification. The advantages of CNNs over traditional ML models can be summarized as, (1) minimum preprocessing for input data requires. (2) automating the feature engineering (3) reducing the number of learnable parameters (Jose Dolz et al., 2018). It is one of the most often exploited DL models for the detection, classification, and segmentation model for medical images (J. Sun et al., 2021). 3D CNN was created to assess health images obtained from various modalities, such as MRI and CT. The fundamental benefit of using 3D CNNs is their capacity to immediately extract significant information about the spatial and temporal properties inherent in three-dimensional picture data. As a result, when compared to their two-dimensional counterparts, 3D CNNs have outperformed them. This performance improvement can be vividly exemplified through the notable work conducted by Mohammed Oves and his colleagues in the year 2023. Oves et al. successfully used DL technology to solve the complicated challenge of reliably identifying large-scale medical imaging data, including two-dimensional and three-dimensional pictures. The team’s proposed deep volume classification network showcases remarkable efficiency by skillfully amalgamating and integrating multiple depth features, culminating in highly commendable performance outcomes. Specifically, the area under the curve (AUC) value attained an impressive 93.66%, as the corresponding papers reported [81, 82] values.

Zhao et al. 2018, [83] presented 3D FCN for brain tumor segmentation, which included multi-task learning and feature fusion modules. In the multi-task learning module, VNet style architecture was used to extract high-dimensional characteristics from each imaging modality (PET-CT). The feature fusion module accepts the feature maps as input and executes feature re-extraction procedures using four convolution layers fusion networks. The proposed model’s performance was verified against State-of-The-Art (SOTA) VNet (Milletari et al., 2016 [84]), WNet (Xu et al., 2018 [85]), and fuzzy c-means clustering approaches using a lung cancer dataset of 84 patients. According to the results, the suggested co-segmentation approach achieved a DSC of 0.850 with classification and volume error values of 0.330 and 0.150, respectively.

YOLO has the ability to properly detect and accurately pinpoint the placement of objects inside an image or video frame. Tao Zhou et al., 2023 [86] proposed A Cross-modal Cross-scale Clobal-Local Attention YOLOV5 Lung Tumor Detection Model (CCGL-YOLOV5). They used stochastic gradient descent (SGD) with Localization Loss cost function for performance match. The CCGL-YOLOV5 model achieved an accuracy of 97.83%, a recall rate of 97.39%, an average accuracy of 96.67%, an F1 score of 97.61%, and a frames per second (FPS) of 98.59% on the multimodal lung tumor PET-CT data set. The experimental results demonstrate that the performance of the CCGL-YOLOV5 model in this study surpasses that of other conventional models. Moreau et al. 2021, [87] proposed UNet-based UNet _BL and UNet _FL models for breast cancer metastatic lesions segmentation and treatment response monitoring with longitudinal whole-body PET-CT scans as input. The baseline model UNet _BL was trained on baseline PET-CT images for segmenting baseline lesions, while the follow-up UNet _FL model was trained on four inputs: two for follow-up PET and CT, two for baseline PET, and output of UNet _BL as baseline lesions segmentation. The proposed models were evaluated on the EPICURE _seinmeta dataset consisting of 60 patients with 60 baseline and 104 follow-up images. The results explain that UNet _BL and UNet _FL achieved a mean DSC score value of 0.66 and 0.58, respectively.

Joonas Liedes et al. (2022) [88] study the the applicability of a convolutional neural network in detecting and auto-delineating Head and neck squamous cell carcinoma (HNSCC) from PET-MRI data. They employed a U-net model described in that was built using the TensorFlow Keras version 2.5.0 framework in Python version 3.7.10. The model was trained to identify the main tumor and any metastasis in the PET-MRI slices. In addition, models trained just with PET slices and models trained with augmented PET-MRI slices were created. The model based on PET-MRI attained a precision of 0.71 by employing a 9-pixel classification fidelity. It exhibited specificity and sensitivity values of 0.68 and 0.77, respectively. Conversely, the pure PET model displays a greater susceptibility to false positives, despite its sensitivity being comparable to that of the PET-MRI model. Models trained using augmented PET-MRI data yielded sensitivity, specificity, and precision values of 0.53, 0.77, and 0.65, respectively. The summary of the CNNs and YOLO-based network tumor detection models is presented in Table 5.

The Siam Network, a neural network architecture that has emerged as a viable solution for facilitating image registration and alignment of various modes, exhibits a distinct characteristic comprising two identical subnets. These subnets not only possess identical sets of weights but also possess the capability to encode two dissimilar images into fixed-length characteristic vectors. Leveraging this shared weighting scheme, the Siam network undertakes a comparative analysis of these characteristic vectors, thus enabling the determination of similarity between the two images.

In a recent investigation by Ahmed Sabeeh Yousif et al. (2022), the authors acknowledge the pressing need for accurate disease diagnosis within medical imaging. More specifically, they concentrate on the crucial task of multi-modal image fusion and endeavor to introduce an innovative approach that successfully amalgamates two potent techniques: sparse representation and orthogonal matching pursuit (OMP) with Siamese convolutional neural network (SCNN) methods. By seamlessly integrating these methods, the researchers aspire to tackle several pivotal challenges, including the augmentation of pixel positioning, the refinement of sparse characteristics, and the mitigation of undesirable artifacts. Ultimately, their proposed approach strives to yield superior fusion outcomes within the medical imaging domain when juxtaposed with previously employed methodologies [89].

Zhaisheng Ding et al. (2021), [90] proposed a study on tumor detection from PET-MRI, a critical aspect of medical imaging. They introduce a novel framework that combines the local extrema scheme (LES) and a Siamese network to improve efficiency and overcome limitations in tumor detection. Through extensive experiments and analysis, the proposed approach demonstrates superior performance compared to existing methods, offering promising advancements in tumor detection within medical imaging. Ning Xiao et al. (2022), [91] proposed a Siamese Pyramid Fusion Network (SPFN) and introduced feature pyramid transformation to the Siamese convolution neural network to extract multi-scale information from the fusion of PET and CT images to detect lung tumors. Their findings concluded that the proposed fusion-based Siamese method has a particular competitive performance in the quality improvement and information retention of PET-CT. A summary of the Siamese network-based models is presented in Table 6.

A specially designed model based on fusion has been meticulously created to effectively merge and amalgamate the extracted information from various models, resulting in a substantial enhancement of the precision and reliability of medical image analysis. “Image fusion generates an informative composite image, which can promote the performance of subsequent computer vision tasks. In this domain, multimodal medical image fusion has drawn increasing attention due to its significant clinical applications, including tumor segmentation, cell classification, neurological research, and treatment strategies for recurrent high-grade gliomas” [92]. A novel DL-based multimodal medical image fusion method via a multiscale adaptive Transformer called MATR was proposed by Wei Tang et al. (2022), [92] for analysing SPECT-MRI images from the Harvard database. They adopted seven representative and state-of-the-art methods for qualitative and quantitative comparisons. On the whole, when compared with the proposed MATR, the other seven algorithms have several drawbacks. The multi-modal deep neural network (MDNN) is a renowned example of a fusion-based model that has received significant attention and accolades for its outstanding capacity to smoothly integrate numerous layers of varied modes inside the depths of neural networks. Many references have validated and supported this exceptional result, further consolidating its fame and relevance in medical image analysis [34, 38, 40, 43, 63, 78, 81, 82, 89, 93–95].

Lei Bi et al. (2021), [96] propose a recurrent fusion network (RFN) for automatic PET-CT tumor segmentation. Their recurrent fusion network (RFN) consists of multiple recurrent multi-modalities down sampling (RMD) and up sampling (RMU) processes, which are connected via interconnect link modules (ILMs) backed with ResNet, DenseNet and 3D-UNet. According to their statement, the considerable improvement (> 5% in DSC) of RFN to 3D-UNet suggests that RFN can alleviate this training initialization limitation for 3D-based FCNs. A. Kumar et al. 2019, [54] checked CNN with fused inputs (FSs), multi-branch (MB) and multi-channel (MC), and found that CNN had a significantly higher foreground detection accuracy (99.29%, p <; 0.05) than the fusion baselines (FS: 99.00%, MB: 99.08%, and TC: 98.92%) and a significantly higher Dice score (63.85%) than the recent PET-CT tumor segmentation methods. Sebastian Jinu et al. (2022) [97] proposed a novel technique for fusing PET-MRI images using the YUV color space and wavelet transform. The fusion aims to integrate complementary information from both modalities for enhanced diagnosis. Comparative analysis suggests that the Dmey wavelet at decomposition level 3 and the maximum fusion rule yield optimal results for brain tumor detection. Quality assessment confirms promising outcomes for medical image fusion. The summary of the Fusion network-based models is presented in Table 7 with added articles referenced as [98–100].

In the vast realm of medical imaging, attention-based models have been subjected to extensive research and development to concentrate on pertinent features prevalent effectively and selectively in images procured from a myriad of models. These intricate models harness the power of attentional mechanisms, allowing them to direct their focus toward specific areas within an image that encompasses the utmost informational value of a given task. Through the utilization of attention mechanisms, attention-based models have proven to be exceedingly efficacious in their ability to discern and prioritize the most germane and pivotal regions present in images, consequently leading to a noteworthy enhancement in the overall performance and precision of medical image analysis and interpretation [101].

M Fallahpoor et al. 2023, [102] conducted a comprehensive investigation focused on using AI and DL on PET-CT images and its use in oncology, neurology, cardiology, and other emerging medical fields. Their review underscores the power of DL in PET-CT imaging, with successful applications in lesion detection, tumor segmentation, and disease classification with special focus on Attention-based models. Xiao Yang et al. 2022, [103] present a multimodality relation attention network (MMRACR-net) for breast tumor classification that uses consistency regularization. A multi-modality relation attention module (MMRAM) and a classification consistency module (CCM) are included in the suggested network. The MMRAM investigates the correlation information between two modalities to learn the attentive aspect of each. The CCM provides classification consistency by reducing the classification difference between the diffusion-weighted imaging (DWI) and apparent dispersion coefficient (ADC) image modalities. The summary of the Attention-Based Models is presented in Table 8 with added articles referenced as [104, 105].

GANs are commonly employed in image-to-image translation, where their primary objective is to ease image conversion between discrete modes. The generator and the discriminator are the two subnets that comprise the base GAN. Together, the generator and discriminator produce the desired results. The generator performs a critical part in ensuring the effective translation process by creating an image that closely resembles the target mode. On the other hand, the discriminator has been explicitly taught to discriminate between the produced image and the genuine one. This complex interaction between the generator and discriminator enables GANs to efficiently produce visually cohesive and contextually accurate translations across various languages.

In the field of MMI analysis, GAN has become a strong and significant technology, demonstrating its potential and effect. GAN makes synthesizing realistic and coherent multimodal images possible, representing a significant achievement in the field. This accomplishment is made feasible by teaching the generator to create synthetic samples that are so reliable that they can almost be mistaken for real images. Additionally, discriminators are taught to distinguish between genuine and synthetic samples with skill, which adds to the overall effectiveness of this system. GANs have a wide range of uses in multimodal analysis, with numerous potential applications. Multimodal image synthesis, which entails the production of missing modalities to acquire a comprehensive and holistic perspective of a patient’s overall status, is one of the main uses of GANs in this context. The capacity to more thoroughly assess and evaluate a patient’s medical condition makes this capability very valuable. The enhancement of domain adaptability is another essential function of GANs. Domain-invariant expressions can be learnt by aiding knowledge transfer from one modality to another or from a source domain to a target domain using the power of GAN. This is incredibly helpful when data gaps exist across different imaging modalities, scanners, or agencies or when annotated data is lacking. To advance the MMI analysis field, overcoming these obstacles and creating links between other disciplines [106–108].

A. Liebgott et al. 2019, [109] used 2D Conditional GAN backed with preprocessing techniques contrast-limited adaptive histogram equalization (CLAHE), Histogram Matching (HM) and Gaussian blurring (GB) on PET-CT images. The average mean square error (MSE = 0.7107) score was achieved. The summary of the GANs-Based Models is presented in Table 9 with added articles referenced as [51, 53, 110–112].

6.1 Most frequently used modalities and DL architecture in MMI analysis

According to our findings PET-CT is one of the most used multimodality imaging techniques in healthcare for tumor detection segmentation and classification compared to other, while Fusion and GAN based DL architectures are multimodality imaging analysis. Table 10 reveals some facts about different modalities and DL architecture with respect to published articles from 2019 to 2023.

Numerous research studies have documented substantial enhancements in the accuracy of multimodal image analysis utilizing DL in contrast to traditional approaches. As an illustration, in 2019, a study by Tohidul Islam et al. concentrated on the segmentation of brain tumors, where multimodal DL achieved an astonishing accuracy rate of 98%, surpassing the performance of traditional techniques [113]. The study also presented several compelling discoveries, some elaborated upon in Table 11.

DL methods that integrate data from diverse imaging modes are typically more advantageous compared to single-modal methods when it comes to performance and accuracy. In a comprehensive investigation conducted by Guo et al. (2018), scholars harnessed DL technology to identify soft tissue sarcomas by utilizing MMI, such as MRI, CT, and PET. To augment the process of segmentation, the researchers put forth an image fusion architecture, which is visually depicted in Fig. 9. Astonishingly, despite the potential challenges associated with robustness, the authors discovered that the most optimal outcomes were achieved by amalgamating information at the feature level [77]. Furthermore, the authors expounded on the efficacy of their proposed approach and showcased sample label maps that were generated by employing a type I fusion network that incorporated PET, CT, and T2 images (Fig. 10). As evidenced by the visualization, it becomes abundantly clear that the CNN model, which was trained on the accumulated data, effectively predicted the contour of the tumor area with exemplary precision.

OPEN IN VIEWER

Fig. 9

Illustration of the structure for (a) type-i fusion networks, (b) type-ii fusion network and (c) type-iii fusion network. The yellow arrows indicate the fusion location [45].

OPEN IN VIEWER

Fig. 10

Contour line of the ground truth annotation (yellow line) and labelmap (red line) overlaid on the T2-weighted MR image from one randomly selected subject [77].

According to a study by Nahed Tawfik et al. (2021), [114] intramodality fusion improves performance in multimodality imaging. The article highlights the significance of combining images from various imaging modalities to enhance image quality and clinical applicability. It discusses how medical image fusion captures complementary information from modalities like MRI, PET, CT, etc., aiming to address their limitations. While existing fusion methods have shown promising results, the article emphasizes the need for further advancements to tackle emerging challenges. This comprehensive survey serves as a valuable resource for researchers in the field, laying the groundwork for developing more effective fusion techniques to improve medical imaging applications.

Manoj Diwakar et al., [115] introduced a novel method for multi-modality medical image fusion using shearlet domain processing. Input images undergo non-subsampled shearlet transform (NSST) decomposition, with base and detail layers fused using a local extrema (LE) approach and Co-occurrence filter (CoF). High-frequency components are fused using sum-modified Laplacian (SML) for edge preservation. The comparative analysis demonstrates superior edge preservation with the proposed method over existing techniques, validated through subjective and objective evaluations on multi-modal medical image datasets.

According to Niharika S.D Souza et al. (2024) [116] multiplexed graphs approach using a graph neural network, which enables task-informed reasoning to effectively fuse information from multiple modalities. Evaluation of benchmark datasets and clinical data for Tuberculosis treatment outcome prediction and autism spectrum disorder classification demonstrates robust performance improvements over state-of-the-art methods.

Accurate breast cancer prediction and prognosis is very important for treatment planning and quality life of patients. Integrating multi-modal data like genomics and pathology images enhances predictive accuracy. Existing approaches face challenges: the Kronecker product technique is computationally expensive, and methods often overlook modality-specific relations. To address these challenges, Honglei Liu et al. (2024) [117] propose an attention-based multi-modal network that efficiently captures both intra-modality and inter-modality relations without high-dimensional features. their method outperforms existing approaches in breast cancer prognosis prediction.

The application of DL has demonstrated remarkable efficacy in various tasks, including segmentation and lesion localization. In a recent investigation conducted by Guo Zhe et al. (2019), the primary objective was to accurately outline the lesion profile of soft tissue sarcoma, a significant challenge in medical imaging. The researchers discovered that DL-based segmentation surpasses other existing methods, particularly when utilizing multi-modal images and integrating fusion techniques into networks [45]. The investigation’s findings in Fig. 11 include performance comparisons obtained through several model combinations. Moreover, Fig. 12 visually presents the obtained results when executing the same task using a single-modal network. These visual representations further emphasize the benefits of DL-based segmentation in lesion contouring and localization. Extensive research has established that utilizing multi-modal images alongside fusion techniques in DL networks constitutes an effective approach to enhance the precision and accuracy of lesion segmentation, thereby facilitating improved diagnosis and treatment planning for patients afflicted with soft tissue sarcoma. The results of this study profoundly contribute to the field of medical imaging and underscore the potential of DL to enhance the clinical prognosis of sarcoma patients. Overall, DL-based segmentation technology has raised as a valuable practice in medical imaging, offering a promising avenue for further research and development in this domain.

OPEN IN VIEWER

Fig. 11

Contour line of the ground truth annotation (red line) and segmentation result (yellow line). (a) Single-modality network on T2. (b) Multimodality network on T2 + PET (Type-I). (c) Multimodalities network on T2 + PET+CT. (d)–(f) 3-D surface visualization of the segmentation results in (a)–(c) [45].

OPEN IN VIEWER

Fig. 12

Box chart for the statistics (median, first/third quartile and the min/max) of the DICE coefficient across 50 subjects. Red box stands for network train and test on Type-I network, blue box stands for Type-II network, and green stands for Type-III. Performances of single-modality network are shown as gray boxes to the left for reference [45].

The selection of PET-CT, PET-MRI, and SPECT-CT modalities for the purpose of medical image analysis is contingent upon the precise clinical scenario, the requisite information, as well as the inherent merits and constraints associated with each model. Each individual model possesses distinct advantages and is typically determined based on the precise clinical investigation at hand.

PET-CT combines functional and anatomical information in a single scan and is excellent for cancer imaging, as it provides metabolic information (PET) along with detailed anatomical localization (CT). It is widely used for cancer staging, treatment planning, and monitoring. Very high radiation exposure due to the CT component is one of the shortcomings of this technology. It has limited soft tissue contrast compared to MRI. PET-MRI provides functional information (PET) and excellent soft tissue contrast (MRI) with no ionizing radiation exposure in the MRI component. It is an ideal technology for imaging soft tissues and neurological conditions. Longer scan times compared to PET-CT, limited availability and higher cost are the shortcomings of this technology. SPECT-CT combines functional information (SPECT) with anatomical localization (CT) and is widely used in nuclear medicine for various applications, including bone scans and myocardial perfusion imaging with lower radiation dose compared to PET-CT. Limitations: Lower spatial resolution compared to PET. Limited in soft tissue imaging compared to MRI. Selection Criteria: Cancer Imaging: For comprehensive cancer imaging, especially in staging and treatment monitoring, PET-CT is often preferred due to its ability to combine metabolic and anatomical information. Neurological Imaging: PET-MRI is valuable for neurological conditions where soft tissue contrast is crucial. It avoids ionizing radiation and provides detailed information about brain structures and functions. Bone Scans: SPECT-CT is commonly used for bone scans, providing functional information about bone metabolism along with precise anatomical localization. Soft Tissue Evaluation: In scenarios where soft tissue evaluation is a priority and ionizing radiation is a concern, PET-MRI may be preferred.

The ultimate determination hinges upon the clinical indications, the precise information requisite, and the equilibrium between the advantages and disadvantages of each paradigm. In various clinical scenarios, the progress in technology and the continuous investigation may sway individuals’ inclinations towards a particular model as opposed to another. It is of utmost importance for healthcare professionals to meticulously mull over the distinct prerequisites of individual patients when ascertaining the most suitable imaging model.

The selection of a DL framework for the purpose of identifying and categorizing tumors in multimodal imaging is contingent upon various factors, such as the distinctive characteristics of the data, the intricacy of the undertaking, and the available resources. Different architectures have shown success in different scenarios. Here’s a brief overview of each type:

Convolutional Neural Networks: CNNs are well-suited for image-based tasks and have succeeded highly in tasks such as image classification and segmentation. They automatically learn hierarchical features from the input data, making them effective for capturing patterns in medical images. CNNs can be applied to each modality individually or to fused multimodal images for tumor detection and classification.

Siamese Networks: Siamese Networks are designed for tasks like image similarity and dissimilarity. They can be used for comparing and contrasting different regions within the same or across different modalities. Siamese Networks can be beneficial when comparing corresponding regions in multimodal images to identify tumor characteristics.

Fusion-based Networks: Fusion-based networks combine information from multiple modalities to enhance overall performance. They leverage complementary information from different modalities, improving the model’s ability to detect and classify tumors. When dealing with multimodal imaging, fusion-based networks can effectively combine PET, CT, MRI, or other modalities to provide a more comprehensive analysis.

Attention-based Models: Attention mechanisms allow the model to focus on specific regions of interest, improving interpretability and performance. It is useful for emphasizing relevant information in multimodal images for tumor detection. Attention-based models can be applied to highlight important features in each modality or guide the fusion process in multimodal tumor classification.

Generative Adversarial Networks: GANs can be used for generating synthetic data, potentially augmenting limited datasets in medical imaging. They may aid in enhancing the quality of images, which can be valuable for training more robust tumor detection models. GANs can be used for data augmentation or for generating realistic images that simulate various tumor characteristics.

Choosing the Best Model: The choice depends on the nature of the imaging data, the availability of labeled samples, and the specific requirements of the task. For multimodal imaging, fusion-based networks and attention-based models are often beneficial to leverage the complementary information from different modalities. Siamese Networks might be useful for tasks involving direct comparisons between regions or structures in different modalities. Data availability, computational resources, and interpretability considerations should also guide the selection. Ultimately, the “best” model depends on the specific context and requirements of the tumor detection and classification task in multimodality imaging. Experimentation and comparative evaluations on the specific dataset of interest are often necessary to determine which model performs optimally.

6.6 Clinical validity

The clinical effectiveness of DL models holds immense significance due to their pivotal role in determining their applicability and dependability. A study conducted by Yinhao Wu et al. (2022) focusing on the classification of skin cancer has shed light on the potential and practicality of DL in the field of clinical practice, thereby showcasing its viability. Nonetheless, it is imperative to highlight that further validation is indispensable to ascertain the strength and reproducibility of these models, especially when considering different cohorts of patients. The requirement for additional validation arises from the acknowledgment that the efficacy and effectiveness of the DL model can potentially vary across diverse patient groups, thereby necessitating a comprehensive evaluation and validation process to ensure its clinical utility [118]. Recent interest in AI, machine learning, and DL in cardiovascular medicine promises personalized care through advanced CV imaging applications [119]. DL models can simultaneously analyze diverse imaging modes, thereby enabling the enhancement of tumor diagnostic accuracy through the utilization of additional data furnished by each model. In the realm of oncology, the amalgamation of MRI and PET images serves to furnish a more thorough comprehension of tumor characteristics, exemplifying this phenomenon. For instance, DL algorithms can accurately delineate tumor boundaries across MRI, CT, and PET images, facilitating precise radiotherapy planning [120–124].

A study by Ahmed Hosny et al. (2022) [125] presents a comprehensive clinical validation strategy for DL models in segmenting primary non-small-cell lung nodules (NSCLC) tumors and lymph nodes from PET-CT images. It includes interobserver and intraobserver benchmarking, primary validation on multiple datasets, functional validation, and end-user testing. Results demonstrate improved segmentation accuracy over interobserver benchmarks, consistency with intraobserver benchmarks, and equivalent radiation dose coverage compared to expert segmentations. Despite variations across datasets, the models show potential clinical utility by reducing segmentation time and interobserver variability. This study highlights the importance of evaluating clinical utility beyond geometric segmentation metrics.

According to a study by Zhaoshuo Diao, et al. (2021) [126] focuses on enhancing the clinical validity of tumor delineation in PET-CT images. Existing segmentation methods often lack consideration for uncertainty and consume significant computing resources. To address this, the proposed evidence fusion network (EFNet) reduces uncertainty by separately outputting PET and CT results and fusing them through evidence fusion. EFNet simplifies the network architecture, achieving improved segmentation results and enhancing efficiency compared to existing methods. Experimental validation on soft-tissue sarcomas and lymphoma datasets demonstrates significant improvements in segmentation accuracy, enhancing the clinical utility of PET-CT imaging for tumor delineation.

DL algorithms can analyze large volumes of multimodal images rapidly, potentially reducing the time required for image interpretation. This can enhance workflow efficiency in clinical practice, allowing radiologists to focus more on complex cases or patient care. According to a study performed by Amirhossein Sanaat et al. (2021) [127] summarize that DL techniques, specifically CycleGAN, successfully synthesize full-dose PET images from low-dose scans, aiding in reducing radiation exposure and enhancing patient comfort. The study demonstrates comparable diagnostic quality and tumor detectability between synthesized and full-dose images, validating the efficacy of DL in PET-CT.

Bonardel Gerald et al. (2022) [128] evaluates the clinical validity of a DL-based denoising algorithm, SubtlePET, applied to PET/CT images acquired at reduced count statistics compared to regular acquisition. Phantom and patient images from three different PET-CT scanners were assessed. Results indicate that SubtlePET effectively denoised images acquired at reduced count statistics while maintaining lesion detectability and SUVmax values. The study suggests that SubtlePET could be applied in clinical practice for PET-CT acquisitions with reduced injected doses, aligning with European recommended injected dose guidelines, without compromising diagnostic confidence.

A study by Yngve Mardal et al. (2021) [129] focuses on evaluating the clinical validity of convolutional neural networks (CNNs) for automatically delineating gross tumor volumes (GTV) in head and neck cancer (HNC) patients using FDG-PET/CT images. By comparing CNN-generated delineations to manual ones by specialists and introducing new structure-based evaluation metrics, the study demonstrates the accuracy and reliability of CNN models in accurately identifying GTVs. Importantly, the study emphasizes the significance of PET/CT imaging in improving delineation accuracy, with PET/CT-based CNN models outperforming those based solely on CT images. Overall, the findings underscore the clinical utility of CNNs in enhancing tumor delineation in HNC patients, providing valuable insights for improving treatment planning and patient outcomes.

6.7 DL network computational demands

DL requires substantial computational and memory resources throughout the training and inference phases, but its accuracy remains high. To fulfil these resource demands, cloud computing has become a conventional technique despite presenting obstacles concerning data migration. To expedite the process of DL inference, specialized hardware solutions like ASICs (such as Google’s TPU and ShidianNao) and FPGA-based DNN accelerators have emerged, delivering exceptional energy efficiency.

In 2019, Chen Jiashi and her colleagues authored a publication that delves into the intricate discussions surrounding the application, implementation, and training of DL with edge computing. This comprehensive study addresses the numerous challenges in system performance, network technology, benchmarking, and privacy. The notion of edge computing, which is viewed as a very promising method with the ability to efficiently address the computational and low latency needs of DL on edge devices, is of great interest. Given this context, it becomes imperative to initiate an in-depth investigation into the various techniques employed to safeguard invaluable privacy concepts within the complex and intricate process of transferring substantial volumes of data between edge devices and the ethereal realm of the cloud. By meticulously examining these technologies, we can acquire invaluable insights into effectively alleviating privacy concerns and seamlessly integrating DL at the network edge. These earnest efforts will undoubtedly significantly contribute to advancing the field and pave the way for the widespread adoption of DL in edge computing environments [130]. An overview of the taxonomy of these strategies, with examples of various scenarios that will be described in further depth in Table 12.

Interpretability remains a concern. Some studies, like Saad Bin Ahmed et al. (2022) on explainable AI (XAI) in chest X-ray analysis, raise issues crucial for clinical acceptance. There is an overall lack of universally accepted quantitative evaluation metrics for XAI techniques, so additional research in this direction is needed [131].

6.9 Transfer learning

Transfer learning techniques, as discussed by Yi Li et al. (2023) in their review, enable knowledge transfer from one dataset to another, reducing the need for large, annotated datasets. In their work, they presented a unique multimodal dataset for glaucoma diagnosis and classification (GMNNnet) and proposed a new multimodal neural network of transfer learning with little training data for glaucoma diagnosis and classification. GMNNnet performs well in capturing complex glaucoma information from multiple modalities, offering promising results for accurate diagnosis and classification of glaucoma [132]. Parvin Razzaghi et al. (2022), Multimodal brain tumor detection using a multimodal deep transfer learning model approach significantly outperforms the comparable approaches. Their study shows that the multimodal feature encoder and multimodal domain adaptation technique successfully learn and transfer knowledge, as shown in Table 13 [133].

Transfer learning and domain adaptation approaches can assist models in overcoming the limitation of limited data by allowing them to harness information learnt from comparable datasets or modalities. These techniques have become instrumental in addressing challenges related to data scarcity, domain adaptation, and improving the generalization of DL models in MMI analysis. These strategies use information learned from previously trained models on one job or domain to enhance performance on a related task or domain with little data.

Benchmark data sets are frequently utilized to evaluate DL models in the domain of MMI, playing a pivotal and crucial role in assessing and comparing the performance of these models. The utilization of these data sets for this specific objective has been comprehensively explained and expounded upon in Table 14. These data sets encompass a vast array of medical imaging models and clinical applications, thereby presenting an invaluable and highly significant resource for developing and evaluating DL models in the realm of multi-modal imaging research. Scholars and researchers commonly depend on these data sets to gauge the effectiveness and accuracy of algorithms associated with many tasks, including but not limited to classification, segmentation, tumor and other disease diagnosis. Using these esteemed benchmark data sets, researchers can effectively and efficiently assess the strengths and weaknesses inherent in their DL models, refining and optimizing their performance meticulously and discerningly.

Using DL-based methods for MMI analysis has numerous potential advantages compared to traditional ML methods. The utilization of DL models allows for auto-learning and intricate feature extraction from medical images, eliminating manual feature engineering. This characteristic has proven exceptionally beneficial in medical imaging, where images often contain substantial noise levels and are influenced by various imaging artefacts. DL models can handle noise and variability, enhancing MMI task performance. Moreover, DL models can effectively integrate information from different imaging modes, improving accuracy and diagnostic performance.

This scholarly article extensively investigates the dynamic and ever-evolving landscape of DL in MMI tumor analysis. It uncovers DL significance and progress that has been made in this domain. Healthcare workers may merge information from many imaging techniques, such as MRI, CT, PET, and others, by leveraging the power of DL. This integration empowers the DL model to exhibit an elevated ability to provide accurate diagnosis of tumors. Undoubtedly, this emerging trend marks a substantial advancement in the field of MMI analysis, offering tremendous potential for more precise and comprehensive patient care.

7.1 Main challenges in MMI analysis

MMI analysis presents challenges due to data heterogeneity, limited annotated datasets, and the need for interpretability. Integrating diverse imaging modalities while preserving their unique features is complex. Domain adaptation is required to handle variations between datasets. The lack of generalization and computational complexity are additional hurdles [62, 63, 80]. Some of the key challenges, as in Fig. 13, are as follows:

OPEN IN VIEWER

Fig. 13

Challenges in multimodal medical image analysis.

Data heterogeneity: Distinct imaging techniques capture distinct anatomy and physiology components, resulting in image quality, contrast, and noise variances. Integrating and analyzing these disparate data kinds necessitates using specific algorithms and models to manage multimodal data heterogeneity [134].

Limited availability of annotated multimodal datasets: Annotating medical images with ground truth data takes time and resources, especially for multimodal datasets. The scarcity of large-scale annotated datasets across multiple imaging modalities hinders development and evaluation of DL models for MMI analysis [135].

Interpretability and explainability: DL models are often black-box models, particularly those based on complex architectures such as DNNs and GANs. Understanding the decisions made by these models and explaining their predictions to clinicians and patients is challenging. In medical settings, interpretability and explainability are crucial for gaining the trust and acceptance of AI-driven image analysis [135, 136].

Integration of multimodal information: It is challenging to effectively fuse information from several imaging modalities while keeping their distinct properties. It is a huge issue to develop effective multimodal fusion algorithms that may leverage complementary information from distinct modalities without sacrificing critical features [137].

Domain adaptation: Variations in imaging protocols, hardware, and patient populations can lead to domain shifts when working with data from different sources or institutions. Adapting DL models to handle these domain shifts and maintain performance across diverse datasets is a critical challenge in MMI analysis [138].

Limited generalization: DL models trained on large datasets may not generalize well to new patients or clinical scenarios. The generalization performance of multimodal models must be carefully validated to ensure robust and reliable performance in real-world clinical settings [92].

Computational complexity: DL models for multimodal analysis can be computationally intensive, requiring significant computational resources and time for training and inference. To make these models practical, efficient model architectures and optimization techniques are essential for clinical use [139]. Addressing these challenges requires collaborative efforts among researchers, clinicians, and policymakers. Advances in data sharing, standardization of imaging protocols, and the development of explainable and interpretable AI or DL models will play a vital role in overcoming these challenges and unlocking the full potential of MMI analysis for improved patient care.

Despite the promising potential of DL models in multimodality imaging, several challenges exist, including the need for large and diverse datasets for training, interpretability of DL-based predictions, and integration into existing clinical workflows. Additionally, DL models may encounter difficulties in handling data heterogeneity, variability in imaging protocols, and generalization to unseen patient populations.

7.3 Future directions and emerging trends

From the literature we presented on the role of DL in MMI analysis, we identified some emerging trends and provided some insight into future directions for research development in this field. Some of them are presented below, and we hope that this information will serve as a guide for future researchers. An overview of this guidance and emerging trends can be found in.