Brain multi modality image inpainting via deep learning based edge region generative adversarial network

Abstract

A brain tumor (BT) is considered one of the most crucial and deadly diseases in the world, as it affects the central nervous system and its main functions. Headaches, nausea, and balance problems are caused by tumors pressing on nearby brain tissue and affecting its function. The existing techniques are challenging to analyze diseased brain images since abnormal brain tissues lead to distorted or biased results during image processing, like tissue segmentation and non-rigid registration. To overcome these issues, proposed a DS-GAN model for inpainting brain MRI images. Initially, the input MRI images are segmented using a Gated shape convolution neural network (GS-CNN). In the first GAN, grayscale pixel intensities and the remaining image edges are utilized to create edge generators or edge reconstruction Generative Adversarial Networks (EGAN), which are capable of creating false edges in areas that are missing. The results of the experimental results demonstrated that the Jaccard Index (JI) was 0.82, while the Dice Index (DI) was 0.86. The proposed DS-GAN in terms of L1 loss, PSNR, SSIM, and MSE obtained was 2.18, 0.972, 32.04, and 26.42. As compared to existing techniques, the proposed DS-GAN model achieves an overall accuracy of 99.18%.

Keywords

brain tumor generative adversarial network MRI images gated shape convolution neural network

1. Introduction

BT segmentation is a hard venture due to their numerous behavior each in phrases of shape and function.¹ MRI is the over different imaging modality for the analysis and preparation of mind tumors due to its non-invasive assets without the publicity to ionizing radiations and superior image contrast in soft tissues.^2,3 Various tissue contrast image types are employed by MRI modalities. These images enable the extraction of structural information that is valuable for the diagnosis and treatment of tumors and their sub-regions.⁴ The function of image inpainting is to provide visually realistic and semantically cohesive material to an image's blank regions.^5,6 The public benefits from image inpainting in many real-world situations, such as when images require to be edited, distracting aspects removed, or facial defects corrected.^7,8 All these techniques need the definition of the inpainting zones. Segmentation-based techniques are greatly influenced by the effectiveness of segmentation.⁹

Inpainting models can suffer from sudden color distortions despite the substantial improvements in image inpainting, especially when there are large missing portions.^10,11 Generative models, such as GANs, utilize probability distributions to produce artificial data.¹² A discriminator that utilizes the produced data as input for generating real data and a generator that generates synthetic data in response to random inputs make up GAN two major fundamental components.¹³ GANs are extensively employed in healthcare technology due to their strong performance and durability.^14,15 In addition, the limited dataset has encouraged the usage of GANs to supply the necessary quantity of images to support training processes, which is crucial to achieving results with increased accuracy.¹⁶ For general computer vision applications, GAN-based data augmentation has shown excellent results. It demonstrates the excellent generalization capabilities of GAN by comparing the generated noise variable distribution to the real one using a sharp value function.¹⁷ GANs may create high-fidelity, high-resolution images in the medical image processing domain without supervision. By produced T1-weighted brain MRI images are equally accurate as real images.¹⁸

GANs can supply more variant instances by using a min-max two-player game to transform the separate distribution of confined training models into a continuous distribution.¹⁹ By combining the genuine and false data generated by GANs, it enhances the performance of the model. However, a significant portion of these works are confined to 2D images.²⁰ Moreover, segmenting brain MR with conditional GAN generators allows training the Cycle GAN model to accurately recognize geometric violations in growing MR and generates more training data for BT segmentation.²¹ The main contribution of the paper is summarized as,

In this research, a novel DS-GAN model has been proposed for inpainting multi-modality brain MRI images.

Initially, the input images are semantic segmentation utilizing a Gated Shape Convolution Neural Network (GS-CNN).

In the first GAN, grayscale pixel intensities and the remaining image edges are utilized to create edge generators, or EGAN, which are capable of creating false edges in areas that are missing.

The second GAN uses the RGAN approach to fill in the gaps by combining edge information from the absent regions with color and texture information from the nearby regions.

The overall evaluation of the proposed DS-GAN model was estimated by some parameters such as specificity, recall, accuracy, precision, JI, and DI, respectively.

The rest of the papers have five sections were separated as follows. Section 2 defines literature reviews. The suggested DS-GAN model is discussed in Section 3. The results and discussion are described in Section 4. The conclusion is described in Section 5.

Research Hypothesis:

The proposed DS-GAN model raises the following research questions:

SQ1: How can brain MRI images with missing or distorted regions be accurately inpainted?

SQ2: How does the proposed DS-GAN model compare to existing GAN-based techniques in terms of image inpainting performance?

SQ3: What improvements does the Gated Shape Convolutional Neural Network (GS-CNN) bring to brain MRI image segmentation?

2. Literature survey

Researchers have recently published a number of strategies and techniques, mostly for the effective inpainting of MRI brain images. This section provides an overview of a few of those semantic segmentation, edge detection, and area identification methods.

In 2020 Armanious et al.²² proposed an ipA-MedGAN for inpainting medical images. A novel structure for the localized distortion inpainting of medical modalities. It makes it possible to inpaint regions of any shape without needing to know where the pixels are located within the regions of interest. By combining two discriminator networks with distinct receptive fields, a pre-trained feature extractor network, and a cascading MultiRes-UNets generator.

In 2024 Zhao et al.²³ designed TBI-GAN, a unique generative adversarial network, to improve brain area segmentation data in TBI MR images. The proposed TBI-GAN is appropriate for augmentation of traumatic brain segmentation since it can inpaint both the MRI image and brain area label maps based on templates of a normal brain. The greatest results in 2D and 3D brain region segmentation in TBI MRI images are improved by the segmentation network.

In 2021 Wang et al.²⁴ suggested a medical image inpainting model based on edge and structural information (ESMII). Moreover, the features are from the image at three scales using multi-scale residual blocks in order to get around the deep network's degradation issue. The network is generating more substantial structural textures because of the multi-level loss. On the datasets for abdominal CT, abdominal MRI, and COVID CT, it performs optimally and numerically.

In 2022 Tang et al.²⁵ proposed 3D Tensor-Wise Brain-Aware Gate network (TW-BAG) for distorted DTI inpainting. The suggested approach has been tailored to the situation by making use of several tensor-wise decoders and a dynamic gate mechanism. The Human Connectome Project (HCP) dataset, which is accessible to the public, uses scalar DTI measurements and mutual image similarity metrics that result from the projected tensors. Fractional anisotropy (FA) errors in the gathered region are decreased by 0.1561 and 0.0087, respectively.

In 2024 Pitchai et al.²⁶ developed a fuzzy K-Means clustering segmentation system for brain tumors using deep learning. First, the wiener filter is used to denoise the MRI images to remove noise. To classify the images as abnormal or normal, an ANN is utilized. Lastly, the tumor region has been segmented using the fuzzy K means technique on the aberrant images. With a maxi accuracy of 94%, the proposed semantic segmentation technique produces better segmentation results.

In 2020 Jia et al.²⁷ suggested a brain tumor semantic segmentation called fully automatic heterogeneous segmentation utilizing support vector machines (FAHS-SVM). Integrates relaxed metrics, morphology, and structural information into the cerebrovenous system's MRI imaging. A high degree of homogeneity between the neighboring brain tissue and anatomy characterizes the segmenting function. The experimental findings demonstrated that the suggested technique could identify diseased and normal tissues in MRI images with 98.51% accuracy.

In 2024 Haque et al.²⁸ designed a deep neural NeuroNet-19 network architecture to detect brain tumors. The iPPM ensures the retrieval of multi-scale feature maps for both local and global image conditions. Four classes of meningioma, BTs glioma, pituitary tumors, and no tumor, are used to train NeuroNet 19. It finds that, at 99.3%, NeuroNet19 obtains the maximum accuracy.

In 2021 Chong et al.²⁹ created a 3D MRI image volume of the human brain with T1-contrast, utilizing a GAN approach trained on 11 × 12 MRI images. The texture network is used to precisely restrict contrast patterns in image sections and to understand the 3D shape modifications in BT. The shape network creates a 3D voxel-wise deformation map that is used to twist brain images from the Montreal Neurological Institute (MNI). The texture network is then used to style axially aligned slices.

In 2021 Abu-Srhan et al.³⁰ developed a paired-unpaired Unsupervised Attention-guided GAN model called uagGAN to convert MRI data to CT images. After initializing on a paired dataset, the uagGAN model undergoes a cascade process to retrain on an unpaired dataset. It generates fine structure images by combining non-adversarial losses with the Wasserstein GAN adversarial loss function. It's used to produce attention masks, which enhance image quality and accuracy.

In 2020 Sun et al.³¹ proposed an MRI segmentation and augmentation based on generative adversarial networks, 3D MRI images without disturbing against the disease. A large quantity of labeled data when using segmentation networks. A set of analyses conducted on the BRATS17 dataset validates the efficiency of MM-GAN in improving and anonymizing data. It improves the tumor core's and the entire tumor's dice scores by 0.16 and 0.17, respectively.

From these literature surveys use overly deep layers, causing essential low-level features to disappear before reaching higher layers, which reduces model efficiency. Compared to other existing models, the proposed method for brain image inpainting offers significant advancements in the field of medical image processing. While existing techniques primarily focus on inpainting general or medical images, often lack specialization for the unique challenges presented by brain images. These methods may not fully address the intricacies of brain structures, leading to inaccurate reconstructions or loss of critical information. In contrast, the proposed brain image inpainting method is specifically designed to handle the complex patterns and textures of brain tissues. It provides a more precise and reliable reconstruction by incorporating specialized algorithms that better preserve anatomical details. This method not only improves the quality of inpainted brain images but also enhances diagnostic accuracy, making it a valuable contribution to medical imaging, particularly in scenarios where image defects or data loss hinder effective diagnosis. Likewise, generators are needed to detect edges and regions. To solve this issue, the DS-GAN model is presented, which inpaints brain MRI images to increase their accuracy.

3. Proposed DS-GAN model

The DS-GAN model for inpainting brain MRI images has been proposed in this research. In the first GAN, grayscale pixel intensities and the remaining image edges are utilized to create EGAN, which is capable of creating false edges in areas that are missing. An overall diagram of the Proposed model is shown in Figure 1.

Figure 1.

The overall workflow of the proposed DS-GAN methodology.

Figure 2.

GS-CNN architecture.

3.1. Dataset description

In the proposed method, a brain MRI image dataset containing 2D images is used. This dataset has a 256 × 256 resolution. It considers a total of 70 images; 45 images are abnormal and they are affected by different kinds of diseases. For every disease, only 15 images are used. The remaining 25 images are normal, and they are not affected by any kind of injury. This method used a split of 65% for training and 35% for the testing process.

3.2. Segmentation

In this Proposed method, the Gated Shape-CNN network for segmentation. This network is made up of two network streams and a fusion module. Two streams comprise the network: a shape stream that analyzes semantic boundaries as shape information, and a normal stream that is a standard segmentation CNN. Through the use of local supervision and our thoughtfully crafted Gated Convolution Layer (GCL), we are able to compel the shape stream to analyze only boundary-connected data. Next, combine boundary features from the semantic region and form stream characteristics from the regular stream to generate a more precise semantic segmentation result, particularly in the area of boundaries. Figure 2 illustrates the GS-CNN, which integrates a shape stream and a regular stream for segmentation. ASPP applies geometric features in the shape stream, while the regular stream focuses on context and spatial features.

There are two types of streams: regular and shaped. An arbitrary backbone architecture can be the normal stream. As shapes are processed, supervision, Gated Convolutional Layers (GCL), and residual blocks are used. A fusion part uses an Atrous Spatial Pyramid Pooling module (ASPP) to merge the data from the two streams at multiple scales. An effective dual-task regularizer is used to guarantee high-quality boundaries on semantic segmentation masks.

3.2.1. Regular steam

This technique, designated as $Q_{θ} (J)$ , with parameters $θ$ , uses an image $J \in N^{3 x H x W}$ with H height and W width as input and generates dense pixel features. Any predictive, fully convolutional network, such as a semantic segmentation network based on SegNet or VGG, can serve as the regular stream. Considering that SegNet represents the pinnacle of semantic segmentation technology, The regular stream's output feature representation is given as $t \in N^{e x \frac{H}{n} x \frac{W}{n}}$ , where n is the regular stream's stride.

3.2.2. Shape stream

The output of this stream, denoted as $R_{\emptyset}$ with parameters $\emptyset$ , is semantic boundaries. Its inputs are the image gradients $\nabla J$ and the output of the regular stream's first convolutional layer. Gated convolution layers (GCLs) cover residual blocks in the network architecture. As elucidated below, the shape stream solely uses GCL to handle boundary-relevant data. The shape stream's output boundary map is indicated $r \in N^{H x W}$ . Utilizing GT segmentation masks to produce Ground-Truth (GT) binary edges, control the form stream by applying binary cross entropy loss on the output boundaries.

3.2.3. Fusion module

This module, designated as $E_{γ}$ and using parameters γ, fuses the edge map v produced by the shape branch with the dense feature design u from the regular branch, preserving multi-scale contextual information in the process. It provides a more sophisticated semantic segmentation output by combining boundary and region characteristics. In formal terms, it produces a categorical distribution e = q (x|u, v) = Eγ (u, v) ∈ R M × H × W, which is the probability that a given set of pixels belongs to each of the M classes, for a segmentation prediction of a M semantic class. Use an atrous spatial pyramid pooling to specifically integrate the border maps u and v. It has been demonstrated that this semantic segmentation network is crucial and permits us to maintain multi-scale contextual information.

3.2.4. Gated convolution layer

The GCL layer, which connects the semantic segmentation and semantic boundaries, makes it easier for data from the normal to shape stream. GCL is a fundamental part of the design that assists in filtering out unnecessary information so that the shape stream processes only pertinent data. The form stream differs from the ordinary stream in that it does not include characteristics from it. Instead, it makes use of GCL to deactivate its activations that aren't considered pertinent by the higher-level data that is part of the regular stream. As a result, the shape stream can process the image at a very large resolution by implementing an efficient, shallow architecture.

GCL is in a few different places between the two streams. Let t ∈ 0, 1, · · ·, n be an index, and let n be the quantity of locations. The average illustrations of the matching shape and regular streams are processed by a GCL are represented by $u_{t}$ and $v_{t}$ Using concatenating $u_{t}$ and $v_{t}$ to create an attention map αt ∈ N H × W is the first step in using GCL. Next, a sigmoid function σ develops a standardized 1 × 1 convolutional layer D 1 × 1.

α_{t} = σ (D_{1 x 1} (u_{t} | | v_{t})),

(1)

where the symbol

| |

stands for feature map concatenation. GCL is used to st as an element-wise product ⊙ with α map of attention, using a kernel to weight channels

w_{t}

, residual connection, and given the attention map

α_{t}

. GCL * at each pixel (j, k) is calculated as,

\begin{aligned} \hat{u_{t}^{j, k}} & = (u_{t} * w_{t})_{(j, k)} \\ = ((u_{t (j, k)} ⊙ α_{t (j, k)}) + u_{t (j, k)})^{S_{w t}} \end{aligned}

(2)

The following layer in the shape stream receives (

\hat{u_{t}}

). α can alternatively be automatically interpreted as an attention map that gives greater weight to regions with significant boundary information. Connect the final, fourth, and third levels of the normal stream to the three GCLs connected in this step. The typical stream feature maps are upsampled using bilinear interpolation.

3.3. Edge-Region reconstruction-based GAN

Figure 3 shows the edge reconstruction and region reconstruction stages of the image inpainting network in our proposed DS-GAN model. Each stage has a generator and a discriminator, and both stages use an adversarial model. Let $G_{1}$ and $D_{1}$ denote the edge reconstruction generator and discriminator, while $G_{2}$ and $G_{2}$ denote the region reconstruction generator and discriminator, respectively.

Figure 3.

Architecture of the proposed DS-GAN model.

3.3.1. Edge reconstruction module

A canny edge detector is used to compute the edge maps. It is a basic method for creating edge maps. Their grayscale and edge map will be represented as $E_{g t}$ and $J_{g r a y}$ , $J_{g t}$ input images.

Input the masked grayscale image ${\tilde{J}}_{g r a y} = J_{g r a y} ⊙ (1 - M)$ , its edge map ${\tilde{E}}_{g t} = E_{g t} ⊙ (1 - M)$ , and image mask M as required into the edge generator.

\begin{aligned} E_{p r e d} = F_{1} ({\tilde{J}}_{g r a y}, {\tilde{E}}_{g t}, M) \end{aligned}

(3)

The discriminator that ascertains the validity of an edge map receives two inputs:

E_{g t}

and

E_{p r e d}

conditioned on

J_{g r a y}

. The objective of the network's training involves the building of a feature-matching loss and an adversarial loss.

mi n_{G 1} ma x_{D 1} L_{G 1} = mi n_{G 1} (λ_{a d, 1} ma x_{D 1} (L_{a d, 1}) + λ_{P M} L_{P M})

(4)

Where

λ_{a d, 1}

, and

λ_{P M}

are regulating variables. The definition of the adversarial loss is,

L_{a d, 1} = D_{(E_{g t}, J_{g r a y)}} [l o g D_{1} (E_{g t}, J_{g r a y})] + D_{(J_{g r a y)}} log [1 - E_{1} (E_{p r e d}, J_{g r a y})]

(5)

The feature-matching loss

L_{P M}

is used to compare the activation maps of the intermediate layers of the discriminator. As a consequence, the training process is stabilized since the generator is compelled to provide outcomes that have representations that match actual pictures. Comparing the activation maps to those from the dilated CNN network reveals that it is comparable to perceptual loss. Nonetheless, the Dilated CNN network first misses the outcome as it is inadequate to provide cutting-edge information. The following is the definition of the feature matching loss

L_{P M}

L_{P M} = D [\sum_{m = 1}^{n} \frac{1}{N_{m}} | | E_{1}^{(m)} (L_{g t}) - E_{1}^{(m)} (E_{p r e d}) | |]

(6)

Where

E_{1}^{(j)}

is the activation in the discriminator's j-th layer, n is the final discriminator's convolution layer, and

N_{m}

is the quantity of components in the m-th initiation layer.

3.3.2. Region reconstruction module

Shuffled Grouped Convolution is the term for the method of using group convolution to obtain information. In convolutional neural networks, shuffled channel shuffle is an operation that aids in combinatorically determining the information flow between feature channels. Different learning segments result from the application of grouped convolution. These distinct segments each include unique learned kernel-based information that can be merged in different ways with higher layers. There are two convolutions in the stacked group. The term “group convolution” is GConv. To start, two convolution layers are stacked and have an equal number of groups. After that, the only input channels that each output channel is related to are those in the group. Second, once GConv1 has finished processing data from various groups, the input and output channels are fully related. And lastly, a channel-shuffle-based equivalent of GConv2.

4. Results and discussion

A Matlab-2019b result analysis is used to gauge the effectiveness of the proposed DS-GAN model. A further processing step involves dividing MRI images from publicly available datasets into edges. Based on the test sample analysis, precision (PR), recall (RE), accuracy (AC), Specificity (SP) DI, and JI are evaluated. Additionally, the DS-GAN model's efficiency is discussed and examined, along with the overall accuracy rate.

Figure 4 illustrates the experimental results from the proposed DS-GAN model using MRI images. Column (Clm) 1 shows the input images, and Clm 2 shows the output results. The segmented mask images are presented in Clm 3. The edge detection images are shown in Clm 4. Clm 5 shows the brain MRI images painted using the DS-GAN discovered images.

Figure 4.

The experimental results of the proposed DS-GAN model.

4.1. Performance analysis

Recall, precision, and accuracy were the particular measuring methods utilized to assess the overall performance of the proposed DS-GAN model.

\begin{matrix} A c c u r a c y = \frac{T P + T N}{T P + T N + F P + F N} \end{matrix}

(7)

\begin{matrix} P r e c i s i o n = \frac{T P}{T P + F P} \end{matrix}

(8)

\begin{matrix} R e c a l l = \frac{T P}{T P + T N} \end{matrix}

(9)

Whereas TP and FP stand for true negatives and false positives on the MRI scans, TN and FN stand for false positives and true negatives, respectively.

The dice index (DI) involves the use of an index of the three-dimensional edge of the image occlusions in addition to a reproducibility validation parameter. To find the precise ratio of the real region to pixels, DI is computed. Equation (10) computes the anticipated pixels and background pixels. By separating connection sizes by union sizes, the JI calculates how similar two finite samples are to one another. The JI, or similarity index, between anticipated and actual pixels is calculated in equation (11).

\begin{matrix} D I = \frac{2 T_{p}}{F_{p} + 2 T_{p} + F_{n}} \end{matrix}

(10)

\begin{matrix} J I = \frac{T_{p}}{T_{p} + F_{n} + F_{p}} \end{matrix}

(11)

where

T_{p} and T_{n}

assets TP and TN of the sample images,

F_{p}

and

F_{n}

requires FP and FN of the model images. The overall Performance analysis of the proposed DS-GAN model is in Table 1.

Table 1.

The overall Performance analysis of the proposed DS-GAN model.

Images	Accuracy	Precision	Recall	DI	JI
Image 1	93.26	92.42	91.08	90.29	89.41
Image 2	95.19	94.58	92.62	91.83	90.56
Image 3	96.55	96.07	95.39	94.74	92.38
Image 4	94.61	93.91	92.54	91.21	90.39
Image 5	97.38	96.27	95.07	93.78	91.02
Image 6	95.66	94.65	93.29	91.82	90.55
Image 7	98.05	96.36	98.65	96.05	99.18
Image 8	96.69	94.24	98.83	97.34	98.03
Image 9	98.36	96.42	95.26	99.03	97.63
Image 10	99.18	98.48	97.32	98.45	97.08

The PSNR and MSE were also used to evaluate capability. One often-used statistic to assess the output, ground truth, and image quality of a proposed network is the PSNR. An extensive comparison is made between the modified and reference photos. The original and inked image error ratio is measured by the mean squared error (MSE). Models with lower error rates are thought to be more effective when their MSE values are used to measure their efficacy. Table 2 shows the performance as determined by those requirements.

Table 2.

The analysis of the DS-GAN model based on PSNR, and MSE.

Images	PSNR	MSE
Image 1	24.48	2.2
Image 2	22.12	3.0
Image 3	28.36	2.8
Image 4	20.29	2.6
Image 5	20.43	3.8
Image 6	19.06	3.2
Image 7	22.34	2.7
Image 8	24.38	3.2
Image 9	28.08	2.8
Image 10	30.26	1.6

The efficiency of the proposed DS-GAN model inpaints MRI images. The segmentation and inpainting performance are estimated in terms of PR, RE, AC, JI, and DI. The overall accuracy for the publicly accessible dataset achieved by the proposed DS-GAN model is 99.18%. Additionally, the Proposed DS-GAN model attains a total precision of 98.48% and a recall of 97.32%. Based on Table 2, the DS-GAN model is the best performing, indicating accuracy at a high level. The training and testing AC and loss curves are shown in Figures 5 and 6.

Figure 5.

The AC curve of the DS-GAN model.

Figure 6.

The loss curve of the DS-GAN model.

In Figure 5, the accuracy graphs of the transfer learning models are estimated with the number of epochs and accuracy range. The AC of the pre-trained network advances with the no. of epochs. Figure 6 displays the epochs and loss graphs range, demonstrating the loss of the trained network reduces when the epochs are increased. DS-GAN model is proposed that accurately inpaints brain MRI images. The proposed DS-GAN model has an accuracy of 99.18% based on its performance throughout the training phase.

4.2. Comparative analysis

In this section, the DS-GAN model was related to different GAN structures based on various inpainting performance parameters. The capability of existing GAN contexts was analyzed to demonstrate that the outcomes of the proposed DS-GAN are more effective in the process of brain MRI image inpainting. Initially, the proportional estimation was accomplished between previous segmentation networks as demonstrated in Table 3.

Table 3.
A comparative analysis of semantic segmentation networks using JI and DI.

Networks JI DI

Attention U-net 0.48 0.64

Mask RCNN 0.74 0.82

SegNet 0.64 0.56

Attention V-net 0.72 0.78

Gated Shape CNN 0.82 0.86

Networks	JI	DI
Attention U-net	0.48	0.64
Mask RCNN	0.74	0.82
SegNet	0.64	0.56
Attention V-net	0.72	0.78
Gated Shape CNN	0.82	0.86

Using Table 3, the proper proportions of the dice and Jaccard coefficients were obtained, and then several segmentation networks were compared in terms of performance measures. In terms of comparison, the Gated Shape CNN outperformed the segmentation networks. The Gated Shape CNN increases the overall Jaccard index by 0.34, 0.08, 0.18, and 0.1 better than Attention U-net, Mask RCNN, SegNet, and Attention V-net respectively. The Gated Shape CNN increases the overall Dice index by 0.22, 0.04, 0.3, and 0.08 is better than Attention U-net, Mask RCNN, SegNet, and Attention V-net respectively. The proposed DS-GAN model is calculated and built on the segmentation networks in the Jaccard and dice coefficients, as illustrated in Figure 7.

Figure 7.

Comparison of existing and proposed model.

This Gated Shape CNN achieves defined results with an accuracy of 98.46%, outperforming other models. The outcome of the inpainting reveals that the incorporation of the GAN allows the suggested DS-GAN model to operate more quickly and achieve optimal outcomes. According to the above comparison, the proposed DS-GAN performs better in terms of accuracy conditions than the most advanced methods. Table 4 presents a comparison analysis of the proposed DS-GAN and each advanced approach based on loss, similarity index, and noise ratio.

Table 4.

The comparison of the existing GAN model with the proposed DS-GAN model.

Methods	L1 loss	SSIM	PSNR	MSE
ipA-MedGAN²²	0.24	0.961	30.62	24.90
TBI-GAN²³	2.04	0.875	26.42	18.05
TW-BAG²⁵	0.01	0.082	18.42	11.34
uagGAN³²	0.17	0.595	27.59	22.58
Ours	2.18	0.972	32.04	26.42

The comparison of different GAN designs according to the particular performance indicators for image inpainting is shown in Table 4. The L1 loss is the only loss function present in these GAN architectures.³³ The L1 loss, frequently referred to as the perceptual loss, is used to quantify differences between the actual and the predicted. The suggested DS-GAN outperforms the other GAN architectures in terms of L1 loss, SSIM, PSNR, and MSE. The proposed approach outperforms the ipA-MedGAN,²² TBI-GAN,²³ TW-BAG,²⁵ and uagGAN³² in terms of PSNR value while achieving a low degree of loss.

From Table 5, the comparison of several GAN architectures based on their accuracy in painting the images. The Proposed DS-GAN model achieves total accuracy of 0.09%, 9.22%, 3.87%, and 2.17% better than ES-GAN,³² Patch-GAN,³⁴ CDA-GAN,³⁵ and RTGAN³⁶ respectively. It is obvious from Table 5 that our innovative network outperforms the current methods. As a result, the suggested DS-GAN model's calculated fallouts may be used to accurately inpaint brain MRI images.

Table 5.

Comparison of the existing and the proposed models.

Authors	Methods	AC
Hamghalam³²	ES-GAN	90.03%
Zhu³⁴	Patch-GAN	97%
Xu³⁵	CDA-GAN	95.34%
Sille³⁶	RTGAN	97.02%
Proposed	DS-GAN	99.18%

5. Conclusion

This paper presented a novel DS-GAN model for inpainting brain MRI images. In the first GAN, grayscale pixel intensities and the remaining image edges are utilized to create edge generators, or EGAN, which are capable of creating false edges in areas that are missing. A second GAN is used to fill in the missing regions using edge information from the missing regions combined with color and texture information from surrounding regions. Experiment analysis showed JI and DI to be 0.82 and 0.86, respectively. The proposed DS-GAN in terms of L1 loss, PSNR, SSIM, and MSE obtained was 2.18, 0.972, 32.04, and 26.42. The proposed DS-GAN model has a total accuracy of 99.18%, which is relatively better than the existing methods. In the future, utilizing 3D brain MRI datasets will significantly enhance the accuracy of segmentation results. The 3D data allows for a more comprehensive analysis of brain structures, capturing finer details and spatial relationships that are often missed in 2D images. This improvement will lead to more precise and reliable outcomes, particularly in clinical applications such as diagnosis and treatment planning.

Footnotes

Acknowledgements

The author would like to express his heartfelt gratitude to the supervisor for his guidance and unwavering support during this research for his guidance and support.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Vinisha

Boda

. A novel framework for brain tumor segmentation using neuro trypetidae fruit fly-based UNet. Int J Intell Syst Appl Eng 2024; 12: 783–796.

Ramaiah

Balasubramanian

Appathurai

, et al. Narayan aperumal muthukumaran detection of Parkinson’s disease via clifford gradient-based recurrent neural network using multi-dimensional data. Revue Roumaine des Sciences Techniques — Série Électrotechnique Et Énergétique 2024; 69: 103–108.

Siddique

Chow

. Application of nanomaterials in biomedical imaging and cancer therapy. Nanomater 2020; 10: 1700.

Srinivas

Sasibhushana Rao

. Segmentation of multi-modal mri brain tumor sub-regions using deep learning. J Electr Eng Technol 2020; 15: 1899–1909.

Jegatheesh

Kopperundevi

Anlin Sahaya Infant Tinu

. Brain aneurysm detection via firefly optimized spiking neural network. Int J Curr Bio-Med Eng 2023; 01: 23–29.

Prakash Perumal

Gnanaraj

Satish Pandian

, et al. Revolutionizing E-Cycle charging with a novel regenerative battery system. 2023 International Conference on Sustainable Communication Networks and Application (ICSCNA), Theni, India, 2023, pp. 762–767.

Zhao

Chen

Huang

. Deep learning-based forgery attack on document images. IEEE Trans Image Process 2021; 30: 7964–7979.

Bourouis

Alroobaea

Alharbi

, et al. Recent advances in digital multimedia tampering detection for forensics analysis. Symmetry 2020; 12: 1811.

Clèrigues

Valverde

Salvi

, et al. Minimizing the effect of white matter lesions on deep learning-based tissue segmentation for brain volumetry. Comput Med Imaging Graph 2023; 103: 102157.

10.

Gnana Sophia

Thanammal

Sujatha

. Secure storage of lung brain multi-modal medical images using DNA homomorphic encryption. Int J Curr Bio-Med Eng 2023; 01: 16–22.

11.

Junayed

Islam

. Consistent video inpainting using axial attention-based style transformer. IEEE Trans Multimedia 2022; 25: 7494–7504.

12.

Saxena

Cao

. Generative adversarial networks (GANs) challenges, solutions, and future directions. ACM Comput Surv 2022; 54: 1–42.

13.

Jain

Seth

Paruthi

, et al. Synthetic data augmentation for surface defect detection and classification using deep learning. J Intell Manuf 2022; 33: 1–14.

14.

Waheed

Goyal

Gupta

, et al. Covidgan: data augmentation using auxiliary classifier gan for improved COVID-19 detection. Ieee Access 2020; 8: 91916–91923.

15.

Liu

Zhao

Wang

, et al. LSTM-GAN-AE: a promising approach for fault diagnosis in machine health monitoring. IEEE Trans Instrum Meas 2021; 71: 1–13.

16.

Zhong

Liu

, et al. ML-CGAN: conditional generative adversarial network with a meta-learner structure for high-quality image generation with few training data. Cognit Comput 2021; 13: 418–430.

17.

Rajkumar

Hariharan

Dhanakoti

, et al. A secure framework for managing data in cloud storage using rapid asymmetric maximum based dynamic size chunking and fuzzy logic for deduplication. Wireless Netw 2024; 30: 321–334.

18.

Chai

Zhang

, et al. MRI Restoration using edge-guided adversarial learning. IEEE Access 2020; 8: 83858–83870.

19.

Mohebbi Moghaddam

Boroomand

Jalali

, et al. Games of GANs: game-theoretical models for generative adversarial networks. Artif Intell Rev 2023; 56(9): 1–37.

20.

Liu

Huang

, et al. Generative adversarial networks for image and video synthesis: algorithms and applications. Proc IEEE 2021; 109: 839–862.

21.

Fernando

KRM

Tsokos

. Deep and statistical learning in biomedical imaging: state of the art in 3D MRI brain tumor segmentation. Inf Fusion 2023; 92: 450–465.

22.

Armanious

Kumar

Abdulatif

, et al. ipA-MedGAN: Inpainting of arbitrary regions in medical imaging. In 2020 IEEE international conference on image processing (ICIP), 2020, pp. 3005–3009. IEEE.

23.

Zhao

Zang

Wang

, et al. sTBI-GAN: an adversarial learning approach for data synthesis on traumatic brain segmentation. Comput Med Imaging Graphics 2024; 112: 102325.

24.

Wang

Chen

Zhang

, et al. Medical image inpainting with edge and structure priors. Meas 2021; 185: 110027.

25.

Tang

Wang

Zhu

, et al. TW-BAG: tensor-wise brain-aware gate network for inpainting disrupted diffusion tensor imaging. In 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA) 2022, pp. 1–8. IEEE.

26.

Pitchai

Supraja

Victoria

, et al. Brain tumor segmentation using deep learning and fuzzy K-means clustering for magnetic resonance images. Neural Process Lett 2021; 53: 2519–2532.

27.

Jia

Chen

. Brain Tumor Identification and Classification of MRI images using deep learning techniques. IEEE Access 2020: 1–1.

28.

Haque

Hassan

Bairagi

, et al. Neuronet19: an explainable deep neural network model for the classification of brain tumors using magnetic resonance imaging data. Sci Rep 2024; 14: 1524.

29.

Chong

ETW

. Synthesis of 3D MRI brain images with shape and texture generative adversarial deep neural networks. IEEE Access 2021; 9: 64747–64760.

30.

Abu-Srhan

Almallahi

Abushariah

, et al. Paired-unpaired Unsupervised Attention Guided GAN with transfer learning for bidirectional brain MR-CT synthesis. Comput Biol Med 2021; 136: 104763.

31.

Sun

Yuan

Sun

. MM-GAN: 3D MRI data augmentation for medical image segmentation via generative adversarial networks. In 2020 IEEE International conference on knowledge graph (ICKG) 2020, pp. 227–234. IEEE.

32.

Hamghalam

Simpson

. Medical image synthesis via conditional GANs: application to segmenting brain tumours. Comput Biol Med 2024; 170: 107982.

33.

Tang

Wang

Zhu

34.

Zhu

Zhang

Pang

, et al. Advancing brain tumor inpainting with generative models. arXiv preprint arXiv:2402.01509. 2024.

35.

Tang

, et al. Cross-domain attention-guided generative data augmentation for medical image analysis with limited data. Comput Biol Med 2024; 168: 107744.

36.

Sille

Choudhury

Sharma

, et al. A novel generative adversarial network-based approach for automated brain tumour segmentation. Medicina 2023; 59: 119.

Brain multi modality image inpainting via deep learning based edge region generative adversarial network

Abstract

Keywords

1. Introduction

2. Literature survey

3. Proposed DS-GAN model

3.2. Segmentation

3.2.1. Regular steam

3.2.2. Shape stream

3.2.3. Fusion module

3.2.4. Gated convolution layer

4. Results and discussion

Table 3. A comparative analysis of semantic segmentation networks using JI and DI. Networks JI DI Attention U-net 0.48 0.64 Mask RCNN 0.74 0.82 SegNet 0.64 0.56 Attention V-net 0.72 0.78 Gated Shape CNN 0.82 0.86

Footnotes

Acknowledgements

Funding

Declaration of conflicting interests

References

Table 3.
A comparative analysis of semantic segmentation networks using JI and DI.

Networks JI DI

Attention U-net 0.48 0.64

Mask RCNN 0.74 0.82

SegNet 0.64 0.56

Attention V-net 0.72 0.78

Gated Shape CNN 0.82 0.86