Deep convolutional neural network based hyperspectral brain tissue classification

Abstract

BACKGROUND:

Hyperspectral brain tissue imaging has been recently utilized in medical research aiming to study brain science and obtain various biological phenomena of the different tissue types. However, processing high-dimensional data of hyperspectral images (HSI) is challenging due to the minimum availability of training samples.

OBJECTIVE:

To overcome this challenge, this study proposes applying a 3D-CNN (convolution neural network) model to process spatial and temporal features and thus improve performance of tumor image classification.

METHODS:

A 3D-CNN model is implemented as a testing method for dealing with high-dimensional problems. The HSI pre-processing is accomplished using distinct approaches such as hyperspectral cube creation, calibration, spectral correction, and normalization. Both spectral and spatial features are extracted from HSI. The Benchmark Vivo human brain HSI dataset is used to validate the performance of the proposed classification model.

RESULTS:

The proposed 3D-CNN model achieves a higher accuracy of 97% for brain tissue classification, whereas the existing linear conventional support vector machine (SVM) and 2D-CNN model yield 95% and 96% classification accuracy, respectively. Moreover, the maximum F1-score obtained by the proposed 3D-CNN model is 97.3%, which is 2.5% and 11.0% higher than the F1-scores obtained by 2D-CNN model and SVM model, respectively.

CONCLUSION:

A 3D-CNN model is developed for brain tissue classification by using HIS dataset. The study results demonstrate the advantages of using the new 3D-CNN model, which can achieve higher brain tissue classification accuracy than conventional 2D-CNN model and SVM model.

Keywords

Hyperspectral image classification high dimensional features brain tumor detection machine learning convolutional neural network

1 Introduction

The influence of image processing applications in the medical domain, specifically for human body analysis, reduces the exertions of physicians. Diagnostic procedures, surgical advice, and studies of the human body for hindering diseases are effectively performed based on medical images. Improved examination accuracy has been attained by analyzing the medical images. The basic imaging technique of X-ray provides a cross-section view of the body or specific parts so that the specific portion can be ascertained [1, 2]. However, the major objective of any imaging technique is to provide better results without any side effects and to be prominently more cost-efficient. This redirects the research towards low-cost, non-invasive medical image analysis, which is termed as hyperspectral imaging [13]. CNN database learning as well as neighboring network limitation approach are established with multilayer-based metadata learning and merged with CNN layer to deliver reliable information for brain tumor image classification [3].

The recent development of hyperspectral imaging in cerebral diagnosis is reviewed. HSI could indeed obtain two-dimensional spatial information as well as one spectral information of biological samples concurrently, completely covering the ultraviolet, visible, and infrared spectral ranges with high spectral resolution. Furthermore, the constraints of the application of hyperspectral imaging in the domain of diagnosing cerebral diseases were examined [4]. Recent developments in HSI systems had revealed its ability for medicinal uses, particularly in the detection of diseases and image-guided surgery. In order to provide insight on the rapidly expanding domain of research, this review presents an overview of the fundamentals of HSI including optical systems, DL based image processing, and therapeutic applications of HSI [5, 6]. To highlight the significance of HSI in the medical domain, the fundamental concepts, imaging techniques, comparisons, and recent developments in HSI-related medical applications are explored [7]

All these hyperspectral imaging-based medical applications utilize various algorithms to provide more efficient and accurate results in the image classification process. Artificial intelligence, specifically ML algorithms, is quite popular in hyperspectral image analysis. ML algorithms are employed to extract the essential features from the image data and classify the patterns to provide final decisions about the tissue status. Various supervised and unsupervised ML algorithms like k-NN (k-nearest neighbor), SVM, LDA (linear discriminant analysis), etc. are used for hyperspectral image classification applications [8].

The supervised techniques identify the optimal parameters and predict the results by minimizing the cost function. Unsupervised techniques, on the other hand, train the application without a specific data structure and find the essential patterns from clusters. However, to enhance the prediction accuracy and reduce the computation cost of ML algorithms, DL techniques have been introduced in various image processing applications [9].

DL techniques provide better performance results than ML techniques in numerous medical image applications [10]. Specifically, convolutional neural networks are one of the familiar DL models which provide better classification results than conventional deep neural networks and ML algorithms in visual image processing applications. However, the features of CNN are limitedly utilized in hyperspectral image classifications. Considering this as research motivation, a convolutional neural network-based hyperspectral image classification model for brain tissue classification is presented in this research work. The novelty of the research work is present in its CNN architecture. Instead of the conventional CNN architecture, which includes four 3-D convolutional layers which are deployed before the flatten layer to make sure the model can distinguish between spatial information in various spectral bands without suffering any loss. Considering the HSI classification issue, it is necessary to be able to detect spectral information in addition to spatial information. The presented 3D-CNN is a supervised classification model which outperforms conventional algorithms. The summarized research’s contribution is demonstrated in the prescribed sequence:

In this research work, a 3D convolutional neural network is presented for brain tissue classification in HSI. The Benchmark Vivo human brain HSI dataset is used to validate the performance of the proposed classification model.

The proposed research, utilizes the automated feature learning properties of both 2-D as well as 3-D CNNs.

The preprocessing step includes hyperspectral cube creation, calibration, spectral correction as well as normalization.

The preprocessed images are fed into 3D convolutional neural network in which the features are extracted as different patterns in spectral and spatial dimension. In order to utilize the spectral information effectively, instead of the conventional CNN model, a three-dimensional CNN model is used in the proposed work.

The overall accuracy of 97% is observed through experimental analysis and compared with conventional CNN approaches.

The research work is further arranged in the following order. A detailed literature analysis related to brain tissue classification is presented in section 2. The proposed 3D-CNN model architecture and mathematical model are presented in section 3. Experimental outcomes is presented in section 4. The features of the proposed model are presented as a conclusion in section 5.

2 Related works

A detailed literature analysis of existing brain tissue classification algorithms is considered for analysis and the methodology, feature merits, demerits are discussed in this section. The online classification of information received during HSI endoscopy is proposed to be facilitated by a CNN. With the help of training, a five-layered CNN, it was possible for it to discriminate between the 18 colors, it contains an average accuracy of 94.3% [11].

A cutting-edge multi-stream deep CNN [12] architecture for glioma tumor grading/subcategory grading that separates and merges the information from many sensors. One of the major limitation is the fact that CNNs are difficult to optimize in 3D volumetric classification. The classification of 3D brain tumor MR images into high grade (HG) and low grade (LG) gliomas was therefore accomplished by the introduction of a cascade of CNN with Long Short Term Memory (LSTM) Network [13]. Multiclass energy minimized brain tissue classification from magnetic resonance images is presented in [14] utilizes an iterative linearly constrained minimum variance value for the classification process. The presented hyperspectral image classification model differentiates the normal and white matter with minimum computation time even if the input images are noisy. The optimization model for MRI image processing presented in [15] includes bat optimization with fuzzy logic and a clustering algorithm. The K-means clustering is combined with fuzzy logic to segment the tumor features, and the bat optimization algorithm is used to obtain the pixel minimum distance and initial centroids.

The ML -based MRI image processing application presented in [16] extracts the features from stereo electroencephalogram signals to obtain better decision support in the diagnosis process. Time and frequency domain features are considered for the classification process, which is obtained using the epileptic detection process. The extracted features are classified using different ML algorithms and the performances are measured in terms of accuracy and f1-score. A densely connected neural network model for brain MRI image processing is presented in [17] differentiates the healthy and unhealthy tissues from the images. The incomplete clinical scores and baseline MRI data are used in the image processing application, which extracts multiscale patches to obtain the structural information. The extracted features are processed through the network model along with the weighted loss function to obtain better results comparable to ground truth scores.

Using a hybrid deep auto-encoder [18] and a segmentation method based on Bayesian fuzzy clustering, classify brain tumors. After segmenting the image, robust features like information-theoretic measures, scattering transform (ST), and wavelet packet Tsallis entropy (WPTE) methodologies were indeed utilised for feature extraction. Finally, a softmax regression methodology is employed to classify the tumor segment for the classification of brain tumors. The pre-processing stage is initially conducted out using the non-local mean filter for denoising applications. Utilizing a specialized hyperspectral acquisition equipment with the ability to capture data in the Visual Near Infra-Red (VNIR) ranges from 400 to 1000 nm, the HSI database [19] of in-vivo human brain tissues were obtained. When two images of the identical event were taken immediately after one another, repetition was evaluated.

A three-dimensional CNN (3D-PulCNN) relying on convolution combination units (CCUs) has been utilized to classify pulmonary cancer in micro HIS including both spatial as well as spectral information [20]. When contrasted to VGGNet, the above framework employs only two fully connected layers, which minimizes the number of network specifications and complexity. This [21] makes use of axial slices from MRI data to identify the type of brain tumor. The three most frequently diagnosed brain tumors—glioma, meningioma, and pituitary tumors—were represented in the dataset utilized for the research. A 2D CNN was created for classification purposes.

The fuzzy min-max neural network methodology employed by the [22] helps classify normal from pathological tissues and accomplishes an accuracy rate of 95%, which is superior than that of prior work. Grayscale images and sharpening were performed in the preprocessing of MRI images using marker-based watershed classification [23], and the image was segmented utilizing thresholding and the marker-based watershed algorithm. CNN was then used to categorize the images. Finally, the position and dimensions of the tumor were established.

Reference [24] demonstrates how well brain MRI statistical proprieties can be used to distinguish between a normal class and an abnormal class (brain tumor tissues). This method is quicker since it takes less time to diagnose problems and makes sound decisions. It also aids in the selection of the best model by programmers in order to create the optimum performance situation for a better graphical user interface. The method was used in [25] to create classification models for brain cancers using a dataset of MRI images. The models were created using the DL algorithms VGG16 as well as ResNet.

The use of DL in biomedical image processing is expanding fast, and the Segnet architecture [26] is being used to automatically separate hippocampal atrophy in the brain before classifying the data with an accuracy of 97%. The internal features of the augmented image are extracted by the suggested CNN Deep Net [27] classifier, which also maintains spatial invariance and inheritance while classifying the features into normal and malignant tumor. Additionally, the CNN Deep net categorizes the discovered tumor image as either LG or HG malignant dependent on its feature properties.

The Gradient Boosting ML classification method, which includes preprocessing, feature extraction, and classification stages, is proposed by this [28] as an automated method for the detection of meningioma tumors. In this study, the test brain MRI image is used to derive the intensity features, the Gray Level Run Length Matrix features, and the Grey Level Co-occurrence Matrix (GLCM) features. With the help of the GBML classification technique, these resulting feature sets are categorized. In a categorized abnormal brain imaging, morphological functions are applied to segment the tumor region.

The [29] suggests a faster region-based CNN for segmentation problems and a CNN for classification problems that require fewer computations but have a greater level of accuracy. The Hybrid Kernel based Fuzzy C-Means clustering - CN technique is a proposed automatic tumor classification method by [30]. Using the Hybrid KFC technique, the tumour region of the MRI brain image is segmented. Following the segmentation step, discrete cosine transform (DCT) approaches were applied to the segmented image to enhance feature subsets using statistical features and super pixels based SIFT. The CNN classifier received the best feature values as input, and it was able to categorize the many forms of brain tumors.

Thus, from the literature analysis, it can be observed that the few research works utilized ML-based models to extract and classify the medical image features however the performance is not satisfactory for similar signatures or features. DL algorithms perform better than conventional approaches.

In 2D CNNs, convolutions are typically only applied to spatial dimensions that include all of the feature maps from the previous layer. However, in order to solve the HSI classification difficulty, it is necessary to be able to detect both spectral and spatial information. The 2D CNN is unable to handle the spectral data. In contrast, the 3D kernel may acquire both spectral and spatial features simultaneously while also increasing processing complexity. This work presents a 3D deep CNN to exploit the automated feature learning properties of both 2-D and 3-D CNNs.

3 Proposed work

The proposed DL-based hyperspectral brain tissue classification employs a 3D CNN to classify different types of tissues. The mathematical model for the proposed classification process is presented in this section. The overall process of the proposed hyperspectral image classification is presented in Fig. 1.

Fig. 1

Overall process of the proposed classification model.

The process starts from preprocessing the hyperspectral image. The preprocessing step includes hyperspectral cube creation, calibration, spectral correction as well as normalization. In this, cube creation is the process of converting raw images into hyperspectral cubes. In the calibration step, the reproducibility of the data is maintained so that images obtained at different light conditions are balanced to acquire better results. In the spectral correction, the high sensitivity spectrums are corrected using a correction matrix and finally, in the data normalization process, the image pixels are balanced using the standardization procedure. The pre-processed image is fed into a 3D CNN to classify the HSI.

The major reason to adopt a 3D CNN for the proposed hyperspectral image classification purpose is to produce efficient results better than conventional ML and DL techniques. Since conventional ML and CNN models lag in classification performance due to similar spectral signatures in HSI.

The mathematical model for the preprocessing steps is presented as follows. In the hyperspectral cube creation, hyperspectral cubes are created by converting the raw images. Hyperspectral cubes are created from the raw images captured with the snapshot camera. The resolution of the active region of the hyperspectral raw image is 2045×1085 pixels, and the active region is made up of 25 wavelengths in repeating mosaic blocks (5×5). This results in a spectral resolution of 25 bands and a spatial resolution of 419×217 pixels for the hyperspectral cube.

Since the raw images comprised different wavelengths and in which the active area is selected to obtain a hyperspectral block. Different band spectral resolutions are utilized to define the special resolution of the hyperspectral image in the initial preprocessing step. In the calibration step, the images acquired with different light conditions are balanced. The reference images are categorized into black and white images. In which the white references define the maximum reflection of incident light and black reference defines the minimum reflection of incident light.

The calibrated images are obtained based on the black and white reference which are expressed as follows. $J_{c} = \frac{J - D}{W - D}$ (1) where the obtained hyperspectral cube is defined as $J$ , black references is represented as $D$ and white references are represented as $W$ . The next step after calibration is spectral correction, in which the secondary harmonics present in the hyperspectral image is balanced using a correction matrix. The spectral correction matrix includes the spectral correction coefficients for the virtual bands and the process is mathematically expressed as, $J_{S c} = J_{c} \times M_{S C}$ (2) where, the calibrated hyperspectral cube is represented as $J_{c}$ and the correction matrix is represented as $M_{S C}$ . The final process in the pre-processing is normalization in which the different pixels values are normalized to preserve the spectral signature shapes. The normalization process includes the root mean square values of spectral bands as normalizing coefficients as follows. $C [r, c] = \sqrt{\frac{\sum_{b = 1}^{B} {(J_{S c} [r, c, b])}^{2}}{B}}$ (3) $J_{N} [r, c, b] = \frac{J_{S c} [r, c, b]}{C [r, c]}$ (4) where, the terms 𝓇, 𝒸, 𝒷 represents the rows, columns and band dimensions of spectral corrected cube $J_{S c}$ . The preprocessed images are fed into 3D convolutional neural network in which the features are extracted as different patterns in spectral and spatial dimension. In order to utilize the spectral information effectively, instead of the conventional CNN model, a three-dimensional CNN model is used in the proposed work. The series of fully connected layers maps the functions and separates the data into different classes.

The dataset includes more than 3 lakh samples of spectral signatures which are labeled based on the angle mapper algorithm. Four different categories of data are included such as Tumor Tissue (TT), Normal Tissue (NT), Background Elements (BE), and Blood Vessels (BV). Due to insufficient image circumstances to make the first diagnosis of GBM tumour investigated and labelling. The GBM tumours were added in to the data base, but no more tumour sampled were included. The samples were taken into account six HSI images (P008-01 to P020-01), it is examined as shown in Fig. 2(a). The NT, TT, BV, and BE samples were depicted on the ground truth maps by the colors green, red, blue, and black, correspondingly.

Fig. 2

(a) Proposed synthetic RGB image of six data sets and (b) 3D-CNN model for hyperspectral image classification.

The white pixels symbolize the pixels that have no class assigned, making it impossible to evaluate them quantitatively.

3.1 Proposed deep 3D CNN

Any model would find it extremely difficult to take on this problem. The actual HSI data over spectral bands are initially subjected to PCA in order to remove spectral redundancy. The number of spectral bands is decreased using PCA while maintaining the spatial dimensions.

To use image classification, the hypercube is divided into 3-D-patches which overlap each other and are small and their truth labels are decided on the label of the pixel at the center. As a result, from the preprocessed hypercube X, neighboring 3D patches $P \in R S \times S \times B$ were generated. Their centers are situated at the spatial point (α, β), with the width designated by M, overall height being N, as well as the number of spectral bands by B. The n is the overall number of 3D patches obtained from X, determined by (M - S + 1) × (N - S + 1). It covers the S S window or spatial breadth and all spectral bands, i.e., B.

As a result, $P_{α, β}$ is used to indicate the position of the 3-D patch (α, β), which spans the B spectral bands of X and has a width that ranges from $α - \frac{S - 1}{2}$ to $α + \frac{S - 1}{2}$ and a height that ranges from $β - \frac{S - 1}{2}$ to $β + \frac{S - 1}{2}$ .

Convolution is accomplished by adding the dot product between the kernel and the input data. The kernel is moved across the input data’s whole spatial dimension to cover it. The model’s activation function obtains the convolved information in attempt to create nonlinearity. Convoluting a 3D kernel over 3-dimensional data allows for 3-dimensional convolution. In the suggested model for HSI data, the 3-D kernel is convolved over numerous continuous bands in the input layer to produce the feature maps of the convolution layer, facilitating the extraction of the spectral information.

Equation (5) is used to compute the activation value in spatial dimension (x, y, z) in the i^th layer’s j^th feature map in 3-D convolution: $v_{i, j}^{x, y, z}$ . $v_{i, j}^{x, y, z} = φ (b_{i, j} + \sum_{τ = 1}^{d_{l - 1}} \sum_{λ = - η ρ = - γ}^{η} \sum_{σ = - δ}^{γ} W_{i, j, τ}^{σ, ρ, λ} \times v_{i - 1, τ}^{x + σ, y + ρ, z + λ})$ (5) where: φ seems to be the activation function, b_i,j denotes the bias parameter for i^th layers and j^th of the feature map, d_l-1 indicates the total feature map in layer l, and w_i,j would seem to be the kernel depth for layers i^th and i^th of the feature map. Kernel width and height are 2_γ + 1 and 2_δ + 1.

The usage of both spectral and spatial characteristics has grown in popularity. HSI was originally a 3D hypercube with spatial and spectral continuity. With the aid of 3D convolutions, commonly referred to as the spectral-spatial classification approach, it is simple to handle HSI CNN classifications. Using this method, connected spectral and spatial data from 3D spaces may be processed simultaneously. Comparing CNNs to conventional models, they have an added advantage. Its excellent capacity to detect spectral and spatial features makes it a prominent model employed in the HSI classification. The proposed 3D model comprises four 3-D convolutional layers are deployed before the flatten layer to make sure the model is able to discriminate the spatial information within different spectral bands without any loss. Further details regarding the proposed model can be found in Table 1.

Table 1

Summary of proposed 3D CNN classification model architecture

Layer	Output shape	Parameters
Input layer	25×25×30, 1	0
3D conv 1	23×23×24, 8	512
3D conv 2	21×21×20, 16	5776
3D conv 3	19×19×18, 32	13856
3D conv 4	17×17×16 64	331840
Flatten	18496	0
FC1	256	4735232
Dropout 1	256	0
FC2	128	32896
Dropout 2	128	0
FC3	64	2064

Generally supervised procedures supported by the gradient descent optimization method are employed to train the CNN’s kernel weight w as well as bias b. However, the sigmoid activation function performs the classification at the last layer. The first two layers of the fully connected layers employ a dropout approach, which randomly inhibits certain neurons based on a preset probability value to prevent overfitting’s incidence in DL architectures. The softmax function is used as an activation function in the output layer.

A deep three-dimensional convolutional neural network with severe over-fitting to the training data makes up the proposed architecture shown in Fig. 2(b). To deepen and improve the accuracy of the classification, changes to the layer count of the overall three-dimensional model are offered. The suggested architecture classifies the sample spatial contexts rather than classifying pixels by pixels. Eight filters make up the first 3D convolutional layer, and the kernel size is set to 23×23×24. Convolution layer processing of spectral and spatial data during training and testing its main task.

In the second 3D convolution layer, the spatial dimensions are reduced to 21×21×20 with 16 filters to extract the spatial features from the HSI. The third 3D convolution layer further reduces the dimensions into a 19×19×18 kernel with 32 filters. The fourth 3D convolution layer further reduces the dimensions into 17×17×16 kernel with 64 filters. After each convolution, a batch normalization layer is included to perform normalization process. Three fully connected neural networks are used in the proposed architecture which maps the input layers to different classes. The dropout values are selected fully connected layers are 0.1%, 0.3% and 0.5% respectively. Rectified linear unit (ReLU) activation function is used to reduce the spectral information after each layer. The presented model is trained using Adam optimizer which optimizes the layer weights and learning rate. In order to avoid data over-fitting, the dropout function is included before each fully connected layer. Layer details of the proposed architecture are listed in Table 1.

4 Result and discussion

The performance analysis of proposed hyperspectral image classification using 3D CNN model is experimentally verified using in Vivo Human Brain HSI Dataset [27 –31] which includes 22 patient HSI. The dataset includes more than 3 lakh samples of spectral signatures which are labeled based on the angle mapper algorithm. Four different categories of data are included such as Tumor Tissue (TT), Normal Tissue (NT), Background Elements (BE), and Blood Vessels (BV). The background elements include the substances that are present in the surgical scenario. For experimentation six hyper-spectral images are considered and it is labelled into four classes as Tumor Tissue (TT), Normal Tissue (NT), Background Elements (BE), and Blood Vessels (BV). The proposed model experimentation is performed in MATLAB R2019 installed in an Intel i5 processor 2.40 GHz with 16GB memory [36 –39]. In order to train test, and validate the conventional SVM, CNN, and proposed model the data is divided into 70:20:10 ratio, and 5-fold cross-validation is used in the proposed model experimentation. The epoch was initialized at 200, however, there is no record of improvement after 50 epochs for the proposed approach. The K-cross-validation approach (with k = 5) was used in order to assess the predictive ability of the model to overcome under-fitting and over-fitting issues. To properly assess the trained model, fivefold cross-validation was carried out. Utilizing all of the dataset’s data points leads to reduced bias, which is the major benefit of cross validation. Data is split into five sections, four of which are utilized as the training set while one serves as the validation set, in order to execute 5-fold cross-validation. The five iterations of the validation set are chosen so that there is no overlap between the images.

Only four of the eight patients were initially impacted by glioblastoma (GBM) tumors. In this tumor labeled proposed work since, the imaging circumstances were insufficient for performing the initial labeling. The remaining GBM tumor images were added to the database, but no tumor samples were taken into account. In total, six HSI images (P008-01, P008-02, P012-01, P012-02, P015-01, and P020-01) were labeled with the four studied classes (NT, TT, HT, and BG) and were employed as test datasets. The samples from the data set were provided in Fig. 3 in which the first row represents the synthetic RGB images. A yellow line in the image indicates the tumor portion and the respective ground truths are presented in the second row.

Fig. 3

Samples images from the dataset.

The average time taken by the conventional model to train and test the HSI is fifteen minutes whereas the proposed approach took an average of 80 s to train and classify the dataset. The final results obtained by the proposed and conventional models are depicted in Fig. 4. The first row in Fig. 4 indicates the actual dataset image which is a synthetic RGB image. The second row in Fig. 4 depicts the proposed 3D-CNN results. The third, fourth and fifth row depicts the results obtained for linear SVM [31, 32], CNN [33, 34], and hybrid 3D-2D CNN [35] approaches with the same HSI images.

Fig. 4

(a) Synthetic RGB image from dataset, (b) Proposed 3D-CNN results, (c) SVM based classification maps, (d) Conventional CNN based classification maps, (e) Hybrid 3D-2D CNN generated classification maps.

Figure 4 qualitative results make it possible to assess the outcomes of the classification of all HSI images, including those with unlabeled pixels. The tumor region is indicated on the synthetic RGB images in Fig. 4a by a yellow line, and the classification maps from the 3D CNN technique are shown in Figs. 4b to 4d. In these classification findings, the suggested strategies outperform the SVM strategy in terms of labeling the pixels in the tumor area.

Except for the BG class, where the suggested technique demonstrates an accurate identification of the parenchymal area (exposed brain surface) in images P008-01, P008-02, and P012-02, the qualitative results for the other tissue classes are remarkably similar. Additionally, it can be deduced from these data that the primary blood veins in images P008-02 and P015-01 were misclassified as background, contributing to the low classification performance. This phenomenon is not present in 3D-CNN, where the BV zones are typically well defined.

The performance metrics of proposed and conventional models are measured in terms of different metrics like accuracy, specificity, sensitivity, Matthew’s correlation coefficient and F1-score, dice. In order to compute these metrics, binary classification results are obtained as true positive (TP), true negative (TN), false positive (FP) and false negative (FN). The mathematical formulations for the performance metrics are listed as follows. $Accuracy (AC) = \frac{Sum of true postive and true negative}{sum of all values}$ (6) $Specificity (Sp) = \frac{True negative}{sum of true negative and false positive}$ (7) $Sensitivity (Se) = \frac{True positive}{sum of true positive and false negative}$ (8) ${Matthew}^{'} s correlation coefficient (M_{CC}) = \frac{[TP * TN] - [FP * FN]}{\sqrt{(TN + FN) (TP + FN) (TN + FP) (TP + FP)}}$ (9) $F 1 - score = \frac{TP}{TP + \frac{1}{2} (FP + FN)} or 2 * \frac{Precision * Recall}{Precision + Recall}$ (10) $Dice (F) = \frac{TP}{2 TP + FN + FP}$ (11)

Figure 5 6 depicts the obtained sensitivity and specificity values for the images used in the experimentation. For each image the class values are obtained and presented in the figures. Image id’s like 008-01, 008-02, 012-01, 012-02, 015-01, 020-01 and its specificity, sensitivity values for classes like Tumor Tissue (TT), Normal Tissue (NT), Background Elements (BE), and Blood Vessels (BV) are considered for the analysis. Besides that, the sensitivity performance for the TT and BV classes is also equivalent.

Fig. 5

Sensitivity analysis.

Fig. 6

Specificity analysis.

Similarly, the accuracy values are plotted in Fig. 7 for all the four classes for all the patient images. Figure 8 depicts the specificity analysis of proposed model and linear SVM and hybrid CNN models. The minimum value of proposed model indicates the lowest occurrence of false positive. Higher the specificity value observed for the conventional methods indicates the results are inaccurate due to false positive values. For all the four classes the performance of proposed 3D-CNN model is much better than the conventional methods. The accuracy performance in Fig. 7 emphasizes how consistent all indicators are with the classification. The tumor areas in the HSI images can, however, be better defined by using M_CC. For instance, in images P012-02 and P015-01, certain BV pixels in the tumor area were misclassified; however, this is corrected by using M_CC (Fig. 4d). Additionally, M_CC in the same image displays a uniform tumor area in relation to other distances. Noting that the synthetic RGB image for the P020-01 image reveals that the designated tumor area displays a similar colorization to the NT class, which is consistent with the result, none of the metrics are able to identify the TT class. As noted in such prior work, the lack of a more comprehensive database that accounts for the inter-patient spectral variability may have contributed to this misdiagnosis of the TT class.

Fig. 7

Accuracy analysis.

Fig. 8

Specificity comparative analysis.

The sensitivity analysis of proposed and conventional models is depicted in Fig. 9. It can be observed that maximum sensitivity values are obtained by the proposed model than the conventional model. Generally, in the classification process, higher sensitivity values indicate the higher accuracy of the classification model which is observed in the proposed approach. The average sensitivity value obtained by the linear SVM based approach is 93 % which is 5% lesser than the proposed approach.

Fig. 9

Sensitivity comparative analysis.

Whereas hybrid CNN model exhibits 95% average sensitivity which is 3% lesser than the proposed approach. A maximum of 98% sensitivity is produced by the proposed model exhibits the better classification performance in hyperspectral image classification.

Comparative analysis of Mathew correlation coefficient is depicted in Fig. 10. Generally higher Matthew’s correlation coefficient value indicates the best performance which is near to 100% and average performance is nearly 50% and worst performance is indicated below 50%. From the results it could be observed that all the three models exhibit better performance however the proposed 3D-CNN model exhibits better performance than conventional models. The average MCC value obtained by the linear SVM based approach is 91.5% and CNN based approach attains 94.7% which is lesser than the proposed model MCC value 98.5%.

Fig. 10

Comparative analysis of MCC.

Figure 11 depicts the comparative analysis of proposed model and conventional models F1-score for all the four classes. It can be observed from the results; the maximum F1-score is attained by the proposed 3D-CNN based classification model. Higher the F1-score indicate the perfect classification which is observed in the proposed model results. The maximum F1-score attained by the proposed model is 97.25% which is 2.5% greater than CNN model and 11% greater than SVM based classification model. Figure 12 depicts the accuracy analysis for all the three models. The results clearly depict, that the maximum accuracy is attained by the proposed 3D-CNN based classification process. The maximum accuracy attained by the proposed model is 96.13% and Conventional CNN based approach attains 94.4% and SVM based approach attains 94.3% classification accuracy.

Fig. 11

F1-score comparative analysis.

Fig. 12

Accuracy comparative analysis.

As shown in Fig. 13, the Dice coefficient score of the 3D CNN model improved significantly compared with that of the SVN model, and it was comparable to that of the CNN model. There are three factors (dropout, ReLU, and the size of the spatial window) that influence the final classification accuracy significantly. Figure 14 shows the precision comparison.

Fig. 13

Dice score comparative analysis.

Fig. 14

Precision comparative analysis.

The characteristics of the test datasets are described as follows: continuous variables are expressed as the mean±SD. Figure 15 shows the histogram of errors found from modeling the best 3D CNN model. The blue and the green bars are utilized to denote the training and validation data, respectively. The conclusion drawn from Fig. 15 is that the majority of standard errors found values ranged between –0.02834 and 0.002585. The model showed high performance in the test dataset, with an average ROC of 0.99 for the proposed 3D CNN model, 0.939 to SVM and 0.919 for hybrid CNN model in Fig. 16.

Fig. 15

Error histogram.

Fig. 16

ROC curve comparison.

Figure 17 displays the suggested 3D CNN’s confusion matrix for the Invivo human brain dataset. The proposed 3D CNN was also executed with the obtained paired CT and MR datasets from the Gothenburg H70 Birth Cohort Studies. The input samples were shown in Fig. 18. Figure 19 shows the predicted tissue class maps of input CT images. The suggested 3D CNN’s Confusion matrix for the CT and MR datasets from the Gothenburg H70 is shown in Fig. 20 with an accuracy of 95%. The average values obtained by the proposed 3D-CNN model and conventional SVM, CNN based hyperspectral image classification approaches are presented in Table 2 for sensitivity, specificity, Mathew correlation coefficient, F1-score and accuracy. Further to validate the superior performance of proposed model existing methodologies like Min-Max NN [31, 32], ANN [32], SVM, KNN [33], and SVM [33] are compared in terms of accuracy, Specificity, and Sensitivity presented in Table 3. It can be observed from the results; the maximum performance is attained by the proposed model compared to existing methods.

Fig. 17

Confusion matrix for the Invivo human brain dataset with proposed 3D CNN.

Fig. 18

Input samples of the Gothenburg H70 dataset.

Fig. 19

classification maps of input CT images.

Fig. 20

Confusion matrix of the proposed 3D CNN for the CT and MR datasets from the Gothenburg H70.

Table 2

Performance comparative analysis

S. No	Methods	Specificity (%)	Sensitivity (%)	MCC (%)	F1-Score (%)	Accuracy (%)
1	Linear SVM [8]	86.25	93.75	91.50	88.25	94.325
2	CNN [11]	84.50	95.75	94.75	95.00	94.475
3	Proposed 3D CNN	98.00	97.50	98.50	97.25	97

Table 3

Comparison with existing methods

S. No.	Methods	Accuracy	Specificity	Sensitivity
1	Min-Max NN [32]	94.2%	92.00	93.50
2	ANN, SVM, KNN [31]	84%	95.3	93.6
3	SVM [33]	93%	91	88
4	Proposed method	97%	98.00	97.50

The main goal was to employ HSI to develop a methodology to discriminate between tumor and normal brain tissue in surgical-time during surgical procedures. The integration of HSI in a surgery system could have a direct impact on the patient outcomes. Potential benefits would include allowing confirmation of complete resection during the surgical procedures, avoiding complications due to the brain shift phenomenon, and providing confidence that the goals of the surgery have been successfully achieved.

The three key variables that affect the 3D-CNN classification framework’s classification performance are the number of kernels, the spectral depth of the kernels, and the spatial dimension (or “window size") of the samples that need to be classified. In this case, a 3D-CNN model is trained using a different structure (four convolutional layers and two fully connected layers) and a different number of kernels. The scenarios when the first layer has two 3D kernels showed the best performance across all datasets. This suggests that using more kernels may reduce accuracy since it makes training harder and increases the likelihood that the model will over-fit the training set.

The structure with two 3D kernels in the first layer and four 3D kernels in the second layer provides the best 3D-CNN configuration, according to the HSI classification on the two datasets. In general, the spectral-spatial features that are retrieved using data from a neighborhood region serve to reduce intra-class variation, improving classification performance. However, a larger region could contain more noise, particularly if the pixel is situated near the edge or corner of a category. The fourth 3D convolution layer reduces the dimensions into 17×17×16 for extracting the spectral-spatial features, performed the best, which appears to be a reasonable practical decision.

5 Conclusion

A hyperspectral brain tumor image classification using 3D-Convolutional neural network is presented in this research work. Classification of hyperspectral image is complex than the normal image classification process. The spatial and temporal features are obtained in the proposed 3D-CNN model to classify the tumor image. Benchmark brain tumor image dataset were used for experimentation and compared with conventional SVM and convolutional neural network models for better validation. Performance metrics like accuracy, specificity, sensitivity, Mathew correlation coefficient and f1-score are used to demonstrate the performance of proposed model and conventional models as a comparative analysis. Results presents the improved performance of proposed approach compared to conventional approaches. The future direction of research work can be directed based on hybrid DL approaches to enhance the classification accuracy.

References

Shantha Rajiah

, Weber

, Loewen

, et al., Dynamic CT Angiography in Vascular Imaging: Principles and Applications, Radio Graphics 42 (2022), E224–E225.

Alhassan

A.M.

, Mohd Nazmee

and Zainon

, BAT Algorithm with fuzzy C-Ordered Means (BAFCOM) Clustering Segmentation and Enhanced Capsule Networks (ECN) for Brain Cancer MRI Images Classification, IEEE Access 8 (2020), 201741–201751.

Asif

, Shah

M.A.

, Khattak

H.A.

, et al., Intracranial Hemorrhage Detection Using Parallel Deep Convolutional Models and Boosting Mechanism, Diagnostics 13 (2023), 652.

Fei

, Hyperspectral imaging in medical applications, Data Handling in Science and Technology 32 (2019), 523–565.

Ranjbarzadeh

, Jafarzadeh Ghoushchi

, Tataei Sarshar

, et al., ME-CCNN: Multi-encoded images and a cascade convolutional neural network for breast tumor segmentation and recognition, Artif Intell Rev 121 (2023), 98–110.

Murugesan

, Kaliannan

, Balraj

, et al., A Hybrid Deep Learning Model for Effective Segmentation and Classification of Lung Nodules from CT Images, Journal of Intelligent and Fuzzy System 42 (2021), 2667–26791.

Chen

, Chai

J-W.

, Chen

H-C.

, et al., An Iterative Mixed Pixel Classification for Brain Tissues and White Matter Hyper intensity in Magnetic Resonance Imaging, IEEE Access 7 (2019), 124674–124687.

Sadashivaiah

, Tippani

, Page

S.C.

, et al., SUFI: an automated approach to spectral unmixing of fluorescent multiplex images captured in mouse and post-mortem human brain tissues, BMC Neurosci 24 (2023), 1–16.

Grigoroiu

, Yoon

and Bohndiek

S.E.

, Deep learning applied to hyperspectral endoscopy for online spectral classification, Scientific Reports 10 (2020), 1–10.

10.

Ali

M.B.

, Gu

, Lidemar

, et al., Prediction of glioma-subtypes: comparison of performance on a DL classifier using bounding box areas versus annotated tumors, BMC Biomed Eng 4 (2022), 1–18.

11.

Gurunathan

and Krishnan

, A Hybrid CNN-GLCM Classifier for Detection and Grade Classification of Brain Tumor, Brain Imaging and Behavior 16 (2022), 1410–1427.

12.

Hebbale

, Ashapurna Marndi

P.N.

, Achyutha

, et al., Automated medical image classification using deep learning, International Journal of Health Sciences 6 (2022), 1650–1667.

13.

Fabelo

, et al., In-Vivo Hyperspectral Human Brain Image Databasefor Brain Cancer Detection, IEEE Access 7 (2019), 39098–39116.

14.

Ibrahim

, Aboughaleb

and El-Sharkawy

Y.H.

, Hyperspectral imaging for diagnosis and detection of ex-vivo breast cancer, Photo diagnosis and Photodynamic Therapy 31 (2020), 1–11.

15.

Jaware

, Badgujar

, Patil

, et al., Automatic Segmentation of Infant Brain Mri Using Soft Computing Techniques, ICTACT Journal on Soft Computing 12 (2022), 2651–2656.

16.

Padma

, Kumari

C.U.

, Yamini

, et al., Image Segmentation using Mask R-CNN for Tumor Detection from Medical Images, 2022 International Conference on Electronics and Renewable Systems (ICEARS), Tuticorin, India, (2022), 1015–1021.

17.

Rodrigues

E.M.

and Hemmer

, Trends in hyperspectral imaging: from environmental and health sensing to structure-property and nano-bio interaction studies, Anal Bioanal Chem 414 (2022), 4269–4279.

18.

Albert

J.R.

, Selvan

, Sivakumar

, et al., An Advanced Electrical Vehicle Charging Station Using Adaptive Hybrid Particle Swarm Optimization Intended for Renewable Energy System for Simultaneous Distributions, Journal of Intelligent & Fuzzy Systems 43 (2022), 4395–4407.

19.

Manni

, Cai

, van der Sommen

, et al., Hyperspectral imaging for tissue classification in glioblastoma tumor patients: a deep spectral-spatial approach, Medical Imaging: Image-Guided Procedures, Robotic Interventions, and Modeling 11598 (2021), 260–268.

20.

Liu

, Zhang

, Lian

and Shen

, Weakly Supervised Deep Learning for Brain Disease Prognosis Using MRI and Incomplete Clinical Scores, IEEE Transactions on Cybernetics 50 (2020), 3381–3392.

21.

Alanazi

M.F.

, Ali

M.U.

, Hussain

S.J.

, et al., Brain Tumor/Mass Classification Framework Using Magnetic-Resonance-Imaging-Based Isolated and Developed Transfer Deep-Learning Model, Sensors 22 (2022), 372.

22.

Wang

, Mao

, Wang

, et al., Automatic generation of pathological benchmark dataset from hyperspectral images of double stained tissues, Optics & Laser Technology 163 (2023), 109331.

23.

Kafrawy

P.E.l.

, Fathi

, Qaraad

, et al., An Efficient SVM-Based Feature Selection Model for Cancer Classification Using High-Dimensional Microarray Data, IEEE Access 9 (2022), 155353–155369.

24.

Qiu

, Zhao

, Jiang

, et al., Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges, Information Fusion 80 (2022), 241–265.

25.

Raja

P.M.

, Brain tumor classification using a hybrid deep auto-encoder with Bayesian fuzzy clustering-based segmentation approach, Biocybernetics and Biomedical Engineering 40 (2020), 440–453.

26.

Rao

S.K.V.

and Lingappa

, Image analysis for MRI based brain tumour detection using hybrid segmentation and deep learning classification technique, International Journal of Intelligent Engineering and Systems 12 (2019), 53–62.

27.

Wang

, Bie

, Li

, et al., Small Water Body Detection and Water Quality Variations with Changing Human Activity Intensity in Wuhan, Remote Sens 14 (2022), 200.

28.

Vankdothu

, Hameed

M.A.

, Ameen

and Unnisa

, Brain image identification and classification on Internet of Medical Things in healthcare system using support value based deep neural network, Computers and Electrical Engineering 102 (2022), 108196.

29.

Ramaraju

S.K.

, Kaliannan

, Androse Joseph

, et al., Design and Experimental Investigation on VL-MLI Intended for Half Height (H-H) Method to Improve Power Quality Using Modified Particle Swarm Optimization (MPSO) Algorithm, Journal of Intelligent & Fuzzy Systems 42 (2022), 5939–5956.

30.

and Wang

, Segmentation of corpus callosum based on tensor fuzzy clustering algorithm, Journal of X-Ray Science and Technology 29 (2012), 931–944.

31.

Jaware

, Badgujar

, Patil

, et al., Automatic Segmentation of Infant Brain MRI Using Soft Computing Techniques, ICTACT Journal on Soft Computing 12 (2022), 2651–2656.

32.

Bangare

S.L.

, Classification of optimal brain tissue using dynamic region growing and fuzzy min-max neural network in brain magnetic resonance images, Neuroscience Informatics 2 (2022), 100019.

33.

Sreelakshmi

S.G.

and Sherly

, Alzheimer’s Disease Classification from Cross-sectional Brain MRI using Deep Learning, 2022 IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (SPICES) 1 (2022), 401–405.

34.

Babypriya

, et al., An Experimental Simulation Testing of Single–diode PV Integrated MPPT Grid-tied Optimized Control Using Grey Wolf Algorithm, Journal of Intelligent & Fuzzy Systems 43 (2022), 5877–5896.

35.

Yoon

, Hyperspectral Imaging for Clinical Applications, Bio Chip J 16 (2022), 1–12.

36.

Zhang

, Wang

, Qiu

, et al., 3D-PulCNN: Pulmonary cancer classification from hyperspectral images using convolution combination unit based CNN J, Bio-photonics 14 (2021), e202100142.

37.

Periasamy

, Kaliannan

, Selvaraj

, et al., Various PSO methods investigation in renewable and nonrenewable sources, , International Journal of Power Electronics and Drive Systems 13 (2498), 2505.

38.

Kaliannan

, Albert

J.R.

, Begam

D.M.

and Madhumathi

, Power Quality Improvement in Modular Multilevel Inverter Using for Different Multicarrier PWM, European Journal of Electrical Engineering and Computer Science 5 (2021), 19–27.

39.

Thangamuthu

, Albert

J.R.

, Chinnanan

and Gnanavel

, Design and development of extract maximum power from single-double diode PV model for different environmental condition using BAT optimization algorithm, J Intell Fuzzy Syst 43 (2022), 1091–1102.