A multi-modal and multi-stage region of interest-based fusion network convolutional neural network model to differentiate progressive mild cognitive impairment from stable mild cognitive impairment

Abstract

Background

Accurately differentiating stable mild cognitive impairment (sMCI) from progressive MCI (pMCI) is clinically relevant, and identification of pMCI is crucial for timely treatment before it evolves into Alzheimer's disease (AD).

Objective

To construct a convolutional neural network (CNN) model to differentiate pMCI from sMCI integrating features from structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) images.

Methods

We proposed a multi-modal and multi-stage region of interest (ROI)-based fusion network (m2ROI-FN) CNN model to differentiate pMCI from sMCI, adopting a multi-stage fusion strategy to integrate deep semantic features and multiple morphological metrics derived from ROIs of sMRI and PET images. Specifically, ten AD-related ROIs of each modality images were selected as patches inputting into 3D hierarchical CNNs. The deep semantic features extracted by the CNNs were fused through the multi-modal integration module and further combined with the multiple morphological metrics extracted by FreeSurfer. Finally, the multilayer perceptron classifier was utilized for subject-level MCI recognition.

Results

The proposed model achieved accuracy of 77.4% to differentiate pMCI from sMCI with 5-fold cross validation on the entire ADNI database. Further, ADNI-1&2 were formed into an independent sample for model training and validation, and ADNI-3&GO were formed into another independent sample for multi-center testing. The model achieved 73.2% accuracy in distinguishing pMCI and sMCI on ADNI-1&2 and 75% accuracy on ADNI-3&GO.

Conclusions

An effective m2ROI-FN model to distinguish pMCI from sMCI was proposed, which was capable of capturing distinctive features in ROIs of sMRI and PET images. The experimental results demonstrated that the model has the potential to differentiate pMCI from sMCI.

Keywords

Alzheimer's disease fusion network hierarchical convolutional neural network mild cognitive impairment morphological metrics multimodal MRI

Introduction

Mild cognitive impairment (MCI) is an intermediate stage between normal aging and Alzheimer's disease (AD), and the prevalence of MCI in adults older than 60 is approximately 6.7% to 25.2%,¹ displaying a high risk of progression to clinically probable AD without early intervention.² MCI can be further divided into stable MCI (sMCI) and progressive MCI (pMCI). MCI that transitions to AD during the follow-up period (typically 36 months) is classified as pMCI, while MCI that does not transition to AD during the follow-up period is classified as sMCI.^3,4 Differentiating sMCI from pMCI is of great significance for AD early diagnosis and treatment. At present, the conventional imaging-based MCI recognition methods mainly included radiomics-based models and convolutional neural network (CNN)-based models.

The radiomics-based models generally adopted morphological metrics and texture features from the selected dementia-related regions of interest (ROIs) for MCI recognition, which are subsequently modelled by machine learning classifiers. Several studies demonstrated that morphological features of specific anatomical regions may have the potential to identify MCI and AD. Frisoni et al.⁵ found atrophy of medial temporal structures was a feasible diagnostic biomarker for MCI recognition and the ratio of hippocampus to whole brain could quantify the progression of disease. Gupta et al.⁶ found the morphological characteristics of hippocampus, amygdala, and entorhinal cortex could sever as effective biomarkers to recognize MCI and AD. Abrol et al.⁷ concluded that the hippocampus and amygdala subcortical regions in the medial temporal lobe were the most informative regions for early-stage MCI recognition. Schmitter et al.⁸ extracted gray matter volume (GMV) from 9 ROIs for MCI recognition and achieved the accuracy of 71% for MCI versus normal controls (NC). Ma et al.⁹ integrated GMV, Jacobian determinant, cortical thickness, sulcus depth, gyrification index, and fractal dimension to discriminate MCI from NC with random forest and obtained an accuracy of 80% for MCI patients. Long et al.¹⁰ combined GMV, white matter volume and cerebrospinal fluid volume to identity MCI from NC through support vector machine and got the best accuracy of 92% for MCI identification.

Although these radiomics-based models had achieved encouraging results, they usually adopted some predefined features for MCI identification, while these selected features may be varied across different populations. In contrast, CNN-based models could extract deep semantic features from images by an end-to-end manner, which has been demonstrated to be a powerful tool for MCI recognition. Payan et al.¹¹ constructed a 3D convolutional neural network to recognize MCI and AD, and achieved an accuracy of 92.11% for MCI versus NC. Huang et al.¹² constructed a multi-modal 3D convolutional neural network to identify MCI, integrating two types of modality data, structural magnetic resonance imaging (sMRI) and 18-Fluoro-DeoxyGlucose positron emission tomography (FDG-PET). They obtained 326 cases of pMCI and 441 cases of sMCI from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, achieving an identification accuracy of 72.2% between sMCI and pMCI. Liu et al.¹³ proposed a multi-modality cascaded CNNs to differentiate sMCI from pMCI, 76 pMCI and 128 sMCI subjects with PET and sMRI were selected and achieved an accuracy of 82.9% for pMCI versus NC. However, the existed CNN-based models for MCI diagnosis had some limitations. Firstly, some models adopted whole brain images as input for model training, which would introduce potential bias from unrelated brain regions. Secondly, some models just utilized single modality MRI images, neglecting the informative features from multi-modality. Thirdly, some models used a simple multilayer perceptron (MLP) layer to fuse deep semantic features from multi-modality MRI images, which would lose the complementary information between multi-modality. Finally, most CNN-based models did not involve the contributions of morphological metrics, which may lose the most informative features for MCI recognition.

In this paper, we proposed a multi-modal and multi-stage ROI-based fusion network (m2ROI-FN) CNN model to differentiate pMCI from sMCI. Among them, the ROI-based hierarchical CNN (ROIH-CNN) model were applied on two modality neuroimaging of sMRI and PET, the extracted deep semantic features and multiple morphological metrics derived from two modality images were integrated together for final MCI recognition. In particular, a series of cascaded 3D CNNs were applied to extract low-level deep semantic features, and the specially designed multi-modality integration module were utilized to refine the low-level features from structural MRI and PET into high-level deep semantic features. Finally, ROI-based MLP classifier was built to fuse the high-level features into global features, and a softmax layer was adopted to generate a subject-level recognition outcome.

Methods

Data collection

Subjects adopted in this paper were selected from Alzheimer's Disease Neuroimaging Initiative (ADNI) database (http://adni.loni.usc.edu). ADNI dataset is a longitudinal multi-center study aimed to explore clinical, imaging, genetic, and biochemical biomarkers for the early detection and tracking of AD.¹⁴ The MCI subjects were furtherly split into two groups, including pMCI and sMCI. pMCI was determined according to the criteria whether the MCI patient would convert to AD within 36 months after baseline time, and sMCI was MCI patient who did not progress to AD during 36 months following baseline visit. In this study, we collected all patients with both T1-weighted structural MRI (volumetric 3D Magnetization Prepared-Rapid Gradient Echo imaging) and FDG-PET images. More details about structural MRI and FGD-PET can be found at ADNI official website. Totally, 1267 subjects were exploited, including 1052 subjects in ADNI-1&2 (sMCI 229, pMCI 153, NC 427, AD 243) and 215 subjects in ADNI-3&GO (sMCI 117, pMCI 27, NC 8, AD 63), the demographic detail of the subjects were shown in Table 1.

Table 1.

Demographic information of the sample used (ADNI-1&2, ADNI-3&GO).

Dataset	Group	Number	Age	Gender	MMSE
ADNI-1&2	NC	427	74.1 $\pm$ 5.8 [90.2,56.3]	208/219	27.4 $\pm$ 1.2
	AD	243	75.1 $\pm$ 7.7 [90.4,55.2]	146/97	25.4 $\pm 2$ .2
	sMCI	229	72.5 $\pm$ 7.4 [88.7,55.2]	143/86	27.0 $\pm$ 1.0
	pMCI	153	73.8 $\pm 6$ .9 [87.8,55.1]	85/68	26.6 $\pm$ 1.4
ADNI-3&GO	NC	8	66.8 $\pm 6$ .3 [77.8,55.8]	6/2	28.9 $\pm$ 1.1
	AD	63	74.1 $\pm 8$ .7 [90.6,55.3]	37/26	24.9 $\pm$ 2.5
	sMCI	117	71.9 $\pm$ 7.8 [89.5,60.3]	65/52	27.1 $\pm$ 1.1
	pMCI	27	74.8 $\pm 7$ .1 [90.6,55.1]	15/12	26.6 $\pm$ 1.7

Subject age is reported as mean ± standard deviation. The gender is reported as Male/Female. MMSE: Mini-Mental State Examination.

Data processing

T1-weighted MRI images and FGD-PET images were preprocessed with FreeSurfer software.¹⁵ For the T1-weighted MRI images, ‘recall-all’ processing stream with default parameters was used for skull removal, registering images into MNI standard brain template (MNI305, voxel size: 1*1*1), cortical and subcortical surfaces reconstruction, as well as volumetric reconstruction. For the FDG-PET images, ‘Bet’ command was performed for skull removal, ‘mri_coreg’ command was used for linear registration the images into corresponding T1-weighted MRI images (Degrees of Freedom, -dof12). For the dynamic FGD-PET images (PET data has multiple frames) were averaged all the time frames together by ‘mri_concat’ command. After images preprocessing, Z-score intensity normalization was employed for all voxel.

Morphological metrics

In this study, based on T1-weighted MRI images, five morphological metrics from three cerebral cortex regions and two metrics from two subcortex regions were extracted by FreeSurfer software (version 7.1). Specifically, thickness average (TA), surface area (SA), Gaussian curvature (GC), GMV, and local gyrification index (LGI) were computed from entorhinal cortex, insula and middle temporal gyrus, volume in cubic millimeters (Vmm³), and Intensity normMean (IM) were computed from hippocampus and amygdala. Entorhinal cortex, insula and middle temporal gyrus were segmented according to subject-level Automatic Parcellation of Cortical (Aparc) atlas,¹⁶ hippocampus and amygdala were segmented according to subject-level Automatic Subcortical Segmentation (Aseg) atlas. In order to eliminate the influence of intracranial volume (ICV) on Vmm³, we adjusted the Vmm³ metric according to ICV to accommodate individual differences, the adjusted Vmm3 metric can be formulized as:

V_{i}^{'} = \frac{V_{i}}{I C V}

Where i represents different ROIs, V is Vmm³ metric,

V^{'}

is adjusted Vmm3 metric, ICV is the brain mask calculated by FreeSurfer. We computed each center of five ROIs according to Aparc + Aseg atlas, and subject-level 3D ROI patches were cropped according to the centers. For the same ROI of different subjects, the patches were padded to a uniform size to ensure consistent input for CNN feed-forward processing. For the different ROIs, the patches were cropped based on the size of the ROI, since the sizes of different anatomical regions vary significantly (e.g., temporal lobe and amygdala). Our cropping strategy is designed to preserve the entire ROI region while minimizing the non-ROI areas. Due to the symmetry of the left and right cerebral hemispheres, ten ROIs were selected in this study, the patch size, and the computed morphological measures for each ROI were shown in Table 2.

Table 2.

Anatomical region, morphological characteristics and Patch size corresponding to different ROI.

ROI	Morphological Metrics	Patch size
Left entorhinal cortex	TA, SA, GC, GMV, LGI	(29,30,40)
Left hippocampus	Vmm3, IM	(34,35,53)
Left amygdala	Vmm3, IM	(25,22,24)
Left middle temporal gyrus	TA, SA, GC, GMV, LGI	(46,68,103)
Left insula	TA, SA, GC, GMV, LGI	(48,53,72)
Right entorhinal cortex	TA, SA, GC, GMV, LGI	(26,39,43)
Right hippocampus	Vmm3, IM	(36,37,48)
Right amygdala	Vmm3, IM	(26,24,24)
Right middle temporal gyrus	TA, SA, GC, GMV, LGI	(49,65,110)
Right insula	TA, SA, GC, GMV, LGI	(48,51,69)

TA: thickness average; SA: surface area; GC: Gaussian curvature; GMV: gray matter volume; LGI: local gyrification index; Vmm³: volume in cubic millimeters; IM: intensity mean.

m2ROI-FN model

Our m2ROI-FN model was illustrated in Figure 1(A), which is composed of four components, namely ROI-based CNNs for PET images, ROI-based CNNs for MRI images, multi-modality integration (MMI) module, and ROI-based MLP classifier. The ROI-based CNNs are used to generate ROI-based low-level semantic features. The ROI-based CNN consists of base model and ROI Convolutional (ROI-Conv) module. The base model consists of four sequential convolutional blocks, including 3D Convolutional, BatchNormal, Relu and MaxPooling layers, as shown in blue box in Figure 1(B). The base model structures of different ROIs are completely identical, but the parameters are not shared. The ROI-Conv module receives outputs from the base model to generate ROI-based low-level semantic features, as shown in green box in Figure 1(B). The module structures corresponding to different ROIs are shown in Table 3.

Figure 1.

Our multi-modal and multi-stage region of interest (ROI) based fusion network (m2ROI-FN). Part A denotes the framework of our model, and part B denotes the 3D CNNs, and part C denotes the multi-modality integration (MMI) module. TA: thickness average; SA: surface area; GC: Gaussian curvature; GMV: gray matter volume; LGI: local gyrification index; Vmm3: volume in cubic millimeters; IM: intensity mean.

Table 3.

ROI-Conv module structures.

ROI	Module Structure	Feature Map Shape
Left Entorhinal Cortex	Conv (64,64,[2,2,3])	(2,2,3)
Left hippocampus	Conv (64,64,[3,3,4])	(3,3,4)
Left amygdala	Conv (64,64,[2,2,2])	(2,2,2)
Left middle temporal gyrus	Conv (64,64,[3,5,8])	(3,5,8)
Left insula	Conv (64,64,[3,4,5])	(3,4,5)
Right Entorhinal Cortex	Conv (64,64,[2,2,3])	(2,2,3)
Right hippocampus	Conv (64,64,[3,3,3])	(3,3,3)
Right amygdala	Conv (64,64,[2,2,2])	(2,2,2)
Right middle temporal gyrus	Conv (64,64,[4,5,7])	(4,5,7)
Right insula	Conv (64,64,[3,4,5])	(3,4,5)

The Conv3D is utilized to extract local features, and the 3D convolutional operation is expressed as:

F_{i}^{j} (x, y, z) = \sum_{k_{x}} \sum_{k_{y}} \sum_{k_{z}} F_{i}^{j - 1} (x + k_{x}, y + k_{y}, z + k_{z}) \times W_{i}^{j} (k_{x}, k_{y}, k_{z})

F

represents the feature maps, j denotes the

j_{t h}

layer and i is the feature map,

x, y, z

represent the pixel locations for a given feature map,

k_{x}, k_{y}, k_{z}

are the size of convolutional kernel corresponding to coordinates,

x, y, z

. Relu was utilized as the activation function for adding nonlinearity, After nonlinear activation layer, PadMaxPool3d¹⁷ with [2, 2, 2] kernal was adopted to reduce the dimension of the inputs and to save the most important features. The 3D MaxPooling operation can be formulized as:

M a x P o o l i n g (F_{i}^{j} (x, y, z)) = \max (\sum_{p_{x}} \sum_{p_{y}} \sum_{p_{z}} F_{i}^{j} (x + k_{x}, y + p_{y}, z + p_{z}))

p_{x}, p_{y}, p_{z}

are the kernal size of MaxPooling layer. After the sequential convolutional blocks, ROI patches

[B, C, H, W, D]

were converted to feature maps with shape

[B, C, 1, 1, 1]

, where B represents batch size and C denotes numbers of channels, then we flattened the maps to a feature vector

[B, C]

The specially designed MMI module shown by the part C in Figure 1 was adopted to incorporate the semantic features from the multi-modality images of the same ROI, generating local-stage multi-modality features. The multi-modality integration module consists of sequential fully connected layers with the number of neurons [128, 64], in which 128 stands for 64 MRI features plus 64 PET features. By the usage of integration module, the distinct information between different modality features would be preserved.

Finally, a ROI-based MLP classifier was utilized for MCI recognition, which consists of fully connected layers and softmax layer. The fully connected layers are used to concatenate the local-stage multi-modality features with multiple morphological metrics to generate global-stage deep semantic features, and the softmax layer is adopted for MCI and AD recognition. This multi-stage fusion architecture is designed based on the ‘late-fusion’ method. Comparing with ‘early-fusion’ methods, which integrates shallow features such as edges or textures, the proposed ROI-based MLP utilizes the ‘late-fusion’ pattern to capture global task-related features from local-stage to global-stage. The fully connected layers have the number of neurons [678, 100, 2], where 678 is 64 local features multiply 10 ROIs plus 30 cortex (TA, SA, GC, GMV, LGI from left and right entorhinal cortex, insula and middle temporal gyrus) and 8 subcortex (Vmm³, IM from left and right hippocampus and amygdala) morphological metrics. In this manner, the distinct information from different ROIs could be aggregated to precisely classify MCI and AD patients.

Model performance comparison

In this paper, five experiments were designed to evaluate performance of our m2ROI-FN model on the sMCI versus pMCI tasks. The main evaluation metrics included accuracy (ACC), sensitivity (SEN), specificity (SPE), and area under the receiver operating characteristic curve (AUC). The ablation experiments were shown in Table 4.

Table 4.

Ablation experiments of M2ROI-FN model.

Model inputs	Hipp	Amy	Ins	MTG	EC	LH	RH	MRI	PET	Mask	Mor
V1	√					√	√	√	√
V2		√				√	√	√	√
V3			√			√	√	√	√
V4				√		√	√	√	√
V5					√	√	√	√	√
V6	√	√				√	√	√	√
V7	√	√	√			√	√	√	√
V8	√	√	√	√		√	√	√	√
V9	√	√	√	√	√		√	√	√
V10	√	√	√	√	√	√		√	√
V11	√	√	√	√	√	√	√	√
V12	√	√	√	√	√	√	√		√
V13	√	√	√	√	√	√	√	√	√
V14	√	√	√	√	√	√	√	√	√	√	√
V15	√	√	√	√	√	√	√	√	√		√

Hipp: hippocampus; Amy: amygdala; Ins: insula; MTG: middle temporal gyrus; EC: entorhinal cortex; LH: left brain hemisphere; RH: right brain hemisphere. Masked denotes patch masked operation, and Mor denotes the morphological metrics.

In order to evaluate whether the m2ROI-FN model could effectively fuse the deep semantic features from different modality and different ROIs, we firstly compared the performance with the concatenating-based CNN (Cat CNN) model utilizing a simply fully connected layer to combine the learned features from multi-modality images (https://github.com/czp19940707/Cat-CNN). Notably, the whole brain images were applied on Cat CNN model training. The Cat CNN could be regarded as simplified version of B2,¹² whose backbone structure consisted of 5 sequential convolutional blocks that could be also seen in.¹⁷ The identical hyperparameter settings, e.g., learning rate, optimizer, loss function and epochs, and identical subjects were adopted for both our proposed model and the Cat CNN.

The second experiment was designed to verify the influences of different ROI combination. Firstly, m2ROI-FN models were trained with different single ROI regions as inputs, including bilateral hippocampus (V1), bilateral entorhinal cortices (V2), bilateral amygdalae (V3), bilateral middle temporal gyri (V4), bilateral insulae (V5). Secondly, m2ROI-FN models were trained with different ROI combinations as inputs, including 4-ROIs (bilateral hippocampal and amygdala–V6), 6-ROIs (bilateral hippocampal, amygdala, and insula–V7), 8-ROIs (bilateral hippocampal, amygdala, insula, and middle temporal gyrus–V8). Finally, the ROIs in the left and right brain hemisphere were respectively utilized as inputs to train m2ROI-FN model (right brain hemisphere–V9, left brain hemisphere–V10).

The third experiment was designed to compare the model performances between multi-modal fusion and single modality, and the m2ROI-FN model were respectively trained on single sMRI (V11), single PET (V12) and MRI + PET (V13) in identical subjects.

The fourth experiment was designed to verify the recognition contribution of morphological metrics, m2ROI-FN model was applied on different morphological metrics as input, and two m2ROI-FN models were created. Among them, (V13): m2ROI-FN model with no morphological metrics (the V9 in experiment 3 and 4 are completely identical)), and (V15): m2ROI-FN model with all morphological metrics.

The fifth experiment was designed to ascertain the influence of adjacent voxels in the patch. A Mask processing on ROI patches was conducted by setting the voxel intensity values within the patch area but out of the ROI to 0 based on Aparc + Aseg atlas, and the intensity values inside the ROI area remained unchanged. (V14) is the m2ROI-FN model trained with masked patch, (V15) is the m2ROI-FN model trained with no-masked patch (the V15 in experiment 4 and 5 are completely identical).

Finally, the m2ROI-FN model was compared with some relevant popular models, which included Liu's combination of ICA and Cox model (ICA-Cox),¹⁸ Suk's hierarchical feature representation and multi-modal fusion model (HFR-MF),¹⁹ Huang's multi-modality 3D convolutional model (B2),¹² Zhu's relational regularization feature selection model (RRF),²⁰ Hao's multi-modal neuroimaging feature selection with consistent metric constraint model (MMCC).²¹

Experiment

For the ADNI dataset, two experiments were designed to distinguish sMCI and pMCI respectively using the constructed model. The ADNI dataset consists of the relatively independent ADNI-1, ADNI-2, ADNI-3, and ADNI-GO datasets. In the first experiment, a complete ADNI dataset was used for model training, and 20% samples were randomly selected for model validation. In the second experiment, ADNI-1 and ADNI-2 were formed into an independent sample for model training, and ADNI-3 and ADNI-GO were formed into another independent sample for multi-center testing, in which 20% of the ADNI-1&2 dataset randomly selected for validation. The overall predictions were adopted to aggregate the cross-validation predictions for the entire dataset (ADNI and ADNI-1&2 columns in Table 5). The experiments ensured that each prediction was made for a sample that was not used in training, providing a reliable estimate of the model's generalization performance. Similarly, the aggregated ROC curve was plotted using the overall predictions from the cross-validation process (see Figures 2 and 3).

Figure 2.

Aggregated ROC curves of m2ROI-FN model with different inputs. V1-15 have completely identical definitions to Table 4.

Figure 3.

Without morphological metrics and patch masked operation, aggregated ROC curves of m2ROI-FN model and Cat CNN model.

Table 5.

Classification performance of m2ROI-FN for ablation experiments. ADNI column represents m2ROI-FN model trained on complete ADNI with 5-fold cross validation, ADNI-1&2 column represents model trained on ADNI-1&2 with 5-fold cross validation, ADNI-3&GO column represents model trained on ADNI-1&2 and tested on ADNI-3&GO. The values present in ADNI and ADNI-1&2 columns are overall predictions.

Model inputs	ADNI (5-fold Cross Validation)				ADNI-1&2 (5-fold Cross Validation)				ADNI-3&GO (Testing)
Model inputs	ACC	SEN	SPE	AUC	ACC	SEN	SPE	AUC	ACC	SEN	SPE	AUC
V1	71.7	80.5	56.8	70.9	69.8	77	60.3	70.3	67.8	75.5	37.6	58.3
V2	72.8	84.6	52.8	72.7	72.1	77	65.6	73.6	72.7	81.6	37.6	65.2
V3	69.4	81.9	48.3	66.5	68.4	80	53	66.9	67.5	76.7	31.2	60.7
V4	70.9	87.6	42.6	68.5	69.8	85	49.7	67	72	82.4	31.2	59.9
V5	70	83.9	46.6	67.9	65.5	86	38.4	64.6	72	84.5	23.2	54.8
V6	74.9	84.2	59.1	74.1	72.1	74	69 . 5	73.3	73	80	45.6	67.6
V7	74.1	85.6	54.5	73	68.4	76.5	57.6	71.4	71.5	79.6	40	61.8
V8	73.8	82.9	58.5	75	72.4	81.5	60.3	74.3	71.7	82.7	28.8	67.2
V9	70.3	85.6	44.3	67.6	67.8	67	68.9	69.1	64.6	69.6	44.8	63.7
V10	75.5	89.9	51.1	74.5	71.2	86	51.7	72.6	75.3	86.9	29.6	66
V11	72.4	85.2	50.6	70.3	71.2	73.5	68.2	71.1	70.1	78.6	36.8	64.8
V12	74.1	83.6	58	73.7	70.9	83.5	54.3	72.3	72.4	76.9	54.4	71.6
V13	73.2	83.9	55.1	73.6	72.4	78.5	64.2	73.1	71.2	80.6	34.4	65.6
V14	77.4	86.2	62.5	80.1	73.2	79.5	64.9	75	75	82.7	44.8	72.6
V15	74.1	85.9	54	76.2	70.1	77	60.9	72	73.2	81.8	39.2	67.3

The optimal result is represented in bold.

In order to tackle the sample imbalance problem, we adopted an oversample strategy to give more weight to the group with small sample size (e.g., pMCI). The oversample strategy was implemented with Pytorch Weighted Random Sampler toolkit. Our model was deployed using Python (version 3.8) with Pytorch (version 1.9.0) package, and the computing unit contained a single GPU (NVIDIA RTX 3090 24GB). ALL the trainable parameters were randomly initialized in standard normal distribution and optimized using standard back-propagation with stochastic gradient descent by minimizing the loss of cross entropy. The loss can be formulized as:

L_{c e} = \frac{1}{N} \sum_{i} L_{i} = - \frac{1}{N} \sum_{i} \sum_{M}^{c} y_{i c} \log (p_{i c})

Where

y_{i c}

is set to 1 if the prediction is equal to the diagnosis category, otherwise is set to 0.

p_{i c}

is probability that sample i belongs to class c. The Adadelta gradient descent algorithm (Adam)²² with a batchsize of 64 was used to train our m2ROI-FN. Integrated L2 regularization (weight decay coefficient 0.0001) in Adam was used to stabilize training. Model was trained for 60 epochs to ensure complete convergence (The loss curves of our model were shown in Figure 4), which took about 40 min (4 s for each epoch, 10 ROIs). Initial learning rate was set to 0.0001 and it is adjusted based on the ‘cosine’ decay learning rate adjustment method,²³ which can be formulized as:

η_{t} = \frac{1}{2} (1 + \cos (\frac{t π}{T})) η

Figure 4.

Loss curves of M2ROI-FN. The blue curve represents the training loss, the yellow curve represents the validation loss, the red curves represent the loss curves smoothed with a window size of 8, and the gray transparent area indicates the standard deviation range of the 5-fold cross-validation loss.

Where $η$ denotes initial learning rate, T denotes total number of epochs, t denotes the adjusting epoch points (20, 40 in this paper). Besides, L2 regulation were adopted in our network to avoid overfitting case.

Results

The classification performance of m2ROI-FN model under different ROI inputs has been calculated, and the outcomes are shown in Table 4. The aggregated ROC curves are shown in Figure 4. For the same ROI in both hemispheres of the brain, the m2ROI-FN model with amygdala as ROI input acquired the best ACC and AUC (5-fold cross validation achieved ACC 72.8% and AUC 72.7% on complete ADNI dataset, ACC 72.1% and AUC 73.6% on ADNI-1&2, multicenter testing achieved ACC 72.7% and AUC 65.2% on ADNI-3&GO, see V2 in Table 5). Compared with 4-ROIs, 6-ROIs, m2ROI-FN with 8-ROIs inputs achieved best AUC (5-fold cross validation achieved AUC 75% on complete ADNI dataset, AUC 74.3% on ADNI-1&2, multicenter testing achieved ACC 71.7% and AUC 67.2% on ADNI-3&GO, see V8 in Table 5).

Compared with right brain hemisphere ROIs, left brain hemisphere ROIs achieved better performance to differentiate sMCI from pMCI (5-fold cross validation achieved ACC 75.5%, SEN 89.9%, SPE 51.1% and AUC 74.5% on complete ADNI dataset, ACC 71.2%, SEN 86%, SPE 51.7% and AUC 72.6% on ADNI-1&2, multicenter testing achieved ACC 75.3%, SPE 86.9% and AUC 66% on ADNI-3&GO, see V10 in Table 5). In the single-modality recognition task, m2ROI-FN model achieved comparable accuracy by utilizing sMRI or PET for MCI recognition, with the latter achieving better performance (5-fold cross validation achieved better ACC 74.1%, SPE 58%, and AUC 73.7% on complete ADNI dataset, better SEN 83.5% and AUC 72.3% on ADNI-1&2 dataset, multicenter testing achieved better ACC 72.4%, SPE 54.4% and AUC 71.6% on ADNI-3&GO, see V12 in Table 5), and the multi-modal results achieved better MCI recognition performance than single-modality results (5-fold cross validation achieved better ACC 72.4% and AUC 73.1% on ADNI-1&2 dataset, multicenter testing achieved better SEN 80.6% on ADNI-3&GO, see V13 in Table 5). For the influences of patch masked operation, our model with patch-Masked achieved best ACC 77.4%, SPE 62.5%, AUC 80.14% on complete ADNI dataset, best ACC 73.2%, AUC 73.9% on ADNI-1&2 and best AUC 72.6% on ADNI-3&GO (see V14 in Table 5).

Table 6 denotes the performance comparison of our m2ROI-FN model with Cat CNN model, and the corresponding aggregated ROC curves are shown in Figure 3. For a fair comparison, both the m2ROI-FN model and the Cat-CNN model do not consider morphological metrics and patch masked operation, our model obtained more stable MCI recognition capabilities (5-fold cross validation achieved better SPE 55.1% and AUC 73.6% on complete ADNI dataset, better ACC 71.4%, SEN 88.2%, SPE 64.2% and AUC 68.9% on ADNI-1&2, multicenter testing achieved better ACC 71.2%, SEN 80.6% and AUC 65.6% on ADNI-3&GO).

Table 6.

Classification performance between our m2ROI-FN model and cat CNN model. The definitions of ADNI, ADNI-1&2, and ADNI-3&GO column are completely identical to those in Table 5. The values present in ADNI and ADNI-1&2 columns are overall predictions.

Model	ADNI (5-fold Cross Validation)				ADNI-1&2 (5-fold Cross Validation)				ADNI-3&GO (Testing)
Model	ACC	SEN	SPE	AUC	ACC	SEN	SPE	AUC	ACC	SEN	SPE	AUC
Cat-CNN	74.7	87.9	52.3	71.4	68.4	77	57	67.9	70.1	79.2	34.4	63.4
M2ROI-FN	73.2	83.9	55.1	73.6	72.4	78.5	64.2	73.1	71.2	80.6	34.4	65.6

The optimal result is represented in bold.

Though it is challenging to make a fair comparison between the proposed model with the reported models due to the different ways of using the ADNI dataset (i.e., the definition of sMCI/pMCI, different enrolled subjects, different splits of training/validation/testing sets). We roughly compared the sMCI versus pMCI recognition performance of the M2ROI-FN model with a series of models adopted sMRI and PET data. At this time, all models used 5-fold cross-validation on the complete ADNI dataset, as shown in Table 7. Although Suk et al.'s¹⁹ HFR-MF and Zhu et al.'s²⁰ RRF achieved better specificity for MCI recognition (RRF: SPE 94.6%, HFR-MF: SPE 95.2%), our m2ROI-FN achieved the best sensitivity (ACC 82.9%, SEN 80.3%, AUC 87.6%) with more sufficient sample size.

Table 7.

Classification performance among our m2ROI-FN and several reported models. The values present in ADNI and ADNI-2 columns are overall predictions.

Study	Subjects	Architecture	Modality	ADNI (5-fold Cross Validation)
Study	Subjects	Architecture	Modality	ACC	SEN	SPE	AUC
Liu et al.¹⁸	[108,126]	ICA-Cox	sMRI + PET	73.5	76.1	70.3	80.8
Suk et al.¹⁹	[128,76]	HFR-MF	sMRI + PET	75.9	48	95 . 2	74.6
Huang et al.¹²	[441,326]	B2	sMRI + PET	76.9	68.1	83.9	79.6
Zhu et al.²⁰	[56,43]	RRF	sMRI + PET	72.4	49.1	94.6	82.9
Hao et al.²¹	[56,43]	MMCC	sMRI + PET	77.7	67.4	85.5	76.0
Proposed	[346,180]	m2ROI-FN	sMRI + PET	77.4	86.2	62.5	80.1

The optimal result is represented in bold. The numbers within the brackets in the “Subjects” column represent the count of subjects with sMCI and pMCI.

Discussion

In this paper, an effective multi-modal and multi-stage ROI based fusion network (m2ROI-FN) CNN model to differentiate pMCI from sMCI was proposed by integrating deep semantic features and multiple morphological metrics from two modality neuroimaging of sMRI and PET. Our proposed fusion network adopted five dementia-related ROIs as the patch for model training including entorhinal cortex, amygdala, hippocampus, middle temporal gyrus, and insula. Different from whole-brain modeling, the ROI-based CNN modeling could effectively eliminate irrelevant brain regions and reduce redundancy information and computational effort. The specially designed MMI module could effectively incorporate the low-level features from each modality, and generate the high-level local features, while retaining the complementary information of the two modalities. Morphological features and deep semantic features are complementary in the recognition of pMCI and sMCI. The integration of the two types of features would further improve the classification performance. The multiple morphological features derived from sMRI images, mainly included cortical volume, voxel intensity mean, surface area, cortical thickness average and standard deviation. Finally, a ROI-based MLP classifier was created to fuse the high-level local features from different ROIs into global features for MCI recognition.

Our experiments indicated that adopting AD-related ROIs could improve the accuracy of MCI recognition. We compared the proposed m2ROI-FN CNN with the Cat CNN, which adopted the whole brain images for model training. Our m2ROI-FN achieved better recognition results than Cat CNN, as shown in Table 6. Choosing ROI over whole brain is similar to performing preliminary feature selection, reducing computational complexity and redundant features in unrelated brain regions, resulting in more accurate recognition results.

The feature fusion of the two modal images is helpful to improve the accuracy of MCI classification. We computed the recognition performance of our m2ROI-FN on sMRI, PET and sMRI + PET respectively. It was obvious that combining sMRI and PET has better specificity in distinguishing sMCI from pMCI, and better performance in distinguishing AD from NC, as shown in Table 5 (see V11, V12, and V13). To some extent, this indicates that the MMI module in our model could capture the respective features of sMRI and PET images well. Moreover, the classification performance of PET image is better than that of sMRI image on complete ADNI dataset and multicenter testing, which is consistent with previous research results (Tong et al.²⁴, Liu et al.¹³, Huang et al.¹²). The main reason for this may be that PET can reveal changes in brain function that predate morphological changes.

The integration of deep semantic features and morphological features is helpful to improve the accuracy of MCI classification. We employed the deep semantic features and the combination of deep semantic features and morphological features as inputs to calculate the recognition performance of the m2ROI-FN model, respectively. Among them, m2ROI-FN obtained relatively reliable classification results when combined with morphological feature input, as shown in Table 5 (V13 and V15). Morphological measures can improve the accuracy of MCI and AD recognition. The main reason is that MCI and AD both yield different degrees of morphological feature alterations, and morphological measures extracted based on radiomic analysis provide contributions to the recognition of MCI and AD that are different from deep semantic features.

Applying a mask operation to ROI Patches has a significantly effect on MCI classification. By masking the ROI based on brain atlas, redundant information can be further filtered out. After Masking ROI patches, MCI recognition performance is improved, as shown in Table 5 (V13 and V14). The main reasons may be twofold: First, Masking ROI patches enables the model to focus on ROI and exclude non-ROI regions that are irrelevant or disruptive to the diagnosis, thereby enhancing features that are highly correlated with MCI and AD diagnosis. Secondly, setting the voxel value of the non-ROI region to zero can reduce the computational complexity of the model to a certain extent and help to further improve the performance of the model.

Neurodegenerative diseases are usually caused by structural and functional changes in multiple regions of the brain. Capturing and integrating the changing features of multiple brain regions can improve the recognition accuracy of MCI. As shown in Table 5, when the m2ROI-FN model applied each single ROI as input (2-ROI), the recognition performance of the model was limited, among which the recognition performance of hippocampus and amygdala was superior to other ROIs. As the number of input ROIs increases, as shown in Table 5 (V6 to V8 and V13), the recognition performance of the model gradually improves. When all ROIs are selected as model inputs (V13, all-ROI), m2ROI-FN has the best specificity in classification of pMCI versus sMCI and the best performance in classification of AD versus NC. The experimental results show that the feature information of different ROI is complementary, and the integration of these features can improve the recognition performance. Furthermore, although the left and right cerebral hemispheres have similar anatomical structures, the m2ROI-FN model applying the ROI of the left cerebral hemisphere has a higher MCI classification performance than the ROI of the right cerebral hemisphere, as shown in Table 5 (V9 and V10), indicating asymmetric vulnerability of brain morphology and function in patients with MCI.

At present, how to accurately differentiate pMCI from sMCI remains a challenge. The existing main models for MCI recognition included ICA-COX (Liu et al.¹⁸), HFR-MF (Suk et al.¹⁹), B2 (Huang et al.¹²), RRF (Zhu et al.²⁰), MMC-CNN (Liu et al.¹³). Compared with these models, our m2ROI-FN model could achieve relatively good recognition performance in differentiating pMCI from sMCI. To be specific, The B2 model incorporated deep semantic features of multi-modal whole brain images through a simple MLP layer, while our m2ROI-FN model could extract more specific ROI-level features, and integrate local-to-global deep semantic features and morphological features. In MMC-CNN model, the whole brain was evenly divided into 3*3*3 partitions as patches, and a small convolutional layer was designed to fuse multi-modal information. However, our m2ROI-FN model adopted AD associated ROI patches and incorporates morphological features. In addition, our model was designed with a fully connected layer to integrate multi-modal information instead of a small convolutional layer, which has relatively better representation ability than a small convolutional layer. Compared to the HFR-MF and RRF models, although the m2ROI-FN model was relatively less specific in distinguishing between pMCI from sMCI, the HFR-MF and RRF models employed relatively few training samples. In contrast, the m2ROI-FN model utilized more training samples, which made it more robust and reliable.

Although our m2ROI-FN model relatively improves pMCI versus sMCI classification performance, it still has some limitations that need to be improved. Firstly, the ROI mask operation filtered out redundant information and improved the recognition accuracy of the M2ROI-FN. However, FreeSurfer may have a certain bias when segmenting the ROI mask in the central brain area, and inaccurate masks may lead to the loss of valuable features and the introduction of noise. Secondly, we only validated and tested the performance of our model using ADNI, and cross-dataset validation will be used in the future to further verify the robustness and generalization of our model. Thirdly, although multi-modal approaches provide more information, PET is a rare and expensive data modality, which limits the scalability of the method to some extent. Finally, due to data drift between different phases of ADNI, the performance of the M2ROI-FN model declined during testing. Expanding the data and applying data augmentation will be used to further mitigate the issue of data drift.

Conclusion

In this paper, we proposed a multi-modal and multi-stage ROI-based (m2ROI-FN) fusion network CNN model to differentiate MCI. Our model achieved accuracy of 77.4% to differentiate pMCI from sMCI with 5-fold cross validation on complete ADNI dataset, 73.2% on ADNI-1&2 and 75% by multicenter testing on ADNI-3&GO. The m2ROI-FN CNN model is capable of capturing distinctive features in ROIs of sMRI and PET images. Different ROI have different contribution to distinguish between pMCI and sMCI, among which hippocampus and amygdala have higher contribution weight in distinguishing between pMCI and sMCI. Mask operations and morphological metrics can effectively improve model recognition performance. By comparing the m2ROI-FN model with different parameter settings, the proposed model shows potential in the challenging task to differentiate pMCI from sMCI.

Supplemental Material

sj-docx-1-alz-10.1177_13872877241295287 - Supplemental material for A multi-modal and multi-stage region of interest-based fusion network convolutional neural network model to differentiate progressive mild cognitive impairment from stable mild cognitive impairment

Supplemental material, sj-docx-1-alz-10.1177_13872877241295287 for A multi-modal and multi-stage region of interest-based fusion network convolutional neural network model to differentiate progressive mild cognitive impairment from stable mild cognitive impairment by Zhenpeng Chen, Beier Qi, Bin Jing, Ruijuan Dong, Rong Chen, Pujie Feng, Yilu Shou and Haiyun Li in Journal of Alzheimer's Disease

Footnotes

Acknowledgements

The authors have no acknowledgments to report.

ORCID iDs

Zhenpeng Chen

Beier Qi

Ruijuan Dong

Yilu Shou

Haiyun Li

Author contributions

Zhenpeng Chen: Conceptualization; Methodology; Software; Writing – original draft.

Beier Qi: Funding acquisition; Writing – review & editing.

Bin Jing: Funding acquisition; Writing – review & editing.

Ruijuan Dong: Supervision; Writing – review & editing.

Rong Chen: Data curation; Supervision.

Pujie Feng: Software; Supervision.

Yilu Shou: Data curation; Supervision.

Haiyun Li: Conceptualization; Funding acquisition; Methodology; Writing – original draft; Writing – review & editing.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by Beijing Natural Science Foundation no. L192044; The open fund project of Beijing Key Laboratory of Fundamental Research on Biomechanics in 364 Clinical Application (2023KF05); and sponsored by Beijing Nova Program (20220484211).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability

The data used in this study were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (). The ADNI data are publicly available to researchers upon application and approval by the ADNI Data Sharing and Publications Committee. Detailed instructions for accessing the ADNI data can be found on the ADNI website.

Our code is available at: .

Supplemental material

Supplemental material for this article is available online.

References

Chen

, et al. A reparametrized CNN model to distinguish Alzheimer's disease applying multiple morphological metrics and deep semantic features from structural MRI. Front Aging Neurosci 2022; 14: 856391.

Breton

Casey

Arnaoutoglou

. Cognitive tests for the detection of mild cognitive impairment (MCI), the prodromal stage of dementia: meta-analysis of diagnostic accuracy studies. Int J Geriatr Psychiatry 2019; 34: 233–242.

Lian

Liu

Zhang

, et al. Hierarchical fully convolutional network for joint atrophy localization and Alzheimer's disease diagnosis using structural MRI. IEEE Trans Pattern Anal Mach Intell 2018; 42: 880–893.

Hojjati

Ebrahimzadeh

Khazaee

, et al. Predicting conversion from MCI to AD by integrating rs-fMRI and structural MRI. Comput Biol Med 2018; 102: 30–39.

Frisoni

Fox

Jack Jr

, et al. The clinical use of structural MRI in Alzheimer disease. Nat Rev Neurol 2010; 6: 67–77.

Gupta

Lama

Kwon

G-R

, et al. Prediction and classification of Alzheimer’s disease based on combined features from apolipoprotein-E genotype, cerebrospinal fluid, MR, and FDG-PET imaging biomarkers. Front Comput Neurosci 2019; 13: 72.

Abrol

Bhattarai

Fedorov

, et al. Deep residual learning for neuroimaging: an application to predict progression to Alzheimer’s disease. J Neurosci Methods 2020; 339: 108701.

Schmitter

Roche

Maréchal

, et al. An evaluation of volume-based morphometry for prediction of mild cognitive impairment and Alzheimer's disease. Neuroimage Clin 2015; 7: 7–17.

Jing

, et al. Identifying mild cognitive impairment with random forest by integrating multiple MRI morphological metrics. J Alzheimers Dis 2020; 73: 991–1002.

10.

Long

Huang

, et al. A comparative atlas-based recognition of mild cognitive impairment with voxel-based morphometry. Front Neurosci 2018; 12: 916.

11.

Payan

Montana

. Predicting Alzheimer's disease: a neuroimaging study with 3D convolutional neural networks. arXiv:1502.02506, 2015.

12.

Huang

Zhou

, et al. Diagnosis of Alzheimer’s disease via multi-modality 3D convolutional neural network. Front Neurosci 2019; 13: 509.

13.

Liu

Cheng

Wang

, et al. Multi-modality cascaded convolutional neural networks for Alzheimer’s disease diagnosis. Neuroinformatics 2018; 16: 295–308.

14.

Mueller

Weiner

Thal

, et al. The Alzheimer's disease neuroimaging initiative. Neuroimaging Clin 2005; 15: 869–877.

15.

Fischl

. Freesurfer. Neuroimage 2012; 62: 774–781.

16.

Destrieux

Fischl

Dale

, et al. Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. Neuroimage 2010; 53: 1–15.

17.

Wen

Thibeau-Sutre

Diaz-Melo

, et al. Convolutional neural networks for classification of Alzheimer's disease: overview and reproducible evaluation. Med Image Anal 2020; 63: 101694.

18.

Liu

Chen

Yao

, et al. Prediction of mild cognitive impairment conversion using a combination of independent component analysis and the cox model. Front Hum Neurosci 2017; 11: 33.

19.

Suk

H-I

Lee

S-W

Shen

, et al. Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. Neuroimage 2014; 101: 569–582.

20.

Zhu

Suk

H-I

Wang

, et al. A novel relational regularization feature selection method for joint regression and classification in AD diagnosis. Med Image Anal 2017; 38: 205–214.

21.

Hao

Bao

Guo

, et al. Multi-modal neuroimaging feature selection with consistent metric constraint for diagnosis of Alzheimer's disease. Med Image Anal 2020; 60: 101625.

22.

Zeiler

. Adadelta: an adaptive learning rate method. arXiv:1212.5701, 2012.

23.

Zhang

, et al. Bag of tricks for image classification with convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, USA, 2019, pp. 558–567. Piscataway, NJ: IEEE.

24.

Tong

Gray

Gao

, et al. Multi-modal classification of Alzheimer's disease using nonlinear graph fusion. Pattern Recognit 2017; 63: 171–181.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

2.84 MB

Model inputs	Hipp	Amy	Ins	MTG	EC	LH	RH	MRI	PET	Mask	Mor
V1	√					√	√	√	√
V2		√				√	√	√	√
V3			√			√	√	√	√
V4				√		√	√	√	√
V5					√	√	√	√	√
V6	√	√				√	√	√	√
V7	√	√	√			√	√	√	√
V8	√	√	√	√		√	√	√	√
V9	√	√	√	√	√		√	√	√
V10	√	√	√	√	√	√		√	√
V11	√	√	√	√	√	√	√	√
V12	√	√	√	√	√	√	√		√
V13	√	√	√	√	√	√	√	√	√
V14	√	√	√	√	√	√	√	√	√	√	√
V15	√	√	√	√	√	√	√	√	√		√

Model inputs	Hipp	Amy	Ins	MTG	EC	LH	RH	MRI	PET	Mask	Mor
V1	√					√	√	√	√
V2		√				√	√	√	√
V3			√			√	√	√	√
V4				√		√	√	√	√
V5					√	√	√	√	√
V6	√	√				√	√	√	√
V7	√	√	√			√	√	√	√
V8	√	√	√	√		√	√	√	√
V9	√	√	√	√	√		√	√	√
V10	√	√	√	√	√	√		√	√
V11	√	√	√	√	√	√	√	√
V12	√	√	√	√	√	√	√		√
V13	√	√	√	√	√	√	√	√	√
V14	√	√	√	√	√	√	√	√	√	√	√
V15	√	√	√	√	√	√	√	√	√		√

Model inputs	Hipp	Amy	Ins	MTG	EC	LH	RH	MRI	PET	Mask	Mor
V1	√					√	√	√	√
V2		√				√	√	√	√
V3			√			√	√	√	√
V4				√		√	√	√	√
V5					√	√	√	√	√
V6	√	√				√	√	√	√
V7	√	√	√			√	√	√	√
V8	√	√	√	√		√	√	√	√
V9	√	√	√	√	√		√	√	√
V10	√	√	√	√	√	√		√	√
V11	√	√	√	√	√	√	√	√
V12	√	√	√	√	√	√	√		√
V13	√	√	√	√	√	√	√	√	√
V14	√	√	√	√	√	√	√	√	√	√	√
V15	√	√	√	√	√	√	√	√	√		√