Temporal attention-aware evidential recurrent network for trustworthy prediction of Alzheimer’s disease progression

Abstract

Accurate and reliable prediction of Alzheimer’s disease (AD) progression is crucial for effective interventions and treatment to delay its onset. Recently, deep learning models for AD progression achieve excellent predictive accuracy. However, their predictions lack reliability due to the non-calibration defects, that affects their recognition and acceptance. To address this issue, this paper proposes a temporal attention-aware evidential recurrent network for trustworthy prediction of AD progression. Specifically, evidential recurrent network explicitly models uncertainty of the output and converts it into a reliability measure for trustworthy AD progression prediction. Furthermore, considering that the actual scenario of AD progression prediction frequently relies on historical longitudinal data, we introduce temporal attention into evidential recurrent network, which improves predictive performance. We demonstrate the proposed model on the TADPOLE dataset. For predictive performance, the proposed model achieves mAUC of 0.943 and BCA of 0.881, which is comparable to the SOTA model MinimalRNN. More importantly, the proposed model provides reliability measures of the predicted results through uncertainty estimation and the ECE of the method on the TADPOLE dataset is 0.101, which is much lower than the SOTA model at 0.147, indicating that the proposed model can provide important decision-making support for risk-sensitive prediction of AD progression.

Keywords

Alzheimer’s disease disease progression prediction evidential recurrent network temporal attention trustworthy deep learning

1. Introduction

Alzheimer’s Disease (AD) is a progressive and irreversible neurodegenerative disease leading to cognitive impairment and behavioral dysfunction. There are currently no drugs or clinical treatments available that can effectively reverse or cure the disease [1, 2]. However, current research indicated that AD treatment strategies should intervene during the early stages of the disease to delay its onset [3]. Therefore, predicting the progression of AD is clinically significant [4].

Recently, machine learning and deep learning methods have been successfully applied to practical scenarios for AD diagnosis and progression modeling. Some methods focused on predicting one single time point of AD diagnosis using cross-sectional data, such as support vector machine (SVM) [5], Gaussian processes [6], heterogeneous ensemble learning [7], and random forest [8]. These methods exhibited excellent performance for classifying multi-modal data at a single time point. However, the prediction of AD progression often relies on historical longitudinal data, in the actual scenarios. Therefore, recent studies shifted towards deep learning methods represented by recurrent neural networks (RNN) [9, 10, 11, 12, 13] to model the longitudinal data for AD.

However, existing work typically focused on improving predictive accuracy and used softmax activation function at the end of the networks to output the probability and the confidence, ignoring the flaw that the confidence does not match actual accuracy of the model. This leads the model to be in an uncalibrated state [14], which affects user recognition and acceptance. In medical diagnosis, a risk-sensitive scenario, not only high predictive accuracy is required, but also a reliability measure of the predicted results to avoid the threat of incorrect predictions [15, 16]. Figure 1a shows the confidence histogram of the MinimalRNN proposed by Nguyen et al. [11] on the TADPOLE dataset [2]. It can be observed from its distribution that there is a considerable discrepancy between the average confidence and the actual accuracy. This is further illustrated in the reliability diagram in Fig. 1b, which shows the gap between the predicted confidence and the well-calibrated one of each confidence bin. Furthermore, although existing AD progression prediction models employed methods such as ‘gated units’ or bidirectional RNNs [13] to process historical longitudinal data, they failed to distinguish the importance of historical data at different time points for predicting disease progression. This causes information from more important historical time points being neglected and thus affecting the model’s accuracy.

Figure 1.

Confidence histograms and reliability diagrams for MinimalRNN on TADPOLE dataset.

In order to address the problems mentioned above, this paper proposes temporal attention-aware evidential recurrent network for trustworthy prediction of AD progression. This model utilizes historical longitudinal data from AD at-risk individuals to provide reliable diagnoses of disease progress in the future.1

The code for this paper can be accessed via https://github.com/Mr-Talon/Trustworthy-AD.

Specifically, in order to construct a trustworthy prediction model of AD progression, this paper proposes evidential recurrent network, which introduces evidential deep learning [17] regularized by the AvUC loss [21] on the basis of FastGRNN [20], to provide reliable uncertainty estimation for the uncalibrated predictions. Firstly, evidential deep learning regards the model output as the evidence of subjective opinions and parameterizes the evidence as the parameters of the Dirichlet distribution. Then, uncertainty of the output results is modeled [24] based on the subjective logic theory [22] and the Dempster-Shafer (DS) theory of evidence [23]. Finally, the uncertainty is transformed into an output reliability measure. Additionally, in order to fully utilize the historical time points that are important to the task of disease progression prediction, this paper introduces temporal attention [18, 19] in the model, which adaptively weights the historical features of different time points to enhance the model’s predictive performance. Finally, experimental results on the TADPOLE dataset demonstrate that temporal attention-aware evidential recurrent network not only provides reliability measures of the predicted results through uncertainty estimation, but also achieves comparable predictive performance to the state-of-the-art model In general, the main contributions of this paper are as follows:

•

We propose evidential recurrent network, which introduces evidential deep learning regularized by the AvUC loss on the basis of FastGRNN, to address the defect of non-calibration for existing AD progression models. To the best of our knowledge, the proposed model is the first work estimating uncertainty in AD progression.

•

We integrate temporal attention into the model to adaptively weight the historical features of different time points, as AD progression prediction often relies on historical longitudinal data. This approach effectively improves the model’s predictive performance.

•

We conduct experiments on the TADPOLE dataset, which demonstrate that the proposed model not only achieves predictive performance comparable to the state-of-the-art models, but also provides reliability measures through uncertainty, which is not present in current AD progression models.

Table 1

Related works for Alzheimer’s disease progression

Author	Year	Target	Method	Dataset	Performance
Nguyen et al.	2018	AD progression for 6 years	LSTM $+$ MF	TADPOLE	mAUC of 0.925 BCA of 0.866
Ghazi et al.	2019	AD progression with MRI biomarkers	LSTM	TADPOLE	mAUC of 0759
Nguyen et al.	2020	AD progression for 6 years	MinimalRNN $+$ MF	TADPOLE	mAUC of 0.994 BCA of 0.887
Wei et al.	2021	AD progression for M24 to M60	LSTM $+$ attention	TADPOLE	mAUC of 0.897 BCA of 0.752 (M60)
Ghazi et al.	2022	AD biomarkers progression	CARRNN	ADNI	MAE of 0.286 MSE of 0.167
Ho et al.	2022	AD progression with MRI biomarkers	Bi-LSTM	TADPOLE	mAUC of 0782
Xu et al.	2022	AD biomarkers progression	Sequence learning	ADNI	MAE of 5.8689 (ADAS-Cog)

2. Related work

The prediction of the Alzheimer’s disease (AD) progression involves modeling longitudinal data to forecast future progression of the disease based on the historical diagnosis and AD biomarkers from AD at-risk individuals. To this end, Nguyen et al. [9] successively used LSTM which can address missing date with model-filled strategy and MinimalRNN [11] that can also perform model-filling with fewer parameters for longitudinal data modeling. Additionally, Ghazi et al. [10] improved the LSTM architecture and designed a corresponding loss function to address the problem of missing historical longitudinal data. Ho et al. [13] employed bidirectional LSTM and focal loss to enhance the performance of the AD progression prediction. Ghazi et al. [12] used integrated deep learning based on LSTM and continuous-time autoregressive modeling to further improve model performance (CARRNN). Xu et al. [32] proposed deep latent representation collaborated sequence learning strategy to flexibly handle the incomplete variable-length longitudinal multi-modal data of AD progression.

However, these models still fail to distinguish the relative importance of historical data at different time points for disease progression prediction, leading to information from more important historical time points being ignored. Attention [18, 19] is used to address this issue by adaptively assigning weights to data at different time point. By introducing self-Attention [19], the RNN model assigns weights to the historical features of all previous time points instead of concentrating solely on the last hidden state when updating the hidden state, which effectively improves the performance in processing longitudinal data. Recently, a lot of work brought attention into the field of disease progression prediction and achieved excellent results. Tan et al. [33] designed a novel unreliability-aware attention to handle irregular multivariate time series physiological data. Lu et al. [34] introduced attention into the LSTM to perform multi-disease prediction for intelligent clinical decision support. Luiz et al. [35] proposed a model based on Bidirectional Gated Recurrent Units along with attention to predict Parkinson’s Disease using tremor signals collected from handwriting exams. Liang et al. [36] introduced attention and smoothness regularization to handle AD progression prediction. More information about approaches for AD progression is shown in Table 1, where $+$ MF means model with model filling strategy, M24 and M60 are particular time points of AD progression, ADAS-Cog is one of the biomarkers of AD.

Nevertheless, they overlook the fact that the output confidence of the model lacks calibration owing to the wide use of softmax activation function in the classification stage, which affects user recognition and acceptance of such models in risk-sensitive scenarios. Evidential Deep Learning (EDL) [17] recently gained widespread use in medical imaging processing due to its ability to provide an effective uncertainty estimation for uncalibrated predictive confidence and to convert this uncertainty into a reliable measure of the output result. Ghesu et al. [25] employed EDL for binary classification tasks in medical image diagnosis. Specifically, they used Beta distribution, a special case of the Dirichlet distribution to model the uncertainty of output. Li et al. [26] utilized EDL for brain tumor image segmentation, quantifying segmentation uncertainty to boost trustworthiness in the model. Zou et al. [27] applied EDL to a wider range of medical image segmentation fields, including 2D, 3D, and multimodal segmentation tasks, and achieved remarkable results.

3. Preliminary knowledge

3.1 Evidential deep learning (EDL)

The existing deep classification models typically use softmax activation function to obtain predictive confidence. However, due to the lack of calibration, the confidence cannot accurately reflect the reliability of model’s predictions, thus restricting the application of such models in risk-sensitive scenarios. To address this issue, Evidential Deep Learning (EDL) employs ReLU activation function to model the output of neural networks as evidence in subjective opinions, and parameterizes the evidence into parameters of Dirichlet distribution. EDL provides reliable uncertainty estimation for uncalibrated confidence based on the subjective logic theory and DS evidence theory. This enables the model to characterize the reliability measure via the uncertainty of the predicted results while outputting the probability of the category. Specifically, the Dirichlet distribution models the probability distribution of the parameter of category distribution, and its probability density function is shown in Eq. (1):

$\displaystyle D(\bm{p}|\bm{\alpha})=\left\{\begin{array}[]{ll}\frac{1}{B(\bm{% \alpha})}\mathop{\prod}\limits_{k=1}^{K}p_{k}^{\alpha_{k}-1},&\text{for }\bm{p% }\in S_{K},\\ 0,&\text{otherwise}\\ \end{array}\right.$ (1)

where $\bm{\alpha}$ is the parameters of the Dirichlet distribution, $\alpha_{k}$ corresponding to the $k^{\text{th}}$ category in $\bm{\alpha}$ , $S_{K}=\left\{{\bm{p}\left|\mathop{\sum}\limits_{k=1}^{K}\right.p_{k}=1\text{ % and }0\leqslant p_{k}\leqslant 1,\forall k}\right\}$ is the simplex form of the $K$ -dimensional category distribution parameter $\bm{p}$ , $p_{k}$ is the category probability of the $k^{\text{th}}$ category, $B(\cdot)$ is the beta function. According to the subjective logic theory and DS evidence theory, the uncertainty of the predicted results and belief scores of each category are shown in Eq. (2):

$\displaystyle u=\frac{K}{S}\text{ and }b_{k}=\frac{e_{k}}{S}=\frac{\alpha_{k}-% 1}{S},$ (2)

where $S=\mathop{\sum}\limits_{k=1}^{K}\alpha_{k}$ is Dirichlet strength, $e_{k}=\alpha_{k}-1$ is the evidence of the subjective opinion of the neural network on the output of $k^{\text{th}}$ category. The uncertainty $u$ and the belief score of each category $b_{k}$ satisfy the following Equation:

$\displaystyle u+\mathop{\sum}\limits_{k=1}^{K}b_{k}=1.$ (3)

The category probability of the prediction can be obtained by calculating the mean of the Dirichlet distribution, as shown in Eq. (4):

$\displaystyle\hat{p}_{k}=\frac{\alpha_{k}}{S},$ (4)

where $\hat{p}_{k}$ is the $k^{\text{th}}$ output probability of the category probability $\hat{\bm{p}}$ . The category probability $\hat{\bm{p}}$ can be converted into one-hot vector $\hat{\bm{c}}$ representing the categorical predicted result.

Evidence of subjective opinion is learned by optimizing the MSE Bayesian risk loss for modeling the Dirichlet distribution, as shown in Eq. (5):

$\displaystyle L_{\textit{MSE}}({\bm{c}_{i},\bm{\alpha}_{i}})=\mathop{\int}% \limits||\bm{c}_{i}-\bm{p}_{i}||_{2}^{2}\frac{1}{B({\bm{\alpha}_{i}})}\mathop{% \prod}\limits_{k=1}^{K}p_{ik}^{\alpha_{ik}-1}d\bm{p}_{i},$ (5)

where $||\cdot||_{2}^{2}$ is the MSE operator, the subscript $i$ denotes the $i^{\text{th}}$ sample, $\bm{c}_{i}$ represents its label. In order to reduce the evidence of incorrect class (even close to 0), the KL divergence loss is introduced, as shown in Eq. (6):

$\displaystyle L_{KL}=KL[D(\bm{p}_{i}|\tilde{\bm{\alpha}}_{i})||D(\bm{p}_{i}|% \bm{1})],$ (6)

where $\tilde{\bm{\alpha}}_{i}=\bm{y}_{i}+(\bm{1}-\bm{y}_{i})\odot\bm{\alpha}_{i}$ is the parameters of the Dirichlet distribution after removal of the non-misleading evidence from predicted parameters $\bm{\alpha}_{i}$ for sample $i$ , which should be equal to the Dirichlet distribution with all parameters of 1. That is, a uniform distribution that does not provide any evidence, $D({\bm{p}_{i}{|}\langle 1,\cdots,1\rangle})$ , where $\bm{1}$ represents the parameter vector of $K$ ones. After introducing the KL divergence loss, the MSE Bayesian risk loss specific to sample $i$ is adjusted to:

$\displaystyle L_{\textit{AMSE}}({\bm{c}_{i},\bm{\alpha}_{i}})=L_{\textit{MSE}}% ({\bm{c}_{i},\bm{\alpha}_{i}})+\lambda_{e}L_{KL},$ (7)

Figure 2.

Block diagrams for FastRNN.

where $\lambda_{e}=\min\left\{{1,\frac{\textit{epoch}}{10}}\right\}$ is the annealing factor, epoch is the current number of training epochs, which is used to prevent the model from converging to a uniform distribution in the early stages of training.

3.2 FastGRNN

FastGRNN [20] is a RNN model using weight sharing and residual connection to mitigate issues such as gradient explosion, gradient vanishing, and information forgetting. With a small number of parameters, it is very suitable for modeling medical small-sample data. In fact, FastGRNN demonstrates superior performance over traditional gated RNN models such as LSTM and GRU [28] in various applications. The model architecture is depicted in Fig. 2, and the update of the hidden state is illustrated in Eqs (8)–(10):

$\displaystyle\bm{z}_{t}=\sigma({\bm{Wx}_{t}+\bm{Uh}_{t-1}+\bm{b}_{z}}),$ (8) $\displaystyle\tilde{\bm{h}}_{t}=\tanh(\bm{Wx}_{t}+\bm{Uh}_{t-1}+\bm{b}_{h}),$ (9) $\displaystyle\bm{h}_{t}=({\zeta({\bm{1}-\bm{z}_{t}})+\nu})\odot\tilde{\bm{h}}_% {t}+\bm{z}_{t}\odot\bm{h}_{t-1},$ (10)

where $\bm{x}_{t}$ is the input of FastGRNN for time $t$ , $\bm{h}_{t}$ is the hidden state at time $t$ , $\bm{W}$ and $\bm{U}$ are transformation matrix, $\bm{b}_{z}$ and $\bm{b}_{h}$ are the bias terms of the updating gate $\bm{z}_{t}$ , $\tilde{\bm{h}}_{t}$ is the candidate hidden state at time $t$ , $\sigma$ is the sigmoid activation function, $\tanh$ is hyperbolic tangent activation function, $\odot$ is Hadamard product, $\zeta$ and $\nu$ are transformed from the learnable parameter $a$ and $b$ , which can be calculated by $\zeta={\sigma}({a\times\zeta_{\textit{init}}})$ and $\nu={\sigma}({b\times\nu_{\textit{init}}})$ , where $\zeta_{\textit{init}}$ and $\nu_{\textit{init}}$ are the initial values of $\zeta$ and $\nu$ .

4. Methods

4.1 Problem setups

For the prediction of AD progression, the problem considered in this paper is to use the historical diagnostic results and multimodal AD biomarkers from AD at-risk individuals to anticipate future diagnoses of the disease within a certain period of time, while providing the uncertainty of the predicted outcome as a measure of its reliability.

4.2 Proposed model

4.2.1 Architecture of the overall model

To address the above-mentioned problem, this paper proposes temporal attention-aware evidential recurrent network. The model consists of three modules: the model filling module, the feature extraction module and the prediction output module, as shown in Fig. 3. $[{\bm{c}_{i-1},\bm{s}_{i-1}}]$ represents all information for a given sample at time point $i-1$ , comprising of the diagnosis $\bm{c}_{i-1}$ and AD biomarkers $\bm{s}_{i-1}$ . Since there may be instance where data is partially or entirely missing at certain time points, the model filling module is used to adaptively complete missing data by filling in missing parts with the model’s output from last time point. The operation of model filling in the training and testing phases is shown on the right side of Fig. 3. In feature extraction module, we introduce temporal attention (see Section 4.2.3) into evidential recurrent network, which utilizes complete data from the model filling module and extracts temporal features based on prior time points. The prediction output module is composed of a classification evidence-aware subnetwork and a regression subnetwork. The classification evidence-aware subnetwork converts temporal features into evidence of subjective opinions via multi-layer perception (MLP) and then parameterizes the evidence into Dirichlet distribution parameters and calculates the diagnosis $\hat{\bm{c}}_{i}$ and uncertainty $u_{i}$ for the disease progression prediction (see Section 4.2.2). The regression subnetwork uses MLP to convert temporal features into AD biomarkers $\bm{s}_{i}$ , which is in turn used to fill in missing data. It is worth noting that the MLP in classification evidenceaware subnetwork and regression subnetwork don’t share parameters.

Figure 3.

Architecture of the proposed temporal attention-aware evidential recurrent network.

4.2.2 Classification evidence-aware subnetwork

We propose evidential recurrent network, which introduces evidential deep learning [17] regularized by the AvUC loss [21] on the basis of FastGRNN [20]. In its classification evidence-aware subnetwork, the temporal features $\bm{x}$ extracted from the feature extraction module at each time point are processed by MLP. Rather than using softmax activation function, the ReLU activation function is used to map the output of the MLP to the evidence of subjective opinions for classification. The evidence is then parameterized into Dirichlet distribution parameters $\bm{\alpha}$ . The class probability $\hat{\bm{p}}$ and uncertainty $u$ are calculated based on the Dirichlet distribution parameters, as shown in Fig. 4.

Figure 4.

Block diagrams for evidential recurrent network and classification evidence-aware subnetwork.

In order to further calibrate the uncertainty and make it more reliable, i.e., produces smaller uncertainty for correctly classified samples and larger uncertainty for misclassified ones, the AvUC loss [21] is introduced in this paper as shown in Eq. (11):

$\displaystyle L_{\textit{AvUC}}=-\tau\mathop{\sum}\limits_{i\in\{\hat{\bm{c}}_% {i}=\bm{c}_{i}\}}p_{i}\log(1-u_{i})-(1-\tau)\mathop{\sum}\limits_{i\in\{\hat{% \bm{c}}_{i}\neq\bm{c}_{i}\}}(1-p_{i})\log(u_{i}),$ (11)

where $\tau=\tau_{0}e^{-\textit{epoch}/T\ln({\tau_{0}})}$ is the annealing factor, $\tau_{0}$ is the hyperparameter that controls the time for the two components of the equation to reach equilibrium, epoch is the current number of training rounds, $T$ is the total number of training rounds, $p_{i}=\max\{{\alpha_{ik}/S_{i}}\}$ is the original confidence of the prediction. The loss function for the prediction output module with the introduction of the AvUC loss is shown in Eq. (12):

$\displaystyle L_{\textit{EDL}}(\bm{c},\bm{\alpha})=\mathop{\sum}\limits_{i=1}^% {N}L_{\textit{AMSE}}(\bm{c}_{i},\bm{\alpha}_{i})+L_{\textit{AvUC}}=\mathop{% \sum}\limits_{i=1}^{N}L_{\textit{MSE}}(\bm{c}_{i},\bm{\alpha}_{i})+\lambda_{e}% L_{KL}+L_{\textit{AvUC}},$ (12)

where $L_{\textit{MSE}}({\bm{c}_{i},\bm{\alpha}_{i}})$ is defined as:

$\displaystyle L_{\textit{MSE}}(\bm{c}_{i},\bm{\alpha}_{i})=\mathop{\sum}% \limits_{k=1}^{K}\underbrace{\left(c_{ik}-\frac{\alpha_{ik}}{S_{i}}\right)^{2}% }_{L_{ik}^{\textit{error}}}+\underbrace{\frac{\alpha_{ik}(S_{i}-\alpha_{ik})}{% S_{i}^{2}(S_{i}+1)}}_{L_{ik}^{\textit{var}}},$ (13)

the first term of this loss function is designed to guarantee the accuracy of classification, whereas the second term aims to reduce the output variance. $L_{KL}$ is defined as:

$\displaystyle L_{KL}=\log\left({\frac{{\Gamma}\left({\mathop{\sum}\limits_{k=1% }^{K}\tilde{\alpha}_{ik}}\right)}{\Gamma(K)\mathop{\prod}\limits_{k=1}^{K}% \Gamma(\tilde{\alpha}_{ik})}}\right)+\mathop{\sum}\limits_{k=1}^{K}(\tilde{% \alpha}_{ik}-1)\left[\psi(\tilde{\alpha}_{ik})-\psi\left(\mathop{\sum}\limits_% {k=1}^{K}\tilde{\alpha}_{ik}\right)\right],$ (14)

where ${\Gamma}(\cdot)$ is the gamma function, $\psi(\cdot)$ is the digamma function. By optimizing the loss function shown in Eq. (12), the model can estimate the uncertainty, which represents the model’s reliability, while predicting the class probability that characterizes the model’s predicted results. Moreover, the calibration of the model’s predicted uncertainty can be significantly improved with the AvUC loss.

4.2.3 Feature extraction module based on temporal attention

To fully utilize historical time points that are crucial for disease progression prediction, this paper introduces the temporal attention into evidential recurrent network to extract temporal features from the complete data after model filling. Temporal attention is illustrated in Fig. 5.

Figure 5.

Block diagrams for temporal attention.

The evidential recurrent network with the introduction of temporal attention updates its hidden state with a focus on the crucial time points for the task among all previous time points’ hidden states. This is achieved by adaptively weighting the historical features of different time points to differentiate the importance of hidden states. In the process of updating the hidden state, context information $\bm{cx}_{t-1}$ is constructed using all the previous hidden states $({\bm{h}_{1}\ldots\bm{h}_{t-1}})$ that had been observed before time $t-1$ . The update of hidden state ( $\bm{h}_{t}$ ) depends on both $\bm{cx}_{t-1}$ and input $\bm{x}_{t}$ at time $t$ . Specifically, $\bm{cx}_{t-1}$ is calculated from the formula of the self-attention, where the query is the current time $({t-1})$ hidden state $\bm{h}_{t-1}$ , the key and the value are both combination of previous hidden states $({\bm{h}_{1}\ldots\bm{h}_{t-1}})$ . The context information $\bm{cx}$ is computed by taking a weighted sum of the value with the attention weights obtained by the query and key. The update of the hidden state is shown in Eqs (15) to (22):

$\displaystyle\bm{q}_{t-1}=\bm{h}_{t-1},$ (15) $\displaystyle\bm{k}_{t-1}=[\bm{h}_{1},\ldots,\bm{h}_{t-1}],$ (16) $\displaystyle\bm{\nu}_{t-1}=[\bm{h}_{1},\ldots,\bm{h}_{t-1}],$ (17) $\displaystyle\bm{\alpha}_{t-1}=\theta({\bm{w}_{v}^{T}\tanh({\bm{W}_{q}\bm{q}_{% t-1}+\bm{W}_{k}\bm{k}_{t-1}})}),$ (18) $\displaystyle\bm{cx}_{t-1}=\mathop{\sum}\limits_{i=1}^{t-1}\alpha_{t-1}^{i}\bm% {\nu}_{t-1}^{i},$ (19) $\displaystyle\bm{z}_{t}=\sigma({\bm{Wx}_{t}+\bm{Ucx}_{t-1}+\bm{b}_{z}}),$ (20) $\displaystyle\tilde{\bm{h}}_{t}=\tanh(\bm{Wx}_{t}+\bm{Ucx}_{t-1}+\bm{b}_{h}),$ (21) $\displaystyle\bm{h}_{t}=({\zeta({\bm{1}-\bm{z}_{t}})+\nu})\odot\tilde{\bm{h}}_% {t}+\bm{z}_{t}\odot\bm{cx}_{t-1},$ (22)

where $\bm{q}_{t}$ , $\bm{k}_{t}$ , $\bm{v}_{t}$ represent the query, key, and value at time $t$ , $\bm{\alpha}_{t}$ represents the attention weights vector at time $t$ , $\theta(\cdot)$ represents softmax activation function, which maps the attention scores into attention weights, $\bm{W}_{q}$ , $\bm{W}_{k}$ , $\bm{w}_{v}$ are parameters to be learned for the temporal attention, $\bm{cx}_{t}$ is context information at time $t$ .

4.2.4 Overall loss function

In this paper, a composite loss function is designed to train the temporal attention-aware evidential recurrent network. The loss function of the model is shown in Eq. (23):

$\displaystyle L=\mathop{\sum}\limits_{t>1}\lambda L_{\textit{EDL}}(\bm{c}_{t},% \bm{\alpha}_{t})+L_{\textit{MAE}}(\bm{s}_{t},\bm{\hat{s}}_{t}),$ (23)

where $\hat{\bm{s}}_{t}$ is the AD biomarkers output for each time point which is used to handle possible missing data at next time. $L_{\textit{EDL}}({\bm{c}_{t},\bm{\alpha}_{t}})$ represents the EDL loss of all training samples at time $t$ corresponding to Eq. (12). $L_{\textit{MAE}}(\bm{s}_{t},\hat{\bm{s}}_{t})=\mathop{\sum}\limits_{i=1}^{N}1/% M||\bm{s}_{ti}-\hat{\bm{s}}_{ti}||_{1}$ is loss function of AD biomarkers, where $M$ is the dimension of the AD biomarkers, $\bm{s}_{ti}$ is the label of AD biomarkers for sample $i$ at time $t$ , $\lambda$ is the weight of the EDL loss. It is important to note that the loss function only calculates gradients for the part of the data that with no missing. By optimizing the loss function shown in Eq. (23), the model can successfully complete the task of trustworthy prediction of AD progression.

5. Experience

5.1 Dataset

In this paper, we validate the effectiveness of the proposed method using the TADPOLE dataset provided by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) [29]. The dataset consists of 12,741 visit records from 1,737 AD at-risk individuals recorded between 2003 and 2007. Disease status for each individual include AD, mild cognitive impairment (MCI), and cognitively normal (CN). The age of the individuals in this dataset ranges from 54.40 to 91.40 AD biomarkers measurements and diagnostic results are recorded at approximately 6-month intervals starting from baseline time. The sampling times, total sampling durations and time intervals for each individual are different. The average number of samples per individual is 7.3 $\pm$ 4.0, and the average total sampling duration is 3.6 $\pm$ 2.5 years. More demographic information on the dataset is shown in Table 2.

To facilitate comparison, this paper follows the approach of Nguyen et al. and selects 22 feature variables from TADPOLE dataset as the measure of AD biomarkers. These feature variables include neurocognitive test scores, magnetic resonance imaging features, positron emission tomography features and CSF markers and so on. The DX variable is used as the diagnostic result. For the disease diagnosis data, we merge the state of conversion, such as MCI-AD to the state after conversion [10]. For AD biomarkers, the brain organ volume features are normalized by the total intracranial volume (ICV) to compensate for the inter-subject variability in brain size [10]. And then, Z standardization is applied to all AD biomarkers. Additionally, due to the irreversible nature of AD, erroneous data that transitioning from AD to MCI or CN are removed. Moreover, to address variance time intervals of measurements for AD at-risk individuals, the time interval of the proposed model is set to the finest granularity measurement interval. Finally, if data missing occurs at the first time point of the longitudinal data, the mean value of all subsequent times will be used to complete, so that the model filling strategy can work properly [11]. More information about the features and their missing cases in the dataset are shown in Table 3.

Table 2
Demographics of the TADPOLE dataset

Status	Number of males	Number of females	Age of males	Age of females
AD	189	153	75.56 $\pm$ 7.11	74.52 $\pm$ 7.81
MCI	515	357	73.71 $\pm$ 7.11	71.80 $\pm$ 7.85
CN	253	270	75.10 $\pm$ 5.64	74.42 $\pm$ 5.38
Total	1737		73.81 $\pm$ 6.97

Table 3

Feature information and missing cases

Features	Mean ( $\pm$ std)	Missing number	Exist rate
CDRSB	2.17 $\pm$ 2.81 $\times$ 10⁰	3778	70.35%
ADAS11	1.14 $\pm$ 0.86 $\times$ 10¹	3831	69.93%
ADAS13	1.75 $\pm$ 1.17 $\times$ 10¹	3917	69.26%
MMSE	2.66 $\pm$ 0.39 $\times$ 10¹	3809	70.10%
Immediate	3.45 $\pm$ 1.36 $\times$ 10¹	3910	69.31%
Learning	4.02 $\pm$ 2.81 $\times$ 10⁰	3910	69.31%
Forgetting	4.23 $\pm$ 2.53 $\times$ 10⁰	3936	69.11%
Perc_forgetting	5.97 $\pm$ 3.84 $\times$ 10¹	4006	68.56%
MOCA	2.31 $\pm$ 0.47 $\times$ 10¹	7774	38.98%
FAQ	5.59 $\pm$ 7.92 $\times$ 10⁰	3747	70.59%
Entorhinal	3.44 $\pm$ 0.81 $\times$ 10³	6272	50.77%
Fusiform	1.71 $\pm$ 0.28 $\times$ 10⁴	6272	50.77%
Hippocampus	6.68 $\pm$ 1.24 $\times$ 10³	5939	53.39%
ICV	1.53 $\pm$ 0.17 $\times$ 10⁶	4788	62.42%
MidTemp	1.92 $\pm$ 0.31 $\times$ 10⁴	6272	50.77%
Ventricles	4.21 $\pm$ 2.33 $\times$ 10⁴	5296	58.43%
WholeBrain	1.01 $\pm$ 0.11 $\times$ 10⁶	5052	60.35%
AV45	1.19 $\pm$ 0.23 $\times$ 10⁰	10623	16.62%
FDG	1.21 $\pm$ 0.16 $\times$ 10⁰	9389	26.31%
CSF	1.02 $\pm$ 0.60 $\times$ 10³	10370	18.61%
TAU	2.93 $\pm$ 1.31 $\times$ 10²	10370	18.61%
PTAU	2.81 $\pm$ 1.44 $\times$ 10¹	10371	18.60%
DX	–	3849	69.79%

5.2 Evaluation metrics

The multiclass AUC (mAUC) is a performance evaluation metric for multiclass classification. In this experiment, the mAUC is the average of three binary AUCs, which are AD vs. non-AD, MCI vs. non-MCI, and CN vs. non-CN. The higher the mAUC, the better the classification performance. The calculation of mAUC is shown in Eq. (24):

$\displaystyle\textit{mAUC}=\frac{2}{K({K-1})}\mathop{\sum}\limits_{i=2}^{K}% \mathop{\sum}\limits_{j=1}^{i}\hat{A}(c_{i},c_{j}),$ (24)

where $K$ is the number of categories, $\hat{A}(c_{i},c_{j})$ is the average AUC of any two categories, which can be calculated by the following formula:

$\displaystyle\hat{A}(c_{i},c_{j})=\frac{\hat{A}(c_{i}|c_{j})+\hat{A}(c_{j}|c_{% i})}{2},$ (25)

where $\hat{A}(c_{i}|c_{j})$ is the AUC of category $i$ relative to category $j$ , which can be calculated by the following formula:

$\displaystyle\hat{A}(c_{i}|c_{j})=\frac{S_{i}-n_{i}(n_{i}+1)/2}{n_{i}n_{j}},$ (26)

where $n_{i}$ represents the number of samples that belong to class $i$ , $S_{i}$ is the sum of the rank of each sample in class $i$ when the samples are sorted in ascending order based on their predicted probability.

Balanced Classification Accuracy (BCA) is another performance evaluation metric for multiclass classification. The higher the BCA value, the better the classification performance. The calculation of BCA can be represented by Eq. (27):

$\displaystyle\textit{BCA}=\frac{1}{K}\sum_{k=1}^{K}\textit{BCA}_{k},$ (27)

where $\textit{BCA}_{k}$ can be calculated by the following formula:

$\displaystyle\textit{BCA}_{k}=\frac{1}{2}\left[\frac{TP}{TP+FN}+\frac{TN}{TN+% FP}\right],$ (28)

were $T P$ , $T N$ , $F P$ , $F N$ correspond to the number of true positive sample, true negative sample, false positive sample, and false negative sample.

The accuracy versus uncertainty (AvU) utility function [21] is used to assess uncertainty calibration performance. This function divides the predicted outcomes into four groups based on their correctness and the corresponding uncertainty of each sample. The expression for the utility function is shown in Eq. (29):

$\displaystyle\textit{AvU}=\frac{N_{AC}+N_{IU}}{N_{AC}+N_{AU}+N_{IC}+N_{IU}},$ (29)

where $N_{AC}$ represents the number of samples that are both correct and certain, $N_{AU}$ for that are both correct and uncertain, $N_{IC}$ for that are both incorrect and certain, and $N_{IU}$ for that are both incorrect and uncertain. For well-calibrated uncertainty estimation, the AvU value should be higher when there is a larger proportion of $N_{AC}$ and $N_{IU}$ , signifying a superior uncertainty calibration performance of the model.

The expected calibration error (ECE) [30] quantifies the gap between model output confidence and actual accuracy, reflecting the calibration performance of the model [14]. The expression for ECE is shown in Eq. (30):

$\displaystyle\textit{ECE}=\mathop{\sum}\limits_{m=1}^{M}\frac{\left|{B_{m}}% \right|}{n}\left|{\textit{acc}({B_{m}})-\textit{conf}({B_{m}})}\right|,$ (30)

where $M$ is the number of equally spaced intervals from 0 to 1 based on sample confidence, $|{B_{m}}|$ represents the number of samples in each confidence bin $B_{m}$ . In conventional classification models, confidence is defined as the maximum predicted class probability of the model output. However, in this paper, confidence is $\textit{conf}_{i}=1-u_{i}$ , which is defined as reliability. $n$ is the total number of samples, $\textit{acc}({B_{m}})$ represents the accuracy of samples falling in a confidence bin $B_{m}$ , $\textit{conf}({B_{m}})$ stands for the average confidence of samples in a confidence bin $B_{m}$ . A lower value of ECE loss indicates better calibration performance of the model.

5.3 Experimental setups

In this experiment, the proposed method is used to predict future disease progression for AD at-risk individuals based on historical diagnosis and AD biomarkers. Firstly, the performance of the model for disease progression prediction is compared with that of MinimalRNN [11], GRU [28], LSTM [9], LSS [9] and SVM&SVR [9]. This is detailed in Section 5.5.1. Secondly, in order to verify whether the model can effectively model the predicted uncertainty, the relationship between the uncertainty and predicted accuracy of the proposed method is compared with the relationship between the confidence and predicted accuracy of traditional methods, as detailed in Section 5.5.2. Additionally, the effect of the uncertainty threshold on the model’s predictive performance is studied to determine whether the uncertainty estimation can support reliable decision-making, as detailed in Section 5.5.3. Finally, two sets of ablation experiments are conducted. First of all, the effect of the AvUC loss on the uncertainty calibration performance and model calibration performance is validated, as detailed in Sections 5.6.1 and 5.6.2 respectively. Moreover, the effect of the temporal attention on the performance of model’s disease progression prediction is validated, as detailed in Section 5.6.3.

Table 4
Hyperparameter search space of the proposed method

Hyper-parameter	Range
Learning rate	{1 $e^{-2}$ , 5 $e^{-3}$ , 1 $e^{-3}$ , 5 $e^{-4}$ , 1 $e^{-4}$ }
Batch size	{64, 128}
Size of RNN hidden units	{64, 128, 256}
Number of RNN hidden layer	{1, 2}
Size of MLP hidden layer units	{32, 64, 128}
RNN input dropout	{0.1, 0.3, 0.5}
RNN hidden layer dropout	{0.2, 0.4, 0.5}
Attention dropout	{0.3, 0.5}
Weight decay	{1 $e^{-4}$ , 1 $e^{-5}$ , 1 $e^{-6}$ , 5 $e^{-7}$ }
Loss function weight $\lambda$	{1, 2, 3}

In accordance with the method described by VAROQUAUX [31], the dataset for all of the experiments is randomly divided into 20 subsets. The ratio of subjects in the training, validation and test sets is 18:1:1. For each experiment, the training set is used to train the model, the validation set is used to tune hyperparameters and select the optimal model, and the test set is used to evaluate the performance of the model. The random split of the data into training, validation and test sets is repeated 20 times to ensure stability of results. The final result is calculated by averaging the performance evaluation metrics over the 20 experiments. To ensure comparability with Nguyen et al. [11], the data used for prediction is divided into two parts. The first part serves as the input to the model, while the second part is used to assess its performance by comparing it with the model output, as shown in Fig. 3.

The hyperparameter search space is shown in Table 4. After hyperparameter tuning, we select the optimal set of hyperparameters, with a learning rate of 0.001, batch size of 64, 256 RNN hidden units, 1 RNN hidden layer, 64 MLP hidden layer units. We set RNN input dropout to 0.1, RNN hidden layer dropout to 0.2, and attention dropout to 0.5, weight decay of 0.0001, loss function weight of 2. For optimization, we employ the Adam algorithm and train the model for 300 epochs in each experiment.

5.4 Environment

We use an Intel(R) Core (TM) i9-9900 CPU @ 3.10GHz and an NVIDIA GeForce RTX 2080 Ti GPU for our experiments. Our experiments are conducted on a 64-bit Windows 10 operating system, with CUDA version 11.6. We use the PyTorch 1.12.0 deep learning framework, with Python version 3.9.

5.5 Experimental results

5.5.1 Performance of disease progression prediction

Table 5 presents the mean and standard deviation of mAUC and BCA for our method and five comparison methods (without uncertainty estimation) experimented on 20 test sets. The best performance is highlighted in bold. It is evident that our method outperforms most of the comparison methods in terms of predictive performance and is comparable to the state-of-the-art models. However, the purpose of our method is not solely to pursue higher predictive performance. A vital aspect of the model is to effectively estimate the uncertainty of the prediction. Therefore, we conduct experiments to compare the performance of uncertainty modeling.

Table 5
Performance of different methods for predicting disease progression

Methods	mAUC $\uparrow$	BCA $\uparrow$
MinimalRNN	0.943 $\pm$ 0.014	0.884 $\pm$ 0.023
GRU	0.927 $\pm$ 0.014	0.874 $\pm$ 0.015
LSTM	0.925 $\pm$ 0.019	0.866 $\pm$ 0.025
LSS	0.926 $\pm$ 0.025	0.861 $\pm$ 0.029
SVM&SVR	0.929 $\pm$ 0.013	0.841 $\pm$ 0.023
Ours	0.943 $\pm$ 0.015	0.881 $\pm$ 0.017

Figure 6.

The change of predicted reliability with respect to accuracy.

5.5.2 Performance of uncertainty modeling

In order to verify whether our method can effectively model the uncertainty of model’s prediction and better reflect the predictive reliability, we plot the mean of predictive accuracy and reliability for error samples among MinimalRNN and the proposed method on the test sets across 20 experiments, as shown in Fig. 6. The predicted reliability of MinimalRNN’s is measured by the confidence of the predicted category, while our method’s reliability is measured as 1 minus the predicted uncertainty. Figure 6a shows the comparison of the predicted reliability and accuracy for our method. As the accuracy decreases, the predicted reliability for error samples also decreases, i.e., uncertainty increases, in the CN, MCI, and AD, proving that our method can effectively model the uncertainty of model’s predictions. By contrast, Fig. 6b shows the comparison for MinimalRNN, which yields high confidence for all categories even though the prediction is incorrect! Therefore, the confidence cannot effectively identify the potential errors, and cannot serve as a measure of the reliability of model predictions. At the same time, we give the ECE of the two models to quantitatively measure the gap between the two reliability measures. It can be seen that the ECE of the proposed method is much lower than the MinimalRNN, that is, the reliability of the output can better reflect the actual accuracy of the model.

5.5.3 Effect of the uncertainty threshold on predictive performance

The uncertainty of the model’s output can serve not only as a measure of the model’s reliability, but also to reduce potential erroneous predictions by setting an uncertainty interception threshold. In other words, when the uncertainty of the model’s output exceeds a certain threshold, the decision-making authority should be handed over to a medical professional. It is important to note that samples with uncertainty greater than the threshold do not participate in the performance evaluation since the model indicates the low reliability of the prediction result by outputting a larger uncertainty for such samples. Existing methods, on the other hand, are prone to overconfidence and make incorrect predictions for these samples. As shown in Fig. 7, as the uncertainty threshold decreases, the mAUC and BCA of the model’s prediction increase, demonstrating our method can effectively avoid the risks of misjudgment by implementing an uncertainty threshold and supporting reliable decision-making.

Table 6
Effect of AvUC loss on uncertainty calibration performance

Methods	AvU $\uparrow$
Ours (full)	0.831 $\pm$ 0.375
Ours (without AvUC)	0.776 $\pm$ 0.417

Figure 7.

The change of model performance with respect to uncertainty threshold.

5.6 Ablation experiment

5.6.1 AvUC loss and uncertainty calibration

To verify whether the AvUC loss can calibrate uncertainty, we compare our proposed method to a model without the AvUC loss on the test sets across 20 experiments. Table 6 shows the mean and standard deviation of the accuracy vs uncertainty (AvU) utility function values. It can be seen that our proposed method has a higher AvU value compared to the model without AvUC loss, indicating a higher proportion of correct and certain predictions as well as incorrect and uncertain predictions, demonstrating that the uncertainty is effectively calibrated.

5.6.2 AvUC loss and model calibration

Furthermore, we also verify the effect of introducing the AvUC loss on model calibration, as shown in Table 7. The table displays the mean and standard deviation of the expected calibration error (ECE) on the test sets across 20 experiments. Compared to the model without the AvUC loss, the model using the AvUC loss has a lower ECE, thus indicating better calibration performance. This suggests that the AvUC loss can not only improve the reliability of the model’s predictive uncertainty, but also effectively enhance the calibration performance of the model.

5.6.3 Temporal attention and predictive performance

In order to validate whether the temporal attention can boost the predictive performance of the model, we compare our model with a model that have the temporal attention removed (ERN). Table 8 presents the mean and standard deviation of mAUC and BCA on the test sets across 20 experiments. It is evident that the model with temporal attention outperforms ERN without temporal attention in terms of mAUC and BCA. This confirms that the temporal attention enables the model to fully utilize the historically time points that are crucial for the task, and thus improving the performance of the prediction for AD progression.

Table 7
Effect of AvUC loss on model calibration performance

Methods	ECE $\downarrow$
Ours (without AvUC)	0.184 $\pm$ 0.042
Ours (full)	0.101 $\pm$ 0.045

Table 8

Effect of temporal attention on predictive performance

Methods	mAUC $\uparrow$	BCA $\uparrow$
ERN	0.936 $\pm$ 0.019	0.869 $\pm$ 0.026
Ours	0.943 $\pm$ 0.015	0.881 $\pm$ 0.017

6. Conclusion

This paper presents temporal attention-aware evidential recurrent network for trustworthy prediction of Alzheimer’s disease progression. We propose evidential recurrent network, which introduces evidential deep learning regularized by the AvUC loss based on FastGRNN, to explicitly model the output uncertainty, and convert it into a reliability measure for trustworthy AD progression prediction. Meanwhile, the temporal attention allows the model to adaptively weight historical features at different time points to enhance the prediction performance. The performance measure on the TADPOLE dataset shows that the proposed method is comparable to the SOTA method in terms of disease progression prediction, and the ablation experiment proves the validity of temporal attention. For performance of uncertainty modeling, we find that the proposed method can provide a measure of uncertainty that represents the reliability of the predicted results, which is not available in other models. This provides an important decision-making basis for risky-sensitive AD progression prediction applications. Moreover, the AvUC loss can improve not only the performance of uncertainty calibration but also the performance of model calibration in a big degree. Future research will focus on utilizing the predicted uncertainty to adaptively weight training data to further improve the model’s predictive performance while representing the reliability of predicted results.

Footnotes

Acknowledgments

This work is supported by National Key Research and Development Program of China Grant Number 2022YFB3303800, Key Research and Development Program of Jiangsu Province Grant Number BE2021093, National Natural Science Foundation of China Grant Number 62202241, 62006126 and 61872190.

References

Gaugler

et al., 2019 Alzheimer’s disease facts and figures, Alzheimer’s & Dementia 15(3) (2019), 321–387.

Marinescu

R.V.

et al., TADPOLE challenge: Prediction of longitudinal evolution in Alzheimer’s disease, arXiv, 2018, arXiv: 1805.03909v2.

Scheltens

et al., Alzheimer’s disease, Lancet 388 (2016), 505–517.

Borson

et al., Report on milestones for care and support under the US national plan to address Alzheimer’s disease, Alzheimer’s & Dementia 12(2) (2016), 334–369.

Zhang

Shen

et al., Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer’s disease, NeuroImage 59(2) (2012), 895–907.

Young

et al., Accurate multimodal probabilistic prediction of conversion to Alzheimer’s disease in patients with mild cognitive impairment, NeuroImage: Clinical 2 (2013), 735–745.

Han

et al., Method on Alzheimer’s disease classification utilizing fuzzy logic feature selection and heterogeneous ensemble learning, Journal of Electronics & Information Technology 43(11) (2021), 3319–3326.

Moradi

et al., Machine learning framework for early MRI-based Alzheimer’s conversion prediction in MCI subjects, Neuroimage 104 (2015), 398–412.

Nguyen

et al., Modeling Alzheimer’s disease progression using deep recurrent neural networks, in: 2018 International Workshop on Pattern Recognition in Neuroimaging (PRNI), IEEE, 2018, pp. 1–4.

10.

Ghazi

M.M.

et al., Training recurrent neural networks robust to incomplete data: Application to Alzheimer’s disease progression modeling, Medical Image Analysis 53 (2019), 39–46.

11.

Nguyen

et al., Predicting Alzheimer’s disease progression using deep recurrent neural networks, NeuroImage 222(15) (2020), 117203.

12.

Ghazi

M.M.

et al., CARRNN: A Continuous Autoregressive Recurrent Neural Network for Deep Representation Learning From Sporadic Temporal Data, IEEE Transactions on Neural Networks and Learning Systems, 2022.

13.

N.H.

et al., Predicting progression of Alzheimer’s disease using forward-to-backward bi-directional network with integrative imputation, Neural Networks 150 (2022), 422–439.

14.

Guo

et al., On calibration of modern neural networks, in: International Conference on Machine Learning, PMLR, 2017, pp. 1321–1330.

15.

Amini

et al., Deep evidential regression, Advances in Neural Information Processing Systems, 2020, 14927–14937.

16.

Shad

et al., Designing clinically translatable artificial intelligence systems for high-dimensional medical imaging, Nature Machine Intelligence 3(11) (2021), 929–935.

17.

Sensoy

et al., Evidential deep learning to quantify classification uncertainty, Advances in neural information processing systems, 2018, 3183–3193.

18.

Bahdanau

Cho

and Bengio

, Neural machine translation by jointly learning to align and translate, arXiv preprint, 2014, arXiv: 1409.0473.

19.

Cheng

Dong

and Lapata

, Long short-term memory-networks for machine reading, arXiv preprint, 2016, arXiv: 1601.06733.

20.

Kusupati

et al., Fastgrnn: A fast, accurate, stable and tiny kilobyte sized gated recurrent neural network, Advances in neural information processing systems, 2018, 9031–9042.

21.

Krishnan

and Tickoo

, Improving model calibration with accuracy versus uncertainty optimization, Advances in Neural Information Processing Systems, 2020, 18237–18248.

22.

Jsang

, Subjective Logic: A formalism for reasoning under uncertainty, Springer, 2016.

23.

Dempster

A.P.

et al., Upper and lower probabilities induced by a multivalued mapping, Annals of Mathematical Statistics 38(2) (1967), 325–339.

24.

et al., Trustworthy Long-Tailed Classification, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 6970–6979.

25.

Ghesu

F.C.

et al., Quantifying and leveraging predictive uncertainty for medical image assessment, Medical Image Analysis 68 (2021), 101855.

26.

et al., Region-based evidential deep learning to quantify uncertainty and improve robustness of brain tumor segmentation, Neural Computing and Applications, 2022, 1–15.

27.

Zou

et al., EvidenceCap: Towards trustworthy medical image segmentation via evidential identity cap, arXiv preprint, 2023, arXiv:230100349..

28.

Chung

Gulcehre

Cho

K.H.

and Bengio

, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint, 2014, arXiv: 1412.3555.

29.

JACK

C.R.

et al., The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods, Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine 27(4) (2008), 685–691.

30.

Naeini

M.P.

Cooper

and Hauskrecht

, Obtaining well calibrated probabilities using bayesian binning, Proceedings of the AAAI Conference on Artificial Intelligence 29(1) (2019), 2901–2907.

31.

Varoquaux

et at., Cross-validation failure: Small sample sizes lead to large error bars, Neuroimage 180 (2018), 68–77.

32.

et al., Multi-modal sequence learning for Alzheimer’s disease progression prediction with incomplete variable-length longitudinal data, Medical Image Analysis 82 (2022), 102643.

33.

Tan

et al., DATA-GRU: Dual-attention time-aware gated recurrent unit for irregular multivariate time series, Proceedings of the AAAI Conference on Artificial Intelligence 34(1) (2020), 930–937.

34.

Men

et al., Multi-disease prediction using LSTM recurrent neural networks, Expert Systems with Applications 177 (2021), 114905.

35.

Ribeiro

L.C.F.

et al., Bag of Samplings for computer-assisted Parkinson’s disease diagnosis based on Recurrent Neural Networks, Computers in Biology and Medicine 115 (2019), 103477.

36.

Liang

et al., Rethinking modeling Alzheimer’s disease progression from a multi-task learning perspective with deep recurrent neural network, Computers in Biology and Medicine 138 (2021), 104935.