Deep Reinforcement Learning-Based Retinal Imaging in Alzheimer’s Disease: Potential and Perspectives

Deep learning

DL utilizes deep neural networks through representation learning, which can directly process raw data into suitable representations for pattern recognition [22]. In terms of information processing, the neural network is analogous to a biological brain: artificial neurons are arranged into immense layers [23], with the weight of the connections continuously adjusted throughout the learning process to attain remarkably complex functions. Manual steps such as feature extraction, selection, and dimensionality reduction are no longer required as the prerequisite in applying DL algorithms, making it desirable for large-scale, high-dimensional medical imaging analysis [24].

The most commonly used type of DL in analyzing images is the convolutional neural network (CNN), due to its exceptional feature recognizing ability [25]. CNNs are specialized variants of multilayer perceptron, in which each neuron in one layer is fully connected to all neurons of the next layer [26], similar to the human visual cortex. This particular architecture contributes to its two distinguishing properties: shared weights enable CNNs to learn features in a translational equivariance manner, i.e., the same feature would elicit a response in all neurons in a given convolutional layer [27], while pooling offers a certain degree of local translational invariance so that the features would still be recognized with slight distortion or positional shift [27]. The two properties work synergistically to achieve better generalizability in CNN architecture.

Reinforcement learning

RL is one of ML’s three basic paradigms, along with supervised learning and unsupervised learning. Unlike the other 2 paradigms that solely consider instant reward, the RL paradigm (Fig. 1A) contains an RL agent interacting with the environment through trial and error and aims to maximize the cumulative reward [28]. Its distinct nature in balancing exploration and exploitation, as well as not requiring prelabelled input and output, provides it with a substantial edge over supervised learning in real-time scenarios, where sequential decisions are made without an analytic solution.

OPEN IN VIEWER

Fig. 1

A) This is a simple illustration of the reinforcement learning (RL) process. The RL agent interacts with the environment in discrete time steps. First, it receives its current state from the environment. Second, it selects an action out of all possible actions depending on the information received. Finally, it transitions into the new state and the reward generated is determined by the environment. The iterative feedback loop is repeated until the terminal state is reached, such that the RL learns a policy that maximizes the cumulative reward through dynamic programming, which solves the optimization problem by breaking it into smaller sub-problems and finding their optimal solutions recursively. For example, in the detection of Alzheimer’s disease, the environment is constructed using input from imaging modalities such as fundus photography and optical coherence tomography. The task of the agent is to interact with the environment and obtain an optimal policy that maximizes the cumulative reward. The outcome, i.e., whether the patient has AD or his/her predicted progression, can be retrieved through the optimal policy. B) An RL problem is represented by a Markov Decision Process (MDP), a discrete-time stochastic control process that is composed of agent states, possible actions, the actions’ effects in each state and a reward function. It provides a mathematical framework for modelling decision-making in situations where outcomes are partly random and partly under the control of a decision-maker. In this figure, the states S_t are represented in blue circles, while the actions available at that state A_t are represented by the red circles. For instance, when the agent is at state S₀, it will be presented with the action A₀, where it can stay at S₀ or move to S₁. The probability that action A₀ in state S₀ will lead to it staying at state S₀ at the next time state is 0.5, while the probability of it transitioning to state S₁ is 0.5, which are represented by the black numbers on the arrows and are defined by the policy. The reward which the agent will receive by moving into the new state is represented by the green numbers. By repeating the process, the aim is to find an optimal policy for the agent, where the cumulative reward of the actions are maximized.

An RL problem is represented by a Markov Decision Process (MDP), composed of agent states, possible actions, the actions’ effects in each state, and a reward function (Fig. 1B) [29]. The RL agent interacts with the environment in discrete time steps. Firstly, it receives its current state from the environment. Subsequently, it selects an action out of all possible actions depending on the information received. Finally, it transitions into the new state and the reward generated is determined by the environment [28]. The iterative feedback loop is repeated until the terminal state is reached, such that the AI learns an “optimal reward policy” through dynamic programming [30], i.e., solving the optimization problem by breaking it into smaller sub-problems and finding their optimal solutions recursively.

Deep reinforcement learning

Although RL-based approaches have achieved success in a variety of problems, their applicability is limited to domains with fully observed and low dimensional spaces, and their efficacy is bottlenecked by challenges in hand-crafted feature design (i.e., conventional ML) in shallow models. Thus, with the integration of DL as a nonlinear approximator of value function or policy function to generalize across enormous state spaces [28], DRL is particularly useful for sophisticated, consequential decision-making problems with high-dimensional medical images such as three-dimensional (3D) anatomical landmark detection [31], vessel centerline tracing in 3D volumetric magnetic resonance imaging (MRI) and computerized tomography (CT) scans [32]. The value function, policy, and model in RL (i.e., an MDP) are approximated with the application of a deep neural network to manage continuous states or actions that are difficult to be represented in tabular functions.

Optical coherence tomography in detecting retinal neuronal changes in AD

OCT is a non-invasive imaging technique that employs low-coherence interferometry for in vivo ocular measurements [33]. The individual retinal layers could be visualized in the high-resolution images, including the retinal nerve fiber layer (RNFL), the ganglion cell layer (GCL), and the inner plexiform layer (IPL). The quantitative segmental analysis of retinal layers enabled scientists to study their relationship with cognitive function impairment. Neuronal loss in the hippocampus and cerebral neocortex is a typical characteristic of AD [34]. The loss of RGCs and their axons in AD patients [35] is usually exhibited in the RNFL surrounding the optic nerve head (peripapillary RNFL). RNFL thinning in all four quadrants, especially in the superior and inferior quadrants, is observed in patients with AD in multiple OCT studies [36]. Macular GC-IPL thickness can also be measured through OCT, which demonstrates thinning around the fovea in most sectors under the manifestation of AD [37]. While the relative accuracy and sensitivity between the two parameters remain controversial, their association with AD demonstrates potential use as indicators of neurodegeneration for assisting early diagnosis.

Optical coherence tomography angiography in observing retinal capillary changes in AD

Apart from neuronal structures in the retina, vascular structures also demonstrate significant values in exhibiting pathological changes of AD. OCT-A is commonly used to visualize retinal and choroidal microvasculature at capillary level based on the principle of motion contrast: a series of OCT B-scans is taken at the same cross-section, and the laser reflectance from red blood cells in the vascular lumen is compared. The difference in decorrelation signal allows retinal capillaries and blood flow rate to be precisely mapped out non-invasively [38]. Computer algorithms are employed to reduce motion noise and segment the image into multiple zones, most commonly the superficial capillary plexus, the deep capillary plexus, and the radial peripapillary capillary plexus.

In recent years, there has been emerging evidence showing that AD is a multifactorial disease that overlaps neuropathologically with cerebrovascular diseases [39]. Possible speculation about the phenomenon is that hypoperfusion modulates β and γ-secretase which increases Aβ production [40]. Besides, vascular-mediated Aβ clearance is impaired by cerebrovascular diseases, thus further exacerbating the condition of AD patients [41]. The paradigm shift in the understanding of AD pathophysiology encouraged scientists to probe into microvascular changes in the retina. For instance, Bulut et al. have identified a reduction in retinal vascular density and enlargement in the foveal avascular zone in patients with Alzheimer’s type dementia [42]. Yet, most current studies are limited by their small sample size and often neglect other confounding factors [5]. More investigations are required to achieve a credible consensus.

Fundus photography in monitoring retinal arteriolar and venular changes in AD

Besides retinal capillaries, changes in larger vessels may also serve as a potential indicator for observing the microvascular pathology related to AD. FP is a simple but effective technique for capturing signs of vascular diseases from optically accessible retinal vessels. As hypoperfusion is a possible contributing factor to the development of AD, a reduction in retinal vascular density is expected in diseased individuals. Indeed, Cheung et al. reported a reduction in both arteriolar and venular fractal dimensions in association with AD dementia [43], which indicates a sparser vascular network and is consonant with the hypothesis. As for vessel caliber, Frost et al. and Cheung et al. concluded that narrower venule caliber is associated with an increased risk of AD [43, 44], although the effect is not consistently demonstrated in some studies. The relation of vessel tortuosity with the development of AD remains contentious, as Cheung et al. discovered that both increased retinal alveolar and venular tortuosity are associated with the risk of AD [43], while William et al. stated that reduced retinal alveolar tortuosity is associated with the risk of AD, with no significant correlation with retinal venular tortuosity [45]. The disputing results are attributed to cardiovascular comorbidities such as hypertension and diabetes in the subjects, which were believed to increase retinal vessel tortuosity as well [46, 47]. Studies adjusted to other clinical factors are required to confirm the validity of the hypothesis.

Alzheimer’s disease classification

As multi-modal DL is shown feasible for AD classification from multiple ocular imaging modalities [10] and RL further offers a robust alternative by mimicking the cognitive flow of the human brain, the DRL algorithm can potentially be applied to the decision-making of AD classification via multimodal evaluation. For instance, Zhang et al. have proposed a multimodal DRL model for recognizing pathological features from MRI and diagnosing AD, which achieved an AUROC of over 95% in multiple datasets [52]. In the study, DRL was utilized to reconstruct compressed-sensing MRI images in order to improve clinical feasibility. A pixel-level disease probability risk map of the brain was then generated by feeding the reconstructed images into a full CNN. Finally, the risk map of specific brain regions was integrated with individual clinical data for biomarker analysis and obtaining a diagnosis with DL algorithms. With the advancement in the DRL technique itself, such as applying off-policy learning to enable the values of an optimal policy to be learned independently of the agent’s action [53], DRL is more catered towards personalized scenarios compared to other ML modalities, which is of particular importance due to the high degree of individual variance in multifactorial diseases such as AD. Besides, DRL possesses the edge over DL in terms of adaptability in sequential decision-making, especially in unseen cases with incomplete information, where DRL is able to take account into domain knowledge and learn policies that are able to generalize well in uncertain scenarios. Compared with physicians’ diagnosis, which is often suboptimal even in developed countries due to the insidious onset of AD and time-consuming diagnostic process [54], DRL can potentially offer an automated alternative that is less prone to human error and with a lower technical barrier, as it can take account into the diverse factors with minimal bias under an optimal policy. As retinal imaging allows the visualization of various retinal neuronal changes and vascular changes that correlates with AD, DRL exhibits its potential in the manipulation of high-dimensional retinal images such as 3D volumetric OCT images into a more interpretable form, which aids in the development of assessable AD detection techniques (Fig. 3). In the future, more studies are warranted to further investigate the advantages of DRL in AD classification and the comparisons to physicians or other ML models.

OPEN IN VIEWER

Fig. 3

Deep reinforcement learning (DRL) integrates deep learning as a nonlinear approximator of value function or policy function into conventional reinforcement learning. It is particularly useful for sophisticated and consequential decision-making problems with high-dimensional input such as medical images.

The application of DRL also offers an autonomous alternative for meticulous tracing, classification, and segmentation of changes in the retina. By applying DRL, the agent can learn a Q-function that enables it to search for the landmark effectively without examining the image exhaustively [55]. The agent begins at a coarse level to capture the global context and gradually increases its scale level to search for local details until the finest scale is reached. The landmark is pinpointed at the site of convergence, where the trajectory of the agent moves in a small, oscillatory cycle [56]. The implementation of DRL improves both the speed and accuracy of vessel detection and more importantly reduces the computational cost required for multidimensional images. The entire coronary tree can then be extracted with a single initial seed point with DRL, which achieves better performance than the supervised CNN approach with its optimization ability [56], holding great potential in developing multi-modal retinal imaging-based models for AD detection.

Alzheimer’s disease progression prediction

Understanding AD progression is of equal importance to identifying its presence, as it can be used to evaluate the severity of patients in a wide spectrum, formulate personalized management plans, and aid therapy development. In the hope of elucidating the pathophysiological process of AD, Saboo et al. have proposed a contemporary model that incorporates both RL and differential equation (DE), which is often used in mathematics to represent time-depending phenomena [57]. The DE-based model utilizes acknowledged mathematical relationships between factors and leverages the optimization power of RL to derive the unknown associations based on domain knowledge. The objective function aims to minimize the deficit between the cognitive demand of a task (represented by the maximum score of MMSE) and the cognition provided by the brain (proportional to the activated area measured by MRI), as well as reduce the energetic cost of cognition (proportional to the total brain activity measured by fMRI) [57]. By solving the optimization problem through RL, the model was able to outperform state-of-the-art models and the proposed model without RL in predicting a 10-year cognition trajectory, showing the effectiveness of RL implementation in presenting a more comprehensive view of the disease. It was also able to exhibit the compensatory process during disease progression, which involves the recruitment of specific brain regions for cognition to compensate for the neurodegeneration [58], without explicit manual programming.

This model presents a new direction for integrating RL with DE to construct a progression model, potentially bridging the knowledge gap and providing more insight into the biological processes that contribute to neurodegenerative diseases. The generic framework can be further explored to encompass retinal imaging by combination of DRL and application of different DEs, as retinal imaging offers a comprehensive view of the eye through different field-of-view (e.g., optic disc-centered and macula-centered) and neuronal layers (e.g., RNFL and GCIPL). With analogous changes associated with AD progression in the brain such as Aβ accumulation [5], parameters extracted from retinal images can possibly be derived similarly into an optimization problem for the prediction of AD progression. Studies of intervention approaches are also possible due to the mechanistic property of the model through perturbation analysis of the variables [59].

Long-term management of Alzheimer’s disease

As one of the major causes of disability and dependency on elderlies, AD carries a hefty economic burden on the healthcare system, with an estimated total global societal cost of US$ 1.3 trillion in 2019 [60]. With no curative treatment currently, the management of AD mainly falls under symptom relief and caregiving, thus requiring substantial effort from both clinicians and caregivers to keep track of the patient’s status. With this in mind, DRL offers a promising solution to provide management for patients in a more personalized and streamlined approach. For instance, the reward of the DRL algorithm could be defined by diminishing degenerative changes in the retina from images obtained by various modalities or reducing decline in cognitive function. Based on the patient’s state, the DRL agent can choose from a variety of actions, including the prescription of symptom-relieving drugs such as cholinesterase inhibitors and memantine [61], lifestyle modifications such as physical activity, and diet arrangement, or arrange follow-up by a clinician. Through off-policy learning, DRL can potentially learn from the disease progression trajectory of past patients and learn an optimal policy for managing AD patients on a case-by-case basis. Its immediate application can provide more timely intervention under the high patient volume, and potentially derive more systematic management protocols that can immensely benefit patients in low-resource settings.

Difficulty in defining reward function due to complexity of AD

A well-constructed reward function remains the cornerstone for a DRL algorithm’s performance, as it determines how the agent behaves. However, the complexity of AD can greatly convolute the process, especially when immediate reward may not be accessible to provide feedback and navigate the agent towards an optimal policy [56], due to various knowledge gaps in AD pathophysiology. To tackle real-world scenarios where many factors remain unknown, the application of inverse DRL is actively explored. Inverse DRL is a type of apprenticeship learning that extracts a reward function through observing demonstrations of a task performed by experts (expert trajectories) [62]. This reduces the necessity of manual specification, which is often intractable to take account of the diverse factor in medical diagnosis.

Curse of dimensionality

The curse of dimensionality has been a prominent problem in dynamic programming. It refers to the difficulty in identifying meaningful patterns when working with high-dimensional data. The rationale behind the phenomenon is that the same number of data points become sparser when distributed among different dimensions, resulting in an exponential increase in the amount of data and computational power required for ML models to obtain a reliable result [63]. In the context of medical research, dimensionality refers to the number of describing features. For instance, age, gender, medical history, the neuronal and vascular changes in the brain and the retina are all useful features for AD identification. With the extensive interindividual variation, it becomes challenging for the algorithms to generalize the correlation between each feature with the patient’s probability of having AD. Dimensionality reduction is one of the approaches to resolve the issue by projecting the states onto a lower-dimension manifold. Previous studies have utilized dimensionality reduction algorithms such as principal component analysis, kernel principal component analysis, isometric feature mapping, linear discriminant analysis and autoencoder to optimize AD classification [64–66]. The trade-off between reduced performance and faster convergence should be investigated to maximize the effectiveness of DRL in medical imaging.

Data availability

Predictive accuracy is of paramount importance in constructing a disease prediction model, which is generally ‘data-hungry’ when applied to complex, multifactorial diseases such as AD. DRL is generally applied in gaming or robotics at the current developmental stage, where new data can be generated through simulated runs in the training process. However, the data in medical research is generally obtained from individual patients, which can be costly and time-consuming to build extensive datasets. For instance, 12,949 retinal photographs from 648 AD patients and 3,240 control subjects were used to train, validate and test the DL model from Cheung’s group, while the retinal photographs from 36 AD patients and 123 control subjects were used to develop the model from Wisely’s group, where AUROCs of 0.93 and 0.861 were achieved, respectively [10, 12]. While there is no one-size-fits-all answer for the data size required to develop an AI-based model for disease detection, it is generally believed that a larger data size contributes to better accuracies of analytical models. Large longitudinal cohorts are also lacking for retinal imaging on AD patients compared to other chronic diseases [49]. Therefore, the development of more data-efficient algorithms for AD detection and management is vital for its further application.

Reproducibility and reliability of RL results

Sustainable advancement of medical research is often reliant on the reproducibility of existing works, to evaluate the legitimacy of novel discoveries and further build upon the knowledge. However, the reproduction of state-of-the-art DRL methods is often problematic, due to the lack of standard benchmark environments and the high degree of variance inherent to the mechanism of DRL [67]. To ensure straightforward interpretation of previous works, standardization of metrics in experimental reporting is necessary to reduce ambiguity. Regarding the decision-making aspect of RL, whether the model had complete access to all confounding factors such as concomitant systemic diseases and ocular diseases, shifts in hospital diagnosis and treatment standards, as well as errors in problem formulation or data processing can also affect its reliability [68]. Therefore, methods to improve the interpretability of DRL models should be actively explored to reduce the ‘black box’ phenomenon and ensure that it will behave in a regulated and intended way.

Lack of generalizability

Although many studies demonstrate associations between retinal parameters and AD [5, 35–37, 42–45], their practicality in clinical use is in doubt due to the intrinsic bias of the datasets and comorbidities of subjects that alter retinal parameters. To isolate the effect of AD on retinal changes, larger datasets from different populations are required to reduce sampling bias. The diversity of datasets could also be increased by including subjects with concomitant ocular neurodegenerative or cardiovascular pathologies, thus enhancing the generalizability of the result in real-life scenarios with complex patient conditions.

Education and infrastructure changes

Despite the numerous successes of RL in recent years, its development in the medical field has remained rather stagnant. One of the major reasons is the lack of systematic understanding of its potential implementation, as well as its advantages and shortcomings compared to other ML subtypes. Education, through reviews or seminars, are necessary for researchers and physicians to gain more exposure to the mechanisms of DRL and consider it a feasible tool for AD detection and management. Once it has accumulated its popularity, infrastructural changes should be introduced to accelerate the application of DRL in AD.