Diagnostic Validity of the Smart Aging Serious Game: An Innovative Tool for Digital Phenotyping of Mild Neurocognitive Disorder

Abstract

Background:

The Smart Aging Serious Game (SASG) is an ecologically-based digital platform used in mild neurocognitive disorders. Considering the higher risk of developing dementia for mild cognitive impairment (MCI) and vascular cognitive impairment (VCI), their digital phenotyping is crucial. A new understanding of MCI and VCI aided by digital phenotyping with SASG will challenge current differential diagnosis and open the perspective of tailoring more personalized interventions.

Objective:

To confirm the validity of SASG in detecting MCI from healthy controls (HC) and to evaluate its diagnostic validity in differentiating between VCI and HC.

Methods:

161 subjects (74 HC: 37 males, 75.47±2.66 mean age; 60 MCI: 26 males, 74.20±5.02; 27 VCI: 13 males, 74.22±3.43) underwent a SASG session and a neuropsychological assessment (Montreal Cognitive Assessment (MoCA), Free and Cued Selective Reminding Test, Trail Making Test). A multi-modal statistical approach was used: receiver operating characteristic (ROC) curves comparison, random forest (RF), and logistic regression (LR) analysis.

Results:

SASG well captured the specific cognitive profiles of MCI and VCI, in line with the standard neuropsychological measures. ROC analyses revealed high diagnostic sensitivity and specificity of SASG and MoCA (AUC_s > 0.800) in detecting VCI versus HC and MCI versus HC conditions. An acceptable to excellent classification accuracy was found for MCI and VCI (HC versus VCI; RF: 90%, LR: 91%. HC versus MCI; RF: 75%; LR: 87%).

Conclusion:

SASG allows the early assessment of cognitive impairment through ecological tasks and potentially in a self-administered way. These features make this platform suitable for being considered a useful digital phenotyping tool, allowing a non-invasive and valid neuropsychological evaluation, with evident implications for future digital-health trails and rehabilitation.

Keywords

Dementia digital medicine mild cognitive impairment mild neurocognitive disorder neuropsychological assessment serious games telemonitoring vascular cognitive impairment virtual reality

INTRODUCTION

As the world’s general population becomes older, with the number of people aged 60+ expected to reach more than 2 billion (one in five/six individuals) in 2050 [1, 2], the scientific interest in the aging process turns crucial to face with the social, health, and economical requests brought by aging-related issues. With aging, metabolic and neurobiological brain changes challenge the individual’s capacity to cope with the maintenance of previous levels of cognitive performance [3 –5]. In this situation, the brain can reveal important neural capacities to functionally reorganize itself and to redistribute resources in order to maintain the same performances as in youth [3 –5], in a phenomenon called “neurocognitive aging” [5]. However, not all individuals present with the same resources and a proportion of older adults may experience an unsuccessful aging process [2]. Neurocognitive disorders are conditions characterized by major or mild decline from previously attained levels of cognitive functioning [6, 7]. These conditions might have different biological substrates and are subtyped according to their etiology, with amyloid-β deposition and cerebrovascular diseases as potential causes [6].

Major neurocognitive disorders are characterized by significant cognitive decline associated with important interferences in autonomies, as in dementia [6, 7]. Considering the maturing age of the population, dementia is impacting importantly on healthcare systems at the worldwide level, with a forecast of 115–150 million people affected by dementia in 2050 [8, 9]. This evidence makes dementia a public health priority [9]. Prevention and early intervention are thus mandatory in social and sanitary politics, mainly acting on pre-clinical conditions, as the mild neurocognitive disorders. These conditions are characterized by a modest decline in one or more cognitive functions and by a maintained independence in everyday activities [6, 7]. Mild neurocognitive disorders are estimated to be present in about 20%of non-demented older adults aged over 60 years, and the percentage increases in older individuals [10].

Among the mild neurocognitive disorders, mild cognitive impairment (MCI) is a condition affecting 6.7%–25.2%of the population (60–80 + years) [11]. MCI lies on the continuum between healthy aging and dementia (especially Alzheimer’s disease, AD), and it is characterized by the maintenance of personal autonomies even in the presence of mild cognitive deficit in the amnestic (amnesic MCI, aMCI) and/or non-amnestic sphere [7, 12]. Individuals with MCI most commonly experience deficits in episodic memory (especially those subjects most likely to develop AD), but they can also present with difficulties in executive functions, attention, language, or visuospatial skills [13]. The neurobiological substrate is represented by the accumulation of amyloid plaques and neurofibrillary tau protein tangles in medial temporal lobe structures typically involved in AD pathology [14, 15]. People with MCI older than 65 years present with a cumulative incidence for dementia of about 15%when followed for two years [11].

Vascular cognitive impairment (VCI) refers to mild neurocognitive disorders with a dominant (if not exclusive) vascular etiology [6], going from ischemic white matter hyperintensities to infarcts [6, 16]. From a neuropsychological perspective, people with a diagnosis of VCI maintain their complete autonomy, but they might present with deficits primarily affecting processing speed, attention, and executive functions (well captured by timed tests of executive functioning), while other cognitive abilities remain relatively intact [17]. With the accumulation of vascular insults, the worsening of the cognitive profile and the loss of personal autonomy, the diagnosis of VCI can meet the criteria for vascular dementia (VaD), the second most common form of dementia [8]. This makes VCI a risk factor for the development of dementia [8].

Considering the higher risk for dementia of both MCI and VCI, their timely diagnosis is becoming a clinical imperative, as they represent the ideal candidates for early rehabilitative intervention and care [18 –22]. Moreover, the possibility to make a differential diagnosis between the two conditions by digital phenotyping opens the perspective of tailoring the intervention based on their neurobiological basis.

Nowadays, the international guidelines ask for the use of specific biomarkers for the diagnosis of AD and VaD as clinical-biological entities. Whereas for VaD, the presence of neuroimaging data on vascular insults remains the only biomarker available [6], the diagnosis of AD might require the use of specific biomarkers of amyloid-β deposition, tau protein accumulation, and/or brain neurodegeneration [23]. Although with good diagnostic value, these diagnostic methods are invasive and expensive and thus are not suitable for massive use in the screening of the general population. For this reason, in clinical practice, the neuropsychological pencil-and-paper tests still play a crucial role in the diagnosis of dementia. However, the majority of these instruments are often not effective in the detection of slight deviations from healthy aging, typical of the earliest phases of the neurodegenerative diseases [24], such as MCI and VCI. The recent development of new digital technologies opens new diagnostic perspectives. Digital biomarkers can support clinicians in the timely diagnosis of pre-clinical conditions, by assessing cognitive functions using sensitive and ecologically valid tools resembling everyday activities in a realistic fashion and by collecting potentially huge amount of data in each usage, whose analysis can offer more sensitive clinical profiles [24] allowing personalized intervention in order to decrease the probabilities to develop dementia [25, 26].

Among new technological modalities, virtual reality (VR) and serious games (SGs) consist of promising tools for the assessment of functionality loss in pathological aging [27 –30]. They are also used to predict outcomes and to strengthen randomized controlled trials results [31]. One of the major advantages of SGs is the possibility to integrate VR-based real-life routines in the evaluation process, making them ideal tools for the reliable evaluation of functioning in everyday living. SGs, such those hosted in digital applications, are now able to integrate VR environments to assess cognitive functions with different levels of complexity [19 , 32–35] assuring the ecological validity of the measurements, an aspect often lacking in conventional neuropsychological tools, also in neurological conditions [30 , 36–38]. In this line, the higher sensitivity of VR-based measures, compared to pencil-and-paper tests, has been demonstrated for the detection of cognitive impairment [39]. Moreover, VR has been previously presented in the literature not only to investigate and assess, but also to rehabilitate cognitive abilities. A recent six-years review of the effects of VR-based neurorehabilitation on multiple functional domains pointed out the promising beneficial results of the application of VR-based tools for the treatment of neurological acute and neurodegenerative conditions [40]. In terms of the effects on multiple cognitive domains, VR has been shown to impact people’s engagement with the assessment, a crucial aspect to enhance the effects of long-term rehabilitation on cognitive domains [41, 42]. Hence, the systematic adoption of these digital measures in older adults would lead to the efficient detection and treatment of pre-clinical conditions and therefore in the effective prevention of dementia cases.

The Smart Aging Serious Game (SASG) [32, 43] is a digital platform for the evaluation of cognitive impairment in aging able to collect both active (performance) and passive (time of execution) data while the subject performs ecologically-based and multi-domain tasks in a 3D VR scenario. The possibility to potentially collect huge amount of execution data makes this platform a great ally for the future implementation of statistical big data approaches, in neurodegenerative conditions also in early disease stages. The SASG has been demonstrated to be valid for the application in healthy aging [43] and in the detection of aMCI [44], especially considering the significant relation between SASG score and the degree of hippocampal injury associated with the pathology [45]. These findings make SASG suitable as a digital tool for the pre-clinical stage of AD.

The assessment of several neurocognitive domains and the ecological value of the SASG make it an ideal candidate also for the diagnosis of other pre-clinical conditions, such as VCI. The present study has a twofold aim: 1) to confirm the validity of SASG in detecting people with MCI from a condition of neurocognitive aging; 2) to evaluate the diagnostic validity of this tool in differentiating between VCI and healthy aged subjects. We used a multi-modal statistical approach to achieve these goals, including both receiver operating characteristic (ROC) curves comparison, random forest, and logistic regression analysis.

METHODS

Participants

Eighty-eight people with cognitive decline were consecutively recruited for the present study. Among those, 61 subjects had a diagnosis of MCI due to AD (mean age = 74.26±5.00, 26 males) and 27 subjects had a diagnosis of VCI (mean age = 74.22±3.43, 13 males). Seventy-seven healthy controls were included as well (HC, mean age = 75.36±2.83, 38 males). Four subjects (n = 1 MCI, n = 3 HC) were excluded from analysis due to technical issues during the SASG administration session. Patients were invited to take part in the research during periodical neurological screening by physicians at the IRCCS Don Gnocchi Foundation of Milan, the IRCCS Don Gnocchi Foundation of Florence and the Memory Clinic of University Hospital of Careggi of Florence (Italy). The study was approved by the Ethics Committee of the Don Gnocchi Foundation and all subjects gave written and informed consent.

Inclusion criteria for eligibility of participants included: 1) diagnosis of MCI due to AD according to Albert’s et al. [13] criteria or VCI according to Hachinski criteria [46], 2) evidence of the presence of a cognitive impairment at neuropsychological evaluation; Mini-Mental State Examination [47] score≥24, 3) age ranged between 65–80, 4) years of education≥5. The following exclusion criteria were considered: 1) cognitive decay due to an acute or general medical disorder, 2) severe behavioral and/or psychiatric disturbances, 3) substance abuse disorder, 4) auditory or visual disturbances able to impact on the performance of neuropsychological tests, 5) stable pharmacological treatments of cholinesterase inhibitor, memantine, antidepressant, and antipsychotic drugs.

All subjects read and signed the written informed consent before taking part in the research.

After enrollment, participants were divided into two groups (MCI due to AD and VCI) following the classification criteria of the National Institute for Neurological Disorders and Stroke and the Canadian Stroke Network [46] reported below:

MCI due to AD: the presence of the biomarker of amyloid-β deposition or neuronal injury (hippocampal atrophy, FDG-PET imaging or CSF tau/phosphorylated-tau) compatible with MCI due to AD condition, Hachinski ischemic score≤4, absence of other dementia-causes (revealed with TC or MRI), Fazekas score < 2 [48], and amnestic presentation.

VCI: Evidence (MRI/TC) of white matter lesions due to small vessel disease, lacunar state and ischemic lesions, Hachinski ischemic score > 4, Fazekas score≥2 [48].

The HC were an age-, gender-, and education-matched cohort selected from the study of Cabinio et al. [45]. All HC were independently living and were native Italian speakers, in the absence of a major neurological complaint, and with a Mini-Mental State Examination score≥28.

Procedure

Each participant performed a neuropsychological evaluation using conventional paper-and-pencil tools (around 45–60 min) and a session of SASG (around 20–4 min), in a single session.

Measures

Conventional neuropsychological measures

Participants’ cognitive functions were assessed through:

Montreal Cognitive Assessment (MoCA) [49] for a global cognitive level screening. The total score for the scale, ranging 0–30, was adjusted for age and years of education following indications of Conti et al. [49];

Free and Cued Selecting Reminding Test (FCSRT) [50] to evaluate episodic memory, in terms of both immediate and delayed recall. The scores of Immediate Free Recall (IFR, range 0–36) and Delayed Free Recall (DFR, range 0–36), Immediate Total Recall (ITR, range 0–12) and Delayed Total Recall (DTR, range 0–12) were considered for the present study. According to Frasson et al. [50], IFR and DFR scores were corrected for sex, age, and years of education.

Trail Making Test (TMT) [51] for the evaluation of executive function, mental flexibility, visual search, and processing speed. Part A and B of TMT (TMTA and TMTB) were utilized, and total scores were adjusted for age and years of education according to Giovagnoli et al. [51].

The Smart Aging Serious Game

The SASG is a serious game conceived for the screening of cognitive functions in the older adult population who are required to perform tasks in a VR scenario resembling a real-life setting. This innovative tool has been amply presented in previous works and validated as an effective measure of cognitive functions in healthy aging [32, 43]. Recently, its validity in the detection of hippocampal degeneration has been reported [45], demonstrating its potentiality as a digital biomarker. Shortly, SASG offers a multiple cognitive domains screening by assessing working, prospective and long-term memory, spatial orientation, executive functions, divided and selective attention. By easily interacting with a touch-screen, subjects virtually navigate in a 3D scenario resembling a loft fitted with a kitchen, a dining room, and a bedroom. The subject is required to accomplish five tasks, in a fixed order, consisting of functional activities of everyday life (e.g., make a phone call, remember to turn the TV on after an activity, identify and remember the location of objects in the kitchen). In detail, in Task1 (T1) individuals have to recall the position of objects previously shown in the kitchen. Task2 (T2) requires a dual-task in which subjects have to listen to the radio and press a button every time they hear the word “sun” and at the same time water the flowers. In Task3 (T3) subjects have to make a phone call and to remember to turn on the TV after the call. Task4 (T4), the only task with a 2D graphical appearance, consists of a memory task in which 24 images are shown and the subject has to recognize the 12 objects shown in T1. Finally, in Task 5 (T5), subjects are asked to navigate in the kitchen and search for all the objects shown in T1.

To familiarize participants with the technological system, a 10-min interactive demo was used.

During the SASG session, different indices were automatically recorded for each Task, among these: (A) Performance time: milliseconds from beginning to the end of each the Task; (B) Performance accuracy: total number of objects correctly remembered by the subject (for Tasks 1, 3, 4) or total number of correct actions performed (for Tasks 2, 5). Moreover, additional data are collected from the platform (but were not used in the present research) in the different Tasks, such as: (i) Clicks: number of clicks; (ii) Errors: erroneously recognized objects or omissions; (iii) Distance: distance covered within the loft.

In this study, according to [43, 45], only Performance Time (A) and Performance Accuracy (B) data were used. According to [43, 45] all the raw scores were converted to z-scores (using the mean and standard deviation of the HC sample). This transformation both highlighted performance under 2 standard deviations from the HC and allowed to compute the following three composite scores:

SASG-total-time. This score was computed by summing Performance Time (z-scores) of each Task, as follows:

$\begin{matrix} SASG - total - time = \\ ({SASG}_{time - task 1} + {SASG}_{time - task 2} \\ + {SASG}_{time - task 3} + {SASG}_{time - task 4} \\ + {SASG}_{time - task 5}) \end{matrix}$

SASG-total-acc. This score was computed by summing Performance Accuracy (z-scores) of each Task, as follows:

$\begin{matrix} SASG - total - acc = \\ ({SASG}_{acc - task 1} + {SASG}_{acc - task 2} \\ + {SASG}_{acc - task 3} + {SASG}_{acc - task 4} \\ + {SASG}_{acc - task 5}) \end{matrix}$

SASG-total . This score was computed, as follows:

\sum_{N = 1}^{5} (SAS G_{acc - TaskN} - SAS G_{time - TaskN})

Statistical analysis

Demographical statistics and neuropsychological assessment

Statistical analyses were performed using IBM SPSS Statistics software (Version 26), JAMOVI and JASP (Version 0.14). Normal distribution of variables was verified through the Kolmogorov–Smirnov normality test and parametric or non-parametric tests were considered accordingly.

Descriptive statistics were run for the demographical description of the sample and the performance of conventional and non-conventional neuropsychological measures using IBM SPSS Statistics software (Version 26). Chi-squared test and univariate ANOVA were utilized to verify that HC, VCI and MCI groups were balanced in terms of gender, age, and educational level. Parametrical (One-way ANOVA) or non-parametrical (Kruskal-Wallis H test) were run as appropriate to compare the groups’ performance on standard neuropsychological tests (MoCA, FCSRT, TMT) and SASG-total. Post-hoc comparisons were adjusted with Bonferroni’s correction for multiple tests (significance was considered when p < 0.006).

ROC curve comparison

Furthermore, according to Cabinio et al. [45], to compare the discriminant capacity of SASG-total with those of conventional pencil-and-paper tests (MoCA, IFR, DFR, TMTA, and TMTB), ROC curve analyses with the software Jamovi (Version 1.2.25.0; https://www.jamovi.org) were performed considering two conditions: 1) discriminate VCI from HC, and 2) discriminate MCI from HC. The area under the ROC curve (AUC) provides a measure of overall prediction accuracy and corresponds to random chance when AUC is equal to 0.5 and represents perfect accuracy when AUC is 1. According to the literature [52], we considered accuracy values 0.7 to 0.8 as acceptable, 0.8 to 0.9 as excellent, and more than 0.9 as outstanding.

Classification of healthy controls or patients

To further investigate the overall discriminant capacity of SASG-total and neuropsychological pencil-and-paper tests in predicting the two conditions (1. Discriminate VCI versus HC and 2. Discriminate HC versus MCI) and the weights of each potential predictor, random forest (RF) analysis and logistic regression (LR) implemented in JASP (Version 0.14) were adopted. The classification models run included the same variables considered in the previous ROC analysis altogether (see section on “ROC Curve comparison”: MoCA, IFR, DFR, TMTA, TMTB, and SASG-total). RF was run with the default parameter values set in JASP, specifically with respect to the data split we partitioned the data set into a training (60%), validation (20%), and test set (20%). In relation to the number of trees we selected an optimal number of trees (Ntrees (maximum) = 100), optimized with respect to the out-of-bag accuracy. LR was performed using the enter method. Classification accuracy represents the proportion of the instances that were classified correctly. Performance of the classification model was also evaluated by carrying out a ROC analysis. Precision represents the proportion of true positives among all the instances classified as positive, and F1 Score indicates the harmonic mean of Precision and Recall.

RESULTS

Participants

A total of 161 subjects was included in the analyses: 74 HC, 60 people with amnestic multidomain MCI and 27 people with VCI. The three groups of participants were balanced in terms of age, sex, and level of education distribution (see Table 1 for details).

Table 1

Demographic characteristics of the sample

	HC	MCI	VCI	p
N	74	60	27
Age (M, sd)	75.47, 2.66	74.20, 5.02	74.22, 3.43	0.206^∧
Sex (Ma:F)	37:37	26:34	13:14	0.740°
Years of education (5–8, 9–13, 13+)	29:34:11	29:21:10	18:7:2	0.152°

F, females; HC, healthy control group; M, mean; Ma, males; MCI, mild cognitive impairment group; N, number; sd, standard deviation; VCI, vascular cognitive impairment. ^∧Kruskal-Wallis H test; °Chi-squared test.

Conventional and non-conventional neuropsychological measures results

Table 2 reports summary statistics and the three groups’ comparison of the assessment of the cognitive functions through the battery of standard neuropsychological tools. Compared to HC, both MCI and VCI showed a reduction in global cognitive function, as assessed by MoCA test (p < 0.001) and in tasks assessing memory (IFR, ITR, and DTR; p < 0.001). Moreover, in the DFR task only MCI showed a reduced performance compared to HC (DFR, p < 0.001). When evaluating executive function, both VCI and MCI did not differ from HC in TMTA, whereas only VCI but not MCI presented a reduced performance compared to HC in TMTB (p < 0.001).

Table 2

Standard neuropsychological assessment battery groups comparison results

	HC (Me, IR)	MCI (Me, IR)	VCI (Me, IR)	Cut-off (ES = 0)	Test statistic (H or F)	p DF = 2	Post-hoc comparison
MoCA	27.23,3.32	21.84,4.58	23.09,5.15	<17.36	57.32	<0.001^*	MCI < HC; VCI < HC
IFR	27.48,3.87	22.16,11.38	25.20,11.04	≤19.59	14.71	<0.001^*	MCI < HC; VCI < HC
ITR	36.00,0.00	36.00,2.00	36.00,2.00	<35	22.97	<0.001 ^∧	MCI < HC; VCI < HC
DFR	9.56,2.80	8.45,7.46	9.34,3.62	≤6.31	16.54	<0.001 ^∧	MCI < HC
DTR	12.00,0.00	12.00,1.00	12.00,1.00	<11	18.48	<0.001 ^∧	MCI < HC; VCI < HC
TMTA	40.00,35.25	38.00,35.00	39.00,25.75	>93	0.57	0.752^∧	–
TMTB	68.50,64.75	91.00,135.00	175.00,434.50	>283	15.90	<0.001 ^∧	VCI < HC

ES, Equivalent Score; F, ANOVA analysis; H, Kruskal-Wallis H test; HC, healthy control group; IR, interquartile range; Me, median; MCI, mild cognitive impairment group; MoCA, Montreal Cognitive Assessment; IFR, Immediate Free Recall; ITR, Immediate Total Recall; DFR, Delayed Free Recall; DTR, Delayed Total Recall; TMTA, Trail Making Test-Part A; TMTB, Trail Making Test-Part B; VCI, vascular cognitive impairment; ^∧Kruskal-Wallis H test, ^*ANOVA analysis; p < 0.05 were reported in bold. Post-hoc comparisons were adjusted with Bonferroni’s correction.

Comparison between the three groups on SASG composite and total scores were performed (Table 3). Results showed large significant differences among groups in all SASG task scores. In particular, both MCI and VCI differed significantly from HC in equal measure in T2 and T5 (p < 0.001). Instead, a lower VCI score compared to the MCI score was observed in T1 and T3 (p < 0.001). Furthermore, only MCI presented a lower score than HC in T4 (p = 0.001). Finally, we observed that both VCI and MCI differed significantly from HC in SASG total score (p < 0.001), with a lower performance of VCI than MCI. This trend was confirmed in the SASG-total-time score, while MCI and VCI performed equally lower than HC in SASG-total-acc.

Table 3

SASG’s groups comparison results

TASK - composite score (DOMAIN)	HC (Me, IR)	MCI (Me, IR)	VCI (Me, IR)	Test statistic (H or F)	p-value DF = 2	Post-hoc comparison
T1 (Memory/Spatial orientation/Attention)	0.07, 1.98	–1.35, 2.82	–2.95, 2.26	48.84	<0.001 ^∧	VCI < MCI < HC
T2 (Executive functions/ Divided attention)	0.17, 1.40	–0.59, 2.15	–1.35, 2.70	27.82	<0.001 ^∧	VCI < HC; MCI < HC
T3 (Executive functions/Selective attention/ Working memory/prospective memory)	0.43, 2.57	–1.74, 3.65	–3.04, 3.91	46.06	<0.001 ^∧	VCI < MCI < HC
T4 (Memory)	0.06, 1.83	–1.06, 3.39	–0.77, 2.14	13.63	0.001 ^∧	MCI < HC
T5 (Long-term memory/Spatial orientation/Attention)	0.87, 2.92	–2.51, 2.20	–2.90, 2.95	55.76	<0.001 ^∧	VCI < HC; MCI < HC
SASG-total	0.86, 6.92	–8.34, 8.91	–11.14, 11.71	46.14	<0.001 ^*	VCI < MCI < HC
SASG-total-time	–0.19, 5.28	4.28, 4.07	5.63, 2.79	45.73	<0.001 ^*	VCI > MCI > HC
SASG-total-acc	0.58, 3.66	–3.31, 6.52	–4.60, 8.05	48.12	<0.001 ^∧	VCI < HC; MCI < HC

Data reported are z-scores. HC, healthy control group; MCI, mild cognitive impairment group; SASG-total-acc, Smart Aging Serious Game accuracy; SASG-total-time, Smart Aging Serious Game time; SASG-total, Smart Aging Serious Game total; VCI, vascular cognitive impairment; ^∧Kruskal-Wallis H test. Post-hoc comparisons were adjusted with Bonferroni’s correction; ^*ANOVA analysis.

Diagnostic sensitivity and specificity of the Smart Aging Serious Game

ROC curve analysis was carried out to investigate the diagnostic sensitivity and specificity of the SASG and the conventional neuropsychological measures by discriminating between VCI and HC, and between MCI and HC (see Fig. 1). We registered a high diagnostic sensitivity and specificity of SASG and MoCA (in all cases AUC > 0.800) in detecting VCI versus HC and MCI versus HC condition (see Table 4). The paired-sample t-test showed a statistically significant difference under the ROC curves between SAGS and conventional neuropsychological measures including IFR, DFR, TMTA and TMTB (all p < 0.05). No statistical significance difference was observed between SASG and MoCA in both cases (VCI versus HC: p = 0.171; MCI versus HC: p = 0.064).

Fig. 1

ROC curve for the diagnostic sensitivity and specificity of SASG and conventional neuropsychological tests in three conditions: VCI versus HC, MCI versus HC.

Table 4

ROC curve results

State variable		Sensitivity	Specificity	AUC	SD	p (SASG)	Criterion value (J)
VCI-HC	SASG	82.43	92.59	0.925	0.027	–	–4.68 (0.713)
	MoCA	77.03	77.78	0.870	0.036	0.171	25.42 (0.548)
	IFR	78.38	55.56	0.653	0.073	< 0.001	25.38 (0.339)
	DFR	90.54	40.74	0.657	0.069	< 0.001	8.51 (0.313)
	TMTA	58.11	51.85	0.485	0.066	< 0.001	42.00 (0.100)
	TMTB	90.54	59.26	0.753	0.065	0.010	142.00 (0.498)
MCI-HC	SASG	77.03	73.33	0.817	0.036	–	–3.42 (0.504)
	MoCA	83.78	76.67	0.893	0.026	0.064	24.56 (0.605)
	IFR	79.73	61.67	0.735	0.047	0.113	25.27 (0.414)
	DFR	90.54	51.67	0.698	0.048	0.028	8.51 (0.422)
	TMTA	81.08	21.67	0.465	0.051	< 0.001	63.00 (0.027)
	TMTB	70.27	50.00	0.626	0.050	0.001	96.00 (0.203)

AUC, Area Under the Curve; CI, confidence interval; J, Younden index; DFR, Delayed Free Recall; IFR, Immediate Free Recall; MCI, mild cognitive impairment group; MoCA, Montreal Cognitive Assessment; TMT, Trial Making Test; SASG, Smart Aging Serious Game; SE, standard error; VCI, vascular cognitive impairment. P-values refers to a paired T-test between SASG and each pencil-and-paper conventional test.

Classification of healthy controls or patients

Classification analyses revealed excellent accuracy for the classification model built to identify HC versus VCI (RF: 90%; LR: 91%) and an acceptable to excellent accuracy for the model built to classify HC versus MCI (RF: 75%; LR: 87%). Regarding the importance of the variable in both RF classification models SASG performance was associated with the higher mean decrease of accuracy (see Fig. 2).

Fig. 2

Mean decrease in accuracy of the RF classification model considering conventional neuropsychological tests and SASG total scores to classify VCI versus HC (on the left) and MCI versus HC (on the right).

By adopting a LR approach, significant predictors able to discriminate between HC and the two mild neurocognitive disorders were SAGS (HC versus VCI: p < 0.001; HC versus MCI: p = 0.036) and MoCA (HC versus VCI: p = 0.012; HC versus MCI: p < 0.001).

Table 5 shows the predictive performances of RF and LR classification models in terms of Precision, Recall, F1 score and AUC. Precision was above 60%for each class of individuals.

Table 5(A)

A) Evaluation metrics of RF and LR classification models. B) Confusion matrices referred to the two classification models, values are reported as percentages (subjects count)

A	HC versus VCI		HC versus MCI
Measure	RF	LR	RF	LR
Precision	0.917	0.922	0.751	0.870
F1 score	0.899	0.940	0.750	0.887
AUC	0.923	0.965	0.758	0.925
Accuracy	0.900	0.911	0.750	0.873

Area Under Curve (AUC) is calculated for every class against all other classes. MCI, mild cognitive impairment; VCI, vascular cognitive impairment; RF, random forest; LR, logistic regression.

Table 5(B)

B		Predicted		Predicted			Predicted		Predicted
		VCI	HC	VCI	HC		MCI	HC	MCI	HC
Observed	VCI	74.07%(20)	25.93%(7)	77.78%(21)	22.22%(6)	MCI	85.00%(51)	15.00%(9)	83.33%(50)	16.67%(10)
	HC	1.35%(1)	98.65%(73)	4.05%(3)	95.95%(71)	HC	9.46%(7)	90.54%(67)	9.46%(7)	90.54%(67)

Table 6 shows estimates of the LR classification models based on conventional neuropsychological tests and SASG to classify VCI versus HC and MCI versus HC.

Table 6

Logistic regression model classifying HC versus VCI and HC versus MCI based on neuropsychological tests and SASG

HC versus VCI	β	Odds Ratio	Wald Statistic	df	p
(Intercept)	1.466	0.003	1.223	1	0.269
SASG	2.189	1.318	11.423	1	< 0.001
MoCA	1.306	1.49	4.382	1	0.036
IFR	0.058	1.012	0.01	1	0.921
DFR	–0.314	0.845	0.214	1	0.643
TMTA	–1.126	0.952	2.155	1	0.142
TMTB	3.101	1.022	3.781	1	0.052
HC versus MCI
(Intercept)	–0.127	0.000	20.345	1	< 0.001
SASG	0.954	1.133	6.307	1	0.012
MoCA	1.921	1.709	21.227	1	< 0.001
IFR	0.322	1.057	0.480	1	0.488
DFR	0.494	1.161	0.887	1	0.346
TMTA	–0.167	0.997	0.408	1	0.523
TMTB	0.671	1.006	1.917	1	0.166

df, degree of freedom; DFR, Delayed Free Recall; IFR, Immediate Free Recall; MCI, mild cognitive impairment group; MoCA, Montreal Cognitive Assessment; TMT, Trial Making Test; SASG, Smart Aging Serious Game; VCI, vascular cognitive impairment.

DISCUSSION

In the most recent years, the timely and precise diagnosis of conditions at risk of major neurocognitive disorders, such as MCI and VCI, has become of pivotal importance. Early rehabilitative interventions and longitudinal monitoring of the course of the disease can reduce the probability of developing major neurocognitive disorders such as dementia [18 , 54], increasing the quality of life in those individuals and families, and reducing social and sanitary costs. According to this premise, the present study aimed to test the validity of the SASG platform, in detecting mild neurocognitive disorders, such as MCI and VCI, from the successful aging.

Our results showed that SASG well captures the specific cognitive profiles of aMCI and VCI, in line with the standard neuropsychological measures. A prominent deficit in the pure memory task (T4) was observed in typical aMCI while slower performance times (SASG-total-time) were recognized in VCI. These results together with our ROC, LR, and RF findings highlighted the validity of SASG in discriminating between unsuccessful and successful aging, as a robust digital neuropsychological tool and support the evidence that SASG presents with an equal or higher diagnostic validity than conventional neuropsychological tests.

The integration of digital health tools in the standard clinical practice is a recent challenge. It has become increasingly clear that neuropsychology is experiencing a crisis considering that it has essentially remained stable over-time [55] and that several potential advantages can be afforded by greater integration of technology [56]. Indeed, traditionally neuropsychology adopts labor intensive methods and allows data collection in a slow, potentially inefficient, and expensive way with a relatively poor estimate of human behavior outside of laboratory conditions [57, 58]. During the last few years, there were significant innovative changes in the neuropsychological field, characterized by a review leading to a transformation from Neuropsychology 1.0 to 2.0 [59]. In this new framework, neuropsychologists evolved from the utilization of subjective measures of cognitive functioning (with restricted norms) to the adoption of validated assessment tools with more sophisticated psychometric properties and normative procedures. A large-scale initiative of harmonization of assessment procedures in the United States is the Uniform Data Set (UDS) promoted by the University of Washington’s National Alzheimer’s Coordinating Center (https://www.alz.washington.edu/web/forms_uds.html). This initiative was aimed at developing standardized methods and setting up a shared neuropsychological battery (UDS 3.0 neuropsychological battery) to capture the continuum of cognitive decline from normal cognition through AD and to create a normative calculator available online [60].

In line with this transformation, the integration of technologies in the neuropsychological assessment can represent a tremendous advantage in the collection of a great amount of data, with the novel challenge to integrate them into a cohesive patient profile. Accordingly, our innovative tool SASG, opens the perspective to register not only simplistic measures such as accuracy (operationalized in terms of number of corrected responses), but also additional quantitative features such as executive times, clicks counts, navigation tracking coordinates about movements in the virtual scenario, etc. All of these features could allow a more precise patient profile also in studies exploiting machine learning algorithms and big datasets [27 , 62], with the potential for better classification of clinical populations.

Our RF analyses revealed a higher precision of SASG in classifying MCI and VCI from HC. This finding could be explained by the different specificities that differentiate the digital tool from conventional neuropsychological tests. The ecological stimuli of the SASG scenario, presented with increased perceptual complexity associated with the VR-3D presentation, represent an enriched environment for the subject compared to conventional neuropsychological tests. This environment is harder to be learned and approached by the subject than pencil-and-paper tools and makes it more difficult to develop compensatory strategies able to “hide” subtle cognitive impairment [63, 64]. In terms of this graphical feature, multi-dimensional tasks were presented within this 3D environment, constituting functional-led measures of the cognitive domains in a daily life ecological context. The assessment in an ecologically-valid and real-life-mimicking setting could have assured advantages such as triggering a feeling of immersion and offering the closeness of daily life stimuli [65] overcoming the evaluation paradigm of the standard pencil-and-paper tests [33 , 67]. Accordingly, recent contributions revealed how virtual reality assessment tools result in a precise detection of people at risk of major neurocognitive disorders also thanks to their ecological validity [30]. Moreover, SASG presented a multi-dimensional cognitive assessment, fundamental in the evaluation of neurocognitive disorders, that can involve several of the six neurocognitive domains [6]. It is well demonstrated that functional-led tasks are higher sensitive in registering cognitive impairment [63, 64]. Accordingly, we reported a higher sensitivity and specificity of the digital tool if compared to single-domain conventional measures, such as FCSRT and TMT. Coherently, the sensitivity and specificity of SASG were equal to the MoCA test, screening different domains of cognition as SASG does.

Globally, our study opens to the integration of digital biomarkers in the detection of neurocognitive aging, adopting valid, non-invasive, and potentially self-administered methods [31]. In fact, the actual gold-standard biomarkers consist of amyloid-β deposition, tau protein accumulation, and/or brain neurodegeneration [23]. Although standard neuropsychological measures can recognize the presence of cognitive deficits suggesting an initial unsuccessful aging process, differential clusters of mild to major neurocognitive disorders are conventionally diagnosed only by recurring to standard biomarkers, such as the amyloid-β load and tau protein [68]. This procedure often requires expensive and invasive examination, and therefore it is not usually timely available, with critical consequences for the lack of prompt targeted interventions. Our contribution, in line with previous work [45], presented SASG as a promising non-invasive tool for the early detection of MCI and VCI conditions. Also, the SASG owns the prerequisites for self-administration out of the clinic, also at the patient’s home, in a real-life setting. Implications are utmost: a more democratic delivery of services, not affected by physical barriers, less costs and resources-consuming, availability of remote assessment out of the hospital. This application would allow to capture cognitive, and motor changes in daily life preceding the frank manifestation of major neurocognitive disorders [69]. The literature reported valid examples of digital systems for the monitoring of people’s functioning in a real-life setting, such as the measure obtained from smartphone use as a surrogate for laboratory-based neuropsychological assessment [25]. Further innovative evidence consists in the Altoida machine learning platform, an amply valid digital biomarker, combining data from hand movements, gait, posture, eye tracking, visuo-spatial navigation, voice, dual-task activities during daily living [27 , 62], to outline cognitive deficits before people recur to clinical investigations.

This study is not without limitations and some elements might be taken into consideration when planning future research using SASG. In our study, we focused on limited features of SASG available output data. Furthermore, we compared SASG validity only with standard pencil-and-paper measures without including a digital version of neuropsychological conventional tests. However, pencil-and-paper cognitive tests still represent the gold-standard of clinical neuropsychological measures. Moreover, SASG, similar to other digital devices, needs a session to allow for familiarization with the technology that extends the administration time. Nevertheless, it is plausible to assume that future generations of millennials will not need to familiarize themselves with digital devices anymore. Also, our VCI cohort consisted of a small group of patients, especially compared with the group of MCI subjects, preventing to explore the capacity of the SASG in differentiating between MCI and VCI conditions. Future studies need to replicate our findings with a wider sample of VCI.

To conclude, our results demonstrate that SASG is a valid ecologically-based tool for digital phenotyping in aging able to detect MCI and VCI from successful aging. Implications are utmost; in fact, efficient treatments for most neurodegenerative diseases are hindered by the fact that their detection intervenes at late stages when the integrity of the nervous tissue is very compromised. SASG allows for the early assessment of cognitive impairment through functional tasks and potentially in natural everyday life. These features make this tool suitable for being considered a digital candidate to assess and to monitor subjects with evident implications for future digital-health trails.

Footnotes

ACKNOWLEDGMENTS

The authors thank the GOAL Working Group (alphabetical order): Valeria Blasi¹, Francesca Borgnis¹, Monica Di Cesare¹, Camilla Ferrari², Tommaso Migliazza³, Chiara Pagliari¹, Federica Rossetto¹, Federica Savazzi¹, Andrea Stoppini³. ¹IRCCS Fondazione Don Carlo Gnocchi ONLUS; ²Universitá degli Studi di Firenze, Dipartimento di Neuroscienze, Psicologia, Area del Farmaco e Salute del Bambino, Firenze, Italy; ³Consorzio di Bioingegneria e Informatica medica–CBIM, Pavia, Italy.

This research was funded by the ITALIAN MINISTRY OF HEALTH, Ricerca Corrente and by BANDO FAS SALUTE 2014 from the Tuscany Region (Italy).

Authors’ disclosures available online ().

References

United Nations, Department of Economic and Social Affairs (2019) World Population Prospects 2019, Highlights.

World Health Organization (2017) Global Strategy and Action Plan on Ageing and Health.

Cabeza

, Albert

, Belleville

, Craik

FIM

, Duarte

, Grady

, Lindenberger

, Nyberg

, Park

, Reuter-Lorenz

, Rugg

, Steffener

, Rajah

(2018) Maintenance, reserve and compensation: The cognitive neuroscience of healthy ageing. Nat Rev Neurosci 19, 701–710.

Davis

, Dennis

, Daselaar

, Fleck

, Cabeza

(2008) Que PASA? The posterior-anterior shift in aging. Cereb Cortex 18, 1201–1209.

Reuter-Lorenz

, Cappell

(2008) Neurocognitive aging and the compensation hypothesis. Curr Dir Psychol Sci 17, 177–182.

Sachdev

, Blacker

, Blazer

, Ganguli

, Jeste

, Paulsen

, Petersen

(2014) Classifying neurocognitive disorders: The DSM-5 approach. Nat Rev Neurol 10, 634–642.

American Psychiatric Association (2013) Diagnostic and Statistical Manual of Mental Disorders (DSM V).

Iadecola

, Duering

, Hachinski

, Joutel

, Pendlebury

, Schneider

, Dichgans

(2019) Vascular cognitive impairment and dementia: JACC Scientific Expert Panel. J Am Coll Cardiol 73, 3326–3344.

World Health Organization (2012) Dementia. A public health priority.

10.

Luck

, Then

, Schroeter

, Witte

, Engel

, Loeffler

, Thiery

, Villringer

, Riedel-Heller

(2017) Prevalence of DSM-5 mild neurocognitive disorder in dementia-free older adults: Results of the population-based LIFE-Adult-Study. Am J Geriatr Psychiatry 25, 328–339.

11.

Petersen

, Lopez

, Armstrong

, Getchius

TSD

, Ganguli

, Gloss

, Gronseth

, Marson

, Pringsheim

, Day

, Sager

, Stevens

, Rae-Grant

(2018) Practice guideline update summary: Mild cognitive impairment: Report of the Guideline Development, Dissemination, and Implementation Subcommittee of the American Academy of Neurology. Neurology 90, 126–135.

12.

Stokin

, Krell-Roesch

, Petersen

, Geda

(2015) Mild neurocognitive disorder: An old wine in a new bottle. Harv Rev Psychiatry 23, 368–376.

13.

Albert

, DeKosky

, Dickson

, Dubois

, Feldman

, Fox

, Gamst

, Holtzman

, Jagust

, Petersen

, Snyder

, Carrillo

, Thies

, Phelps

(2011) The diagnosis of mild cognitive impairment due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimers Dement 7, 270–279.

14.

Mancuso

, Cabinio

, Agostini

, Baglio

, Clerici

(2020) HSV-1-specific IgG3 titers correlate with brain cortical thinning in individuals with mild cognitive impairment and Alzheimer’s disease. Vaccines (Basel) 8, 255.

15.

Cabinio

, Saresella

, Piancone

, LaRosa

, Marventano

, Guerini

, Nemni

, Baglio

, Clerici

(2018) Association between hippocampal shape, neuroinflammation, and cognitive decline in Alzheimer’s disease. J Alzheimers Dis 66, 1131–1144.

16.

van der Flier

, Skoog

, Schneider

, Pantoni

, Mok

, Chen

CLH

, Scheltens

(2018) Vascular cognitive impairment. Nat Rev Dis Primers 4, 18003.

17.

Vasquez

, Zakzanis

(2015) The neuropsychological profile of vascular cognitive impairment not demented: A meta-analysis. J Neuropsychol 9, 109–136.

18.

Livingston

, Sommerlad

, Orgeta

, Costafreda

, Huntley

, Ames

, Ballard

, Banerjee

, Burns

, Cohen-Mansfield

, Cooper

, Fox

, Gitlin

, Howard

, Kales

, Larson

, Ritchie

, Rockwood

, Sampson

, Samus

, Schneider

, Selbaek

, Teri

, Mukadam

(2017) Dementia prevention, intervention, and care. Lancet 390, 2673–2734.

19.

Fabbri

, Mosca

, Gerli

, Martini

, Pancani

, Lucidi

, Savazzi

, Baglio

, Vannetti

, Macchi

, GOAL Working Group (2019) The Games for Older Adults Active Life (GOAL) project for people with mild cognitive impairment and vascular cognitive impairment: A study protocol for a randomized controlled trial. Front Neurol 9, 1040.

20.

Suzuki

, Shimada

, Makizako

, Doi

, Yoshida

, Ito

, Shimokata

, Washimi

, Endo

, Kato

. (2013) A randomized controlled trial of multicomponent exercise in older adults with mild cognitive impairment. PLoS One 8, e61483.

21.

Nagamatsu

, Handy

, Hsu

, Voss

, Liu-Ambrose

(2012) Resistance training promotes cognitive and functional brain plasticity in seniors with probable mild cognitive impairment. Arch Intern Med 172, 666–668.

22.

Andrieu

, Coley

, Lovestone

, Aisen

, Vellas

(2015) Prevention of sporadic Alzheimer’s disease: Lessons learned from clinical trials and future directions. Lancet Neurol 14, 926–944.

23.

McKhann

, Knopman

, Chertkow

, Hyman

, Jack

Jr , Kawas

, Klunk

, Koroshetz

, Manly

, Mayeux

, Mohs

, Morris

, Rossor

, Scheltens

, Carrillo

, Thies

, Weintraub

, Phelps

(2011) The diagnosis of dementia due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimers Dement 7, 263–269.

24.

Kourtis

, Regele

, Wright

, Jones

(2019) Digital biomarkers for Alzheimer’s disease: The mobile/ wearable devices opportunity. NPJ Digit Med 2, 9.

25.

Dagum

(2018) Digital biomarkers of cognitive function. NPJ Digit Med 1, 10.

26.

Bonnechere

, Langley

, Sahakian

(2020) The use of commercial computerised cognitive games in older adults: A meta-analysis. Sci Rep 10, 15276.

27.

Buegler

, Harms

, Balasa

, Meier

, Exarchos

, Rai

, Boyle

, Tort

, Kozori

, Lazarou

, Rampini

, Cavaliere

, Vlamos

, Tsolaki

, Babiloni

, Soricelli

, Frisoni

, Sanchez-Valle

, Whelan

, Merlo-Pich

, Tarnanas

(2020) Digital biomarker-based individualized prognosis for people at risk of dementia. Alzheimers Dement (Amst) 12, e12073.

28.

Vallejo

, Wyss

, Rampa

, Mitache

, Muri

, Mosimann

, Nef

(2017) Evaluation of a novel Serious Game based assessment tool for patients with Alzheimer’s disease. PLoS One 12, e0175999.

29.

Diaz-Orueta

, Blanco-Campal

, Lamar

, Libon

, Burke

(2020) Marrying past and present neuropsychology: Is the future of the process-based approach technology-based? Front Psychol 11, 361.

30.

Slyk

, Zarzycki

, Kocwa-Karnas

, Domitrz

(2019) Virtual reality in the diagnostics and therapy of neurological diseases. Expert Rev Med Devices 16, 1035–1040.

31.

Babrak

, Menetski

, Rebhan

, Nisato

, Zinggeler

, Brasier

, Baerenfaller

, Brenzikofer

, Baltzer

, Vogler

, Gschwind

, Schneider

, Streiff

, Groenen

PMA

, Miho

(2019) Traditional and digital biomarkers: Two worlds apart? Digit Biomark 3, 92–102.

32.

Zucchella

, Sinforiani

, Tassorelli

, Cavallini

, Tost-Pardell

, Grau

, Pazzi

, Puricelli

, Bernini

, Bottiroli

, Vecchi

, Sandrini

, Nappi

(2014) Serious games for screening pre-dementia conditions: From virtuality to reality? A pilot project. Funct Neurol 29, 153–158.

33.

Serino

, Baglio

, Rossetto

, Realdon

, Cipresso

, Parsons

, Cappellini

, Mantovani

, De Leo

, Nemni

, Riva

(2017) Picture Interpretation Test (PIT) 360 degrees: An innovative measure of executive functions. Sci Rep 7, 16000.

34.

Robert

, Konig

, Amieva

, Andrieu

, Bremond

, Bullock

, Ceccaldi

, Dubois

, Gauthier

, Kenigsberg

, Nave

, Orgogozo

, Piano

, Benoit

, Touchon

, Vellas

, Yesavage

, Manera

(2014) Recommendations for the use of Serious Games in people with Alzheimer’s Disease, related disorders and frailty. Front Aging Neurosci 6, 54.

35.

Dubois

, Feldman

, Jacova

, Hampel

, Molinuevo

, Blennow

, DeKosky

, Gauthier

, Selkoe

, Bateman

, Cappa

, Crutch

, Engelborghs

, Frisoni

, Fox

, Galasko

, Habert

, Jicha

, Nordberg

, Pasquier

, Rabinovici

, Robert

, Rowe

, Salloway

, Sarazin

, Epelbaum

, de Souza

, Vellas

, Visser

, Schneider

, Stern

, Scheltens

, Cummings

(2014) Advancing research diagnostic criteria for Alzheimer’s disease: The IWG-2 criteria. Lancet Neurol 13, 614–629.

36.

Lamargue-Hamel

, Deloire

, Saubusse

, Ruet

, Taillard

, Philip

, Brochet

(2015) Cognitive evaluation by tasks in a virtual reality environment in multiple sclerosis. J Neurol Sci 359, 94–99.

37.

Ouellet

, Boller

, Corriveau-Lecavalier

, Cloutier

, Belleville

(2018) The Virtual Shop: A new immersive virtual reality environment and scenario for the assessment of everyday memory. J Neurosci Methods 303, 126–135.

38.

Kourtesis

, Collina

, Doumas

LAA

, MacPherson

(2021) Validation of the Virtual Reality Everyday Assessment Lab (VR-EAL): An immersive virtual reality neuropsychological battery with enhanced ecological validity. J Int Neuropsychol Soc 27, 181–196.

39.

Negut

, Matu

, Sava

, David

(2016) Virtual reality measures in neuropsychological assessment: A meta-analytic review. Clin Neuropsychol 30, 165–184.

40.

Riva

, Mancuso

, Cavedoni

, Stramba-Badiale

(2020) Virtual reality in neurorehabilitation: A review of its effects on multiple cognitive domains. Expert Rev Med Devices 17, 1035–1061.

41.

Matamala-Gomez

, Maisto

, Montana

, Mavrodiev

, Baglio

, Rossetto

, Mantovani

, Riva

, Realdon

(2020) The role of engagement in teleneurorehabilitation: A systematic review. Front Neurol 11, 354.

42.

Manenti

, Gobbi

, Baglio

, Macis

, Ferrari

, Pagnoni

, Rossetto

, Di Tella

, Alemanno

, Cimino

, Binetti

, Iannaccone

, Bramanti

, Cappa

, Cotelli

(2020) Effectiveness of an innovative cognitive treatment and telerehabilitation on subjects with mild cognitive impairment: A multicenter, randomized, active-controlled study. Front Aging Neurosci 12, 585988.

43.

Bottiroli

, Tassorelli

, Lamonica

, Zucchella

, Cavallini

, Bernini

, Sinforiani

, Pazzi

, Cristiani

, Vecchi

, Tost

, Sandrini

(2017) Smart Aging Platform for evaluating cognitive functions in aging: A comparison with the MoCA in a normal population. Front Aging Neurosci 9, 1–14.

44.

Bottiroli

, Bernini

, Cavallini

, Sinforiani

, Zucchella

, Pazzi

, Cristiani

, Vecchi

, Tost

, Sandrini

, Tassorelli

(2021) The Smart Aging Platform for assessing early phases of cognitive impairment in patients with neurodegenerative diseases. Front Aging Neurosci 12.

45.

Cabinio

, Rossetto

, Isernia

, Saibene

, Di Cesare

, Borgnis

, Pazzi

, Migliazza

, Alberoni

, Blasi

, Baglio

(2020) The use of a virtual reality platform for the assessment of the memory decline and the hippocampal neural injury in subjects with mild cognitive impairment: The validity of Smart Aging Serious Game (SASG). J Clin Med 9, 1355.

46.

Hachinski

, Iadecola

, Petersen

, Breteler

, Nyenhuis

, Black

, Powers

, DeCarli

, Merino

, Kalaria

, Vinters

, Holtzman

, Rosenberg

, Wallin

, Dichgans

, Marler

, Leblanc

(2006) National Institute of Neurological Disorders and Stroke-Canadian Stroke Network vascular cognitive impairment harmonization standards. Stroke 37, 2220–2241.

47.

Folstein

, Folstein

, McHugh

(1975) “Mini-mental state". A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 12, 189–198.

48.

Fazekas

, Barkhof

, Wahlund

, Pantoni

, Erkinjuntti

, Scheltens

, Schmidt

(2002) CT and MRI rating of white matter lesions. Cerebrovasc Dis 13 Suppl 2, 31–36.

49.

Conti

, Bonazzi

, Laiacona

, Masina

, Coralli

(2015) Montreal Cognitive Assessment (MoCA)-Italian version: Regression based norms and equivalent scores. Neurol Sci 36, 209–214.

50.

Frasson

, Ghiretti

, Catricala

, Pomati

, Marcone

, Parisi

, Rossini

, Cappa

, Mariani

, Vanacore

, Clerici

(2011) Free and Cued Selective Reminding Test: An Italian normative study. Neurol Sci 32, 1057–1062.

51.

Giovagnoli

, Del Pesce

, Mascheroni

, Simoncelli

, Laiacona

, Capitani

(1996) Trail making test: Normative values from 287 normal adult controls. Ital J Neurol Sci 17, 305–309.

52.

Mandrekar

(2010) Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol 5, 1315–1316.

53.

Gates

, Valenzuela

(2010) Cognitive exercise and its role in cognitive function in older adults. Curr Psychiatry Rep 12, 20–27.

54.

Woods

, Aguirre

, Spector

, Orrell

(2012) Cognitive stimulation to improve cognitive functioning in people with dementia. Cochrane Database Syst Rev CD005562.

55.

Rabin

, Spadaccini

, Brodale

, Grant

, Elbulok-Charcape

, Barr

(2014) Utilization rates of computerized tests and test batteries among clinical neuropsychologists in the United States and Canada. Prof Psychol Res Pract 45, 368–377.

56.

Miller

, Barr

(2017) The technology crisis in neuropsychology. Arch Clin Neuropsychol 32, 541–554.

57.

Collins

, Riley

(2016) NIH’s transformative opportunities for the behavioral and social sciences.366ed. Sci Transl Med 8, 14.

58.

Leurent

, Ehlers

(2015) Digital technologies for cognitive assessment to accelerate drug development in Alzheimer’s disease. Clin Pharmacol Ther 98, 475–476.

59.

Bilder

(2011) Neuropsychology 3.0: Evidence-based science and practice. J Int Neuropsychol Soc 17, 7–13.

60.

Weintraub

, Besser

, Dodge

, Teylan

, Ferris

, Goldstein

, Giordani

, Kramer

, Loewenstein

, Marson

, Mungas

, Salmon

, Welsh-Bohmer

, Zhou

, Shirk

, Atri

, Kukull

, Phelps

, Morris

(2018) Version 3 of the Alzheimer Disease Centers’ Neuropsychological Test Battery in the Uniform Data Set (UDS). Alzheimer Dis Assoc Disord 32, 10–17.

61.

Rai

, Boyle

, Brosnan

, Rice

, Farina

, Tarnanas

, Whelan

(2020) Digital biomarkers based individualized prognosis for people at risk of dementia: The AltoidaML Multi-site External Validation Study. Adv Exp Med Biol 1194, 157–171.

62.

Tarnanas

, Tsolaki

, Wiederhold

, Tsolaki

(2015) Five-year biomarker progression variability for Alzheimer’s disease dementia prediction: Can a complex instrumental activities of daily living marker fill in the gaps? Alzheimers Dement (Amst) 1, 521–532.

63.

Bonnet

, Allard

, Dilharreguy

, Deloire

, Petry

, Brochet

(2010) Cognitive compensation failure in multiple sclerosis. Neurology 75, 1241–1248.

64.

Bonato

, Deouell

(2013) Hemispatial neglect: Computer-based testing allows more sensitive quantification of attentional disorders and recovery and might lead to better evaluation of rehabilitation. Front Hum Neurosci 7, 162.

65.

Cipresso

, Giglioli

IAC

, Raya

, Riva

(2018) The past, present, and future of virtual and augmented reality research: A network and cluster analysis of the literature. Front Psychol 9, 2086.

66.

Schultheis

, Himelstein

, Rizzo

(2002) Virtual reality and neuropsychology: Upgrading the current tools. J Head Trauma Rehabil 17, 378–394.

67.

Realdon

, Serino

, Savazzi

, Rossetto

, Cipresso

, Parsons

, Cappellini

, Mantovani

, Mendozzi

, Nemni

, Riva

, Baglio

(2019) An ecological measure to screen executive functioning in MS: The Picture Interpretation Test (PIT) 360 degrees. Sci Rep 9, 5690019422011.

68.

Michaud

, Su

, Siahpush

, Murman

(2017) The risk of incident mild cognitive impairment and progression to dementia considering mild cognitive impairment subtypes. Dement Geriatr Cogn Dis Extra 7, 15–29.

69.

Sperling

, Mormino

, Johnson

(2014) The evolution of preclinical Alzheimer’s disease: Implications for prevention trials. Neuron 84, 608–622.