Tele-neuropsychological multidomain assessment in Italian people with cognitive disorders: Reliability and user satisfaction

Abstract

Background

Tele-neuropsychology has already been employed in neurocognitive disorders, however, in Italy, the evidence of its psychometric quality and satisfaction is still limited.

Objective

This study aimed to: (1) evaluate the reliability of a standardized battery of neuropsychological screening and domain-specific tests delivered at home via videoconference to a sample of Italian people with cognitive disorders, compared with traditional face-to-face administration; (2) assess the feasibility and satisfaction about remote administration.

Methods

This crossover study enrolled patients with subjective cognitive disorder, mild neurocognitive disorder, or dementia. All participants performed a brief neuropsychological screening assessment in face-to-face or remote mode. People with milder cognitive disorders (Mini-Mental State Examination ≥ 20) also performed an extended neuropsychological battery. After 15 days, each participant repeated the same assessment in the opposite mode. Finally, participants completed a satisfaction questionnaire.

Results

114 participants were initially enrolled in the study and 100 individuals performed both face-to-face and remote assessments (14 dropouts). All neuropsychological tests showed agreement between the two modes of administration, except for the Digit Span Forward (p = 0.009) and oral-Symbol Digit Modalities Test (p < 0.001). According to Lin's concordance correlation coefficient (LCCCs), tests also showed good or excellent reliability (LCCCs between 0.609 and 0.964); only the Digit Span Backward and the Stroop test showed moderate reliability (LCCCs =0.514–0.441, respectively). Among 100 participants, 65 patients answered the satisfaction questionnaire, declaring high satisfaction for the remote evaluation.

Conclusions

Tele-neuropsychology seems to work with in-home assessments among Italian patients with cognitive disorders and seems to be acceptable to them.

Trial registration

ClinicalTrials.gov, Identifier: NCT06078332 (October 17, 2023).

Keywords

Alzheimer's disease cognitive impairment dementia neuropsychological assessment tele-neuropsychology telemedicine videoconferencing

Introduction

The number of people living with cognitive disorders is significantly increasing worldwide due to the rapid growth of elderly population. Current projections show that in Italy by 2025 there will be 1.5 million people with dementia,¹ with higher costs and a disruptive impact on health systems and patients’ families.² Therefore, it is necessary to adopt innovative solutions to expand access to dementia care, promoting early diagnosis as well as treatment, care, and support throughout the course of the disease. Increasing evidence suggests that telemedicine can be useful to meet these needs.^3–6

As a part of telemedicine, tele-neuropsychology (TNP) uses telecommunication tools to deliver neuropsychological assessments, supporting clinical practice in the diagnosis and treatment of people with cognitive disorders.⁷ Recent TNP guidelines recommend maximizing the reproduction of traditional face-to-face (FTF) assessment, particularly through videoconferencing.^3,8–10

The two main ways in which TNP services are delivered are (1) in-clinic and (2) at home¹¹: in the first model, both the clinician and the patient are in a clinic but in separate rooms using videoconferencing methods to conduct the neuropsychological assessment. The rooms may be in different clinic locations (e.g., the patient in a satellite clinic in a rural area and the neuropsychologist in a main clinic in a city center), or in the same building; in the latter case the staff member may be inside the patient's room to assist the patient with test stimuli and procedures or outside the patient's room, available to troubleshoot. In the second model, the neuropsychological assessment is conducted with the patient in their own home or other remote location via audio or videoconferencing technology.^7,9

Several studies have demonstrated that neuropsychological assessment of people with cognitive disorders may be performed remotely, with high psychometric validity, diagnostic quality, and satisfaction. In particular, the validity and reliability of TNP have been established in adults with and without cognitive impairment,^12–20 and it has been shown that videoconference-based assessment distinguishes between cognitively impaired and non-impaired individuals like a FTF assessment.²¹ Also, TNP appears to be well accepted, with a 98% satisfaction rate.^22–27 Maximization of patient accessibility (individuals who may have restricted mobility or live long distances from the clinic), reduction of waiting times and unnecessary transportation, safety (reduction of infection risk) and equitability (overcoming geographical barriers) are described among the main advantages of TNP.^7,28 TNP also reduces costs, ensures more frequent monitoring, and preserves the caregiver time.²⁹

In the face of a growing international literature on the use of TNP, despite the well-known relevance of socio-demographic, cultural and language differences in neuropsychological testing that require a country/language specific validation,³⁰ evidence about psychometric properties, clinical usability, and experimental applications of Italian TNP tools is scarce, as most studies involved English-speaking people. In this regard, a recent systematic review by the Italian working group on tele-neuropsychology (TELA) identified the areas worthy of further investigations, highlighting the lack of studies on remotely administered domain-specific tests (more attention has been paid to screening tools) as well as the paucity of studies that adopted a videoconference approach (phone represented the most frequent method for remote administration, with obvious limitations on the tests that were used).³¹

Moreover, most of validity studies took place in clinic, likely to reduce the variability inherent the uncontrolled home-environment;^32,33 only one study supported the validity of a comprehensive neuropsychological assessment administered to people with and without cognitive impairment in a real home-to-home scenario.³⁴ The few other prospective studies that explored the reliability of in-home TNP using counterbalancing methodology, involved different neurological population, namely adults with multiple sclerosis,³⁵ adults who have had a stroke,³⁶ or pediatric patients.^37,38 No study exploring the validity of in-home TNP has been conducted on the Italian population suffering from cognitive disturbances, although the in-home approach embodies the main advantage of TNP, that is the possibility of not moving from one's home, a particularly important requirement for elderly people, who often have medical comorbidities that make travelling difficult.

Based on these considerations, the main aim of this study was to evaluate the reliability of a standardized battery of neuropsychological screening and domain-specific tests delivered at home by videoconference to a sample of Italian people with cognitive disorders compared with traditional FTF administration, by means of a crossover design. Considering the characteristics of the target population (elderly, suffering with possible cognitive disorders, probably not very tech savvy^39–42), the secondary aim was to assess the feasibility and the satisfaction level of patients and/or caregivers about the remote assessment (RMT).

Methods

Participants

The study enrolled consecutive patients referred to the Center for Cognitive Disorders and Dementia (CCDD), Section of Neurology of the University Hospital of Verona, between December 2021 and December 2023.

The inclusion criteria were: (1) Diagnosis of Major Neurocognitive Disorder, Mild Neurocognitive Disorder according to the Diagnostic and Statistical Manual of Mental Disorders-5 (DSM-5) criteria, or Subjective Cognitive Disorder;⁴³ (2) Availability of a device equipped with camera, microphone, and internet connection; (3) Presence of a caregiver for technical support; (4) Written informed consent signed by the patient and by the caregiver or the legal representative.

The exclusion criteria were: (1) Severe sensory deficits; (2) Non-native Italian speakers; (3) Primary Psychiatric Disorder; (4) Mini-Mental State Examination (MMSE)⁴⁴ score less than 10; (5) Severe Behavior and Psychological Symptoms of Dementia limiting collaboration; (6) Aphasia.

Participation in the study was completely voluntary and always revocable; participants had the right to withdraw from the trial at any time, without explanation needed, and with no negative effects on the quality of care or treatment.

The study was approved by the local Ethics Committee (Comitato Etico per la Sperimentazione Clinica delle Province di Verona e Rovigo; protocol number 3502CESC) and was conducted in accordance with the Helsinki Declaration. The trial was registered with ClinicalTrials.gov, NCT06078332.

Study design and procedures

This study was conducted as a single center, randomized, crossover trial.

The recruitment took place during the usual outpatient clinical examination. People who met the eligibility criteria and agree to participate were randomly assigned to the group Remote assessment first (RMT_First) or to the group Face-to-Face assessment first (FTF_First). Randomization was performed by means of a simple blocked randomization list generated by an online randomization service in Sealed envelope^TM (free online at: https://www.sealedenvelope.com/simple-randomiser/v1/lists).

All participants underwent a screening assessment of global cognition and level of autonomy in daily life (MMSE, Activities of Daily Living, Instrumental Activities of Daily Living), in FTF or RMT mode. According to our usual clinical practice, a standardized battery of domain-specific neuropsychological tests aimed to investigate different cognitive areas (i.e., memory, attention, executive functions, etc.) was administered only to people with subjective or mild cognitive impairment (MMSE ≥20) (T0). After 15 days (T1), each participant repeated the same assessment in the opposite mode: the two evaluations were carried out by two different psychologists. At T1 participants completed a feasibility and satisfaction questionnaire about the remote neuropsychological assessment.

The FTF assessment was carried out in the neuropsychologist's room, according to the usual procedures of good clinical practice. For the RMT assessment patients were submitted to the videoconference-based evaluation directly at their home: to support the ecological nature of our study, the participants were free to use the device they preferred to perform the videocall (computer, tablet, or phone). A few days before the appointment, an email with a link to the videoconference was sent to the patients by the neuropsychologist through the telemedicine platform Virtual Care-POHEMA, a medical device developed by Gruppo per l’Informatica-GPI. An additional link to confirm the device's appropriateness (audio-video quality, and connection stability) was sent to guarantee the successful completion of the videoconference. Examiners viewed participants on a 15.6” LCD Monitor with integrated HD camera. RMT administration was performed in the same way of the traditional FTF administration; visual stimuli were displayed on the video using the share screen option. For MMSE (writing and drawing tasks), Clock Drawing Test, and Praxis test, the participants were asked to write and draw on a blank sheet and then show it to the camera. For MMSE 3-stage command and the item “prehension behavior” of the Frontal Assessment Battery, the caregiver was involved. In order to minimize the impact of the caregiver's presence on test performance, the caregiver was present near the patient to start the videoconference and at the beginning of the evaluation when the two tests requiring his/her intervention were administered (in this phase we specified to the caregiver to intervene only when requested). During the administration of the other tests, the caregiver was asked to move away, even moving into a nearby room, from which however he/she could notice any connection problems and intervene if needed.

The time required in both conditions (FTF and RMT) was also measured.

Measures

The neuropsychological assessment included the Italian neuropsychological tests most widely used to evaluate cognitive functions in adults with cognitive disorders and consisted of the following tools:

Neuropsychological brief screening tools. The brief screening assessment included tools for the evaluation of global cognition (MMSE)[34] and the level of independence in basic self-care functions (Activities of Daily Living scale-ADL)⁴⁵ and in more complex activities (termed “instrumental activities”) (Instrumental Activities of Daily Living scale-IADL)⁴⁶ necessary for functioning in community settings.

Neuropsychological domain-specific tools. The neuropsychological battery assessed the following cognitive domains: (1) verbal learning and memory: Rey Auditory Verbal Learning Test (RAVLT),⁴⁷ Forward and Backward Digit Span (DS) test;⁴⁸ (2) attention: Oral version of Symbol Digit Modalities Test (oral SDMT);⁴⁹ (3) executive functions: Frontal Assessment Battery (FAB),⁵⁰ Clock Drawing Test (CDT),⁵¹ Stroop Colour and Word Test (SCWT);⁵² (4) phonemic verbal fluency⁴⁷ and semantic verbal fluency;⁵³ (5) visuo-constructive abilities: Constructional Praxis test (CPT).⁵⁴

For the Rey Auditory Verbal Learning Test (RAVLT)⁴⁷ parallel test forms were used at the second assessment.

Feasibility and satisfaction questionnaire. The questionnaire was developed ad hoc and it consisted of 14 six-point Likert Scale questions (from Completely Agree to Completely Disagree) and an open-ended question (feedback or suggestions). A total score was not calculated. Each item of the survey was considered individually.

At the end of the RMT evaluation, the satisfaction questionnaire was sent to the users using the same email to which link for the remote connection was sent; caregivers were encouraged to involve the patient in answering the questionnaire in order to capture the impression of the main user.

Statistical analysis

All the clinical variables were reported using descriptive statistics: mean values, standard deviation (SD) and minimum/maximum (Min-Max) were used to summarize the quantitative variables, while frequencies and percentages (%) were used to describe the categorical variables.

For neuropsychological measures, age-, gender-, and education-corrected scores were calculated from the raw scores according to Italian population-based norms and only corrected scores were included in the statistical analysis.

Kolmogorov-Smirnov test was used to evaluate the normality of data distributions to choose the appropriate statistical test: as the data were not normally distributed, the comparison of the scores of each neuropsychological test obtained in the two administration modalities (FTF versus RMT) was conducted by means of the Wilcoxon paired sample test. For statistically significant differences, the effect size was calculated and classified as small (d = 0.2), medium (d = 0.5), and large (d ≥ 0.8), according to Cohen.⁵⁵

To assess the reliability of each neuropsychological test, the scores obtained in the FTF and RMT modalities were analyzed by means of the Lin's concordance correlation coefficient (LCCC).^56,57 According to Altman,⁵⁸ LCCCs should be interpreted similarly to the Pearson's correlation coefficient as follows: values < 0.2 poor, between 0.2 and 0.399 weak, between 0.40 and 0.599 moderate, between 0.60 and 0.799 good and > 0.80 excellent.

Graphic comparisons were also made using the method described by Bland and Altman.⁵⁹

Regarding the feasibility and satisfaction questionnaire, for each item the absolute and relative frequencies of participants’ responses were calculated for each value of the six-point Likert Scale.

Finally, data about the quality of internet connectivity and audio-video (good, medium, or poor) and the time to perform FTF or RMT assessment were also reported as frequencies or means, as appropriate.

The two-tailed significance level for all analyses was set at p ≤ 0.05.

Statistical analyses were conducted using the IBM Statistical Package for Social Sciences software for Windows v25.0 (IBM Corp., Armonk, NY).

Results

Sample characteristics

Among 403 people with cognitive disorders referred to the CCDD and assessed for eligibility, 114 met the inclusion criteria and were randomly assigned to the FTF_First group (n = 59) or to the RMT_First group (n = 55). In the FTF_First group 9 dropouts were recorded, due to: strong anxious state during the assessment (2), difficulties with internet connection (2), technical difficulties during the remote assessment (2), intercurrent clinical complications (2), not availability of the caregiver (1). In the RMT_First groups the dropouts were 5, due to: strong anxious state during the assessment (2), absence of the caregiver (2), intercurrent clinical complications (1). Therefore, the statistical analyses were performed on the 100 participants with cognitive disorders (50 in FTF_First group and 50 in RMT_First group) who completed both the FTF assessment and the RMT assessment, as shown in Figure 1.

Figure 1.

Flow-chart for study participants.

The mean (SD) age of the total sample was 74.08 (9.99) years, education 9.42 (3.95) years and disease duration 22.43 (27.71) months. The sample included 37 males and 63 females. According to the criteria of DSM-5, 50 participants were diagnosed with Major Neurocognitive Disorder (Major NCD) or dementia, 28 with Mild Neurocognitive Disorder (Mild NCD) and 22 with Subjective Cognitive Disorder (SCD). The socio-demographic and clinical characteristics of the whole sample and of the two groups separately (FTF_First and RMT_First) are summarized in Table 1.

Table 1.

Demographic and clinical characteristics of the whole sample and of the FTF_first and RMT_first groups individually.

		FTF_First group (n = 50)	RMT_First group (n = 50)	Total sample (n = 100)
Age	Mean (SD) [min-max]	74.82 (8.86) [45–89]	73.34 (11.04) [41–89]	74.08 (9.99) [41–89]
Sex
Male	N	16	21	37
Female	N	34	29	63
Education (y)	Mean (SD) [min-max]	9.14 (4.21) [2–18]	9.70 (3.68) [2–18]	9.42 (3.95) [2–18]
Disease duration (mo)	Mean (SD) [min-max]	22.78 (27.90) [3–108]	22.08 (27.80) [3–132]	22.43 (27.71) [3–132]
MMSE (T0)	Mean (SD) [min-max]	23.41 (4.55) [11.7–30]	24.41 (4.76) [13.3–30]	23.91 (4.66) [11.7–30]
Diagnosis
Major NCD	N	25	25	50
Mild NCD	N	14	14	28
SCD	N	11	11	22
Dementia (n = 50)
AD	N	7	12	19
Vascular	N	6	6	12
Mixed	N	11	7	18
PD	N	1	0	1
Mild NCD (n = 28)
a-MCI	N	5	3	8
na-MCI	N	2	4	6
m-MCI	N	7	7	14

FTF: face to face; RMT: remote; SD: standard deviation; MMSE: Mini-Mental State Examination; NCD: neurocognitive disorder; SCD: subjective cognitive disorder; AD: Alzheimer's disease; PD: Parkinson's disease; a-MCI: amnesic mild cognitive impairment; na-MCI: non amnesic-mild cognitive impairment; m-MCI: multiple domain-mild cognitive impairment.

Comparing face-to-face and videoconference neuropsychological assessment scores

Wilcoxon Paired Sample revealed no significant differences between FTF and RMT scores for all neuropsychological tests (MMSE, ADL, IADL, RAVLT, DS Backward, Praxis test, CDT, FAB, Stroop test, and Phonemic and Semantic Fluency), except for the DS Forward test (p = 0.009, Cohen's d = 0.2) and the oral-SDMT (p < 0.001, Cohen's d = 0.3) (Table 2).

Table 2.

Comparison of neuropsychological tests scores obtained in the two administration modes (FTF/RMT).

TEST	N	FTF	RMT	p*
		Mean (SD)	Mean (SD)
MMSE	100	24.03 (4.51)	24.27 (4.55)	0.156
ADL	100	5.21 (1.16)	5.27 (1.15)	0.109
IADL	100	4.45 (2.43)	4.34 (2.49)	0.100
RAVLT immediate	75	39.43 (11.89)	39.56 (12.93)	0.947
RAVLT delayed	75	6.38 (5.05)	6.02 (5.36)	0.174
Forward DS	75	4.96 (0.89)	5.17 (0.77)	0.009* (d = 0.2)
Backward DS	75	3.44 (1.04)	3.58 (1.02)	0.231
Constructional praxis	74	12.82 (1.84)	12.88 (1.64)	0.755
CDT	75	7.09 (3.34)	7.33 (3.06)	0.348
FAB	75	12.26 (3.30)	12.44 (2.95)	0.458
Phonemic Fluency	75	27.72 (9.88)	26.63 (9.70)	0.170
Semantic Fluency	75	31.37 (10.99)	32.29 (11.12)	0.085
Stroop errors	70	3.85 (6.61)	2.81 (5.95)	0.066
Stroop time	70	31.76 (23.26)	32.39 (29.24)	0.691
Oral-SDMT	63	42.11 (13.63)	38.51 (10.96)	<0.001* (d = 0.3)

FTF: face to face; RMT: remote; SD: standard deviation; MMSE: Mini-Mental State Examination; ADL: Activities of Daily Living; IADL: Instrumental Activities of Daily Living; RAVLT: Rey Auditory Verbal Learning Test; DS: Digit Span; CDT: Clock Drawing Test; FAB: Frontal Assessment Battery; SDMT: Symbol Digit Modalities Test.

According to Altman,[48] most of the neuropsychological tests had good or excellent reliability (LCCCs between 0.609 and 0.964). The DS Backward test and the Stroop test (Time score), showed moderate reliability (LCCCs =0.514–0.441, respectively) (Table 3).

Table 3.

Lin's concordance correlation coefficients (LCCCs) between remote and face to face neuropsychological test scores.

Test	N	LCCC	95% CI	Level of Reliability
MMSE	100	0.925	0.891–0.949	Excellent
ADL	100	0.946	0.921–0.963	Excellent
IADL	100	0.964	0.947–0.976	Excellent
RAVLT immediate	75	0.818	0.727–0.880	Excellent
RAVLT delayed	75	0.908	0.860–0.941	Excellent
Forward DS	75	0.644	0.498– 0.755	Good
Backward DS	75	0.514	0.328–0.661	Moderate
Constructional Praxis	74	0.609	0.445–0.733	Good
CDT	75	0.686	0.547–0.789	Good
FAB	75	0.786	0.683–0.858	Good
Phonemic Fluency	75	0.816	0.724–0.879	Excellent
Semantic Fluency	75	0.923	0.881–0.951	Excellent
Stroop errors	70	0.651	0.496–0.765	Good
Stroop time	70	0.441	0.239– 0.607	Moderate
Oral-SDMT	63	0.827	0.743–0.885	Excellent

LCCC: Lin's concordance correlation coefficient; 95%CI: 95% confidence interval; MMSE: Mini-Mental State Examination; ADL: Activities of Daily Living; IADL: Instrumental Activities of Daily Living; RAVLT: Rey Auditory Verbal Learning Test; DS: Digit Span; CDT: Clock Drawing Test; FAB: Frontal Assessment Battery; SDMT: Symbol Digit Modalities Test.

In Figure 2 are shown the Bland-Altman plots that compare the scores acquired in the FTF and RMT modalities. Scores of all tests showed agreement, except for two tests: DS Forward test and oral-SDMT. Specifically, the RMT modality provided better performances on the DS Forward test, and worse performances on the oral-SDMT.

Figure 2.

Bland–Altman plots representing agreement between remote and face-to-face scores. The y-axis shows the values of the difference between remote and face-to-face scores; the x-axis shows the mean of the two scores. The solid line represents the average difference. Dashed lines represent the upper and lower 95% limits of agreement.

Feasibility and satisfaction level about the remote assessment

The mostly used device to perform the videoconference-based neuropsychological assessment was the computer (82%), followed by the smartphone (15%), and the tablet (3%). In remote assessments good video (86%), audio (86%), and internet connection (81%) were reported. Conversely, in few cases, poor video (1%), audio (4%) and internet connection (3%) quality interfered with the remote assessment. The remaining cases had medium video (13%), audio (10%), and connection (16%) quality. 87 patients completed the assessment via videoconference with no interference by the caregiver; 13 had one or more interruptions by the caregiver during the remote assessment. The videoconference-based assessments required a longer time compared to in-person evaluations (61.85 ± 16.06 versus 59.95 ± 13.23 min, p = 0.038, Cohen's d = 0.1)

65 participants answered the satisfaction questionnaire: 83.1% (N = 54) strongly agreed that the remote assessment was advantageous because they didn’t have to go to the hospital; in most cases, this benefit was attributed to: (1) shorter waiting time (38.5%); (2) feeling secure and protected at home (33.8%); and (3) not having to move the patient (33.8%) or (4) spend a lot of time seeking for a parking area near the hospital (43.1%). Data from the satisfaction questionnaire are summarized in Table 4.

Table 4.

Absolute and relative frequencies of the answers for each item of the feasibility and satisfaction questionnaire about the remote neuropsychological assessment. The most frequent answer for each item is bolded.

	Remote assessment questionnaire Item	Agree			Disagree
	Remote assessment questionnaire Item	Strongly N (%)	Moderately N (%)	Slightly N (%)	Slightly N (%)	Moderately N (%)	Strongly N (%)
1	It was advantageous because I avoided going to the hospital	54 (83.1%)	9 (13.8%)	-	2 (3.1%)	-	-
2	I felt discomfort	8 (12.3%)	2 (3.1%)	1 (1.5%)	2 (3.1%)	8 (12.3%)	44 (67.7%)
3	I feared it was not as accurate	3 (4.6%)	4 (6.2%)	5 (7.7%)	5 (7.7%)	10 (15.4%)	38 (58.5%)
4	It was an intrusion on my private life	1 (1.5%)	2 (3.1%)	1 (1.5%)	1 (1.5%)	3 (4.6%)	57 (87.7%)
5	The psychologist-patient interaction was equally successfully	51 (78.5%)	4 (6.2%)	2 (3.1%)	5 (7.7%)	2 (3.1%)	1 (1.5%)
6	I got a clear explanation of the procedures	62 (95.4%)	2 (3.1%)	1 (1.5%)	-	-	-
7	I had technical difficulties	5 (7.7%)	9 (13.8%)	6 (9.2%)	1 (1.5%)	6 (9.2%)	38 (58.5%)
8	It was easy	45 (69.2%)	13 (20.0%)	2 (3.1%)	3 (4.6%)	1 (1.5%)	1 (1.5%)
9	I would suggest it for anyone who have my same condition	51 (78.5%)	10 (15.4%)	-	-	3 (4.6%)	1 (1.5%)
10	I was satisfied with the mode it was carried out	58 (89.2%)	5 (7.7%)	-	-	1 (1.5%)	1 (1.5%)
11	I felt less fatigued	29 (44.6%)	14 (21.5%)	8 (12.3%)	2 (3.1%)	3 (4.6%)	9 (13.8%)
12	I felt calmer during the remote assessment	27 (41.5%)	16 (24.6%)	11 (16.9%)	3 (4.6%)	2 (3.1%)	6 (9.2%)
13	It should be an additional mode to use in the future	48 (73.8%)	8 (12.3%)	3 (4.6%)	4 (6.2%)	1 (1.5%)	1 (1.5%)
14	Overall I was satisfied	57 (87.7%)	6 (9.2%)	1 (1.5%)	-	-	1 (1.5%)

Discussion

The results of this study suggest that the RMT administration of a standardized battery of screening and domain-specific neuropsychological tests is comparable to the standard FTF administration in Italian people with cognitive disorders. In-home assessment requires a similar amount of time and is well accepted by users.

Consistently with previous studies,^{13–15,18,60–62} MMSE showed agreement between the FTF and RMT scores with excellent reliability. Domain-specific neuropsychological tests showed a good to excellent level of reliability and no significant statistical difference between the two administration modes (FTF versus RMT), except for the Digit Span Forward test and the SDMT-oral version.

Indeed, despite a good reliability, the scores at the Digit Span Forward test were significantly higher in the RMT assessment; however, this difference was very small and, in our opinion, not clinically meaningful, as confirmed also by the effect size calculation. Previous research provided conflicting results about the Digit Span Forward test: Wadsworth et al.¹⁸ observed a slightly better performances in the FTF assessment, while later studies showed no difference.^21,63 The Digit Span Backward test showed moderate reliability, but agreement between the FTF and RMT assessments. Although there is moderate evidence supporting the validity of Digit Span Forward and Background in TNP, further research may be required.

SDMT-oral version showed excellent reliability, though worse performances were observed in the RMT mode. Even in this case, however, according to the effect size calculation, the statistical difference does not seem to correspond to a clinically relevant difference. In our opinion, when the visual stimuli are displayed on a screen, the visual search and number-symbol association tasks may become more challenging, resulting in slower and poorer performances. Indeed, a previous study had already shown that participants with lower computer familiarity showed poorer performance on speeded tasks requiring complex attention, raising the possibility that the interaction with the unfamiliar interface demands more cognitive resources.⁶⁴ These aspects may have also contributed to reduce the reliability of the Stroop's Time scores, that showed moderate reliability. Furthermore, also the size of the patient's device's screen may have influenced these results. In our study, a statistical analysis considering the different devices/screen sizes was not feasible due to the small number of subjects in the subgroups; future studies with greater statistical power are needed to clarify this pivotal topic.

Thus, our results seem to suggest that performing visual based tests in TNP may need more time than in the traditional assessment, representing a non-negligible bias for time-based tests. This finding is significant because it highlights important issues, including the relevance of an accurate selection of tests and the need for validation and standardization of tests in remote mode.

Agreement between the RMT and FTF groups and high levels of reliability (from good to excellent) were demonstrated for all other domain-specific neuropsychological tests that were investigated. Our results confirmed the strong validity of the RAVLT verbal memory test (which is comparable to the Hopkins Verbal Learning Test-Revised, HVLT-R, often used in the TNP literature), as well as of the letter fluency in remote assessments.³² Literature data on semantic fluency have shown results that were not always homogeneous, however in our trial the semantic fluency test had excellent reliability.

Our study provided new insights about the validity and reliability of tests that assess the executive and visuo-constructive abilities, for which there is still insufficient or inconsistent evidence.³² CDT, FAB, and Constructional Praxis tests, all showed high reliability. However, the remote administration of these tests may require the participation of the caregiver to: (1) complete some of the items (e.g., FAB prehension behavior), (2) help the patient to hold up their drawings to the camera so that they could be scored by the examiner (CDT or Praxis tests), (3) move the camera when requested by the test (e.g., to frame patient's hand movement during the FAB motor programming task). These difficulties may be partially overcome by the development of digital tools that facilitate more interaction between patient and the examiner and provide an output of the patient's actions.

Overall, our findings proved that for verbally mediated neuropsychological tests administered remotely, both the procedure and the results are comparable to those obtained from traditional face to face assessment; for tests that are based on visual stimuli or that require a written output or motor interaction, slight adaptations to test instructions and protocols were needed. Although our results indicate a high level of reliability also for these tests, in accordance with TNP recommendations,³ the possible implications of these changes should not be underestimated and need to be better explored.

As expected, the scales to assess the activities related to independent living (ADL, IADL) showed excellent reliability: this finding is not surprising as the questionnaires were completed by the same caregiver in both FTF and RMT evaluations and the time elapsed between the two assessments was too short to observe functional changes.

Finally, the RMT mode required a similar amount of time as the FTF assessment: data showed a statistical difference, not clinically meaningful.

Data about the feasibility and the level of satisfaction for the RMT assessment showed that only a small number of people had technical difficulties. In our opinion, this finding was possible thanks to the presence of a caregiver who provided technical support, given that most of today's elderly lack access to technological devices or internet connection and have no familiarity with digital devices.⁴² It is plausible to imagine greater ease for the next generation of elderly people who will be more accustomed to the use of digital technology.

In line with literature data,^22–27 most of participants expressed great satisfaction with the RMT assessment mode (89.2%, “I was satisfied with the mode the remote assessment was carried out”) and thought it should be used as a supplementary option in the future. The feeling of protection, the possibility to avoid travel, and the reduction of waiting times were the primary benefits reported; conversely, the lack of human presence and the need for proximity with the examiner were reported as disadvantages. The use of videoconference modality (instead of phone-based assessment) allows the patient to see the examiner and to get some visual feedback and eye contact, reducing this limit. Based on patients’ reports, another aspect to consider when the remote assessment is performed at home, is the ease to distraction and the possibility that the patient's performance is influenced by the presence of the caregiver. As highlighted in previous studies,⁶⁵ ensuring an appropriate remote environment for patient assessment is necessary to help guarantee the validity and reliability of remote neuropsychological assessments. In the case of in-home TNP, the assessment may be affected or interrupted by the presence of family members, pets, doorbell sound, or others distractions. Therefore, it is important that the examiner schedules a quiet time of day for the patient's visit and informs the caregiver to intervene only at the examiner's express request or to offer technical assistance. It is also critical to ensure that the space is free of any elements that may facilitate the patient's response (such as, clocks or calendars); also in this case, the videoconference modality compared to the phone assessment may reduce this bias (it is possible to observe whether the patient looks for suggestions in the environment) as well as patients’ “cheating” tendency (e.g., patients who wrote down the stimuli to remember), reported in other studies.⁶³

Overall, the use of a neuropsychological battery including domain-specific tests delivered at home, represents the main strength of this work. Indeed, most videoconference administrations were conducted in a controlled environment, usually in a research facility³²: our study proved that the remote at home administration of the most widely used measures to evaluate cognitive functions in adults with suspected cognitive disorders is equally reliable as the most widespread in-clinic assessment, extending the kit of cognitive tools available for videoconference mode, even administered outside the clinic context.

The low educational level of the participants represents another strength and unique feature of this study. Although data from previous studies have suggested that low school attainment may represent a barrier limiting the access to telemedicine, as people may struggle to understand how to use the technology,⁶⁵ our results show that also people with low education are able to understand how to interact with the clinician remotely and that the remote interaction is well accepted, effective, and reliable. Therefore, low education, as well as advanced age or the presence of cognitive disorders should not a priori determine the exclusion of these subjects from telemedicine services.

Overall, our data provide further evidence that videoconferencing may represent a useful, feasible and valid method to assess cognitive functions; however, RMT administration of cognitive tests still represents a nonstandard administration. According to international literature, some methodological-statistical issues need to be addressed including the standardization of Italian TNP tools, the development of norms derived specifically for remote administration, the collection of further data about validity and reliability of TNP tools as well as on the clinical usability of TNP tools in target population.^{31,39,40,66,67}

This study has some limitations. First, we did not establish a specific equipment for remote evaluation: despite most participants used computers, also tablets and smartphones were occasionally used. Thus, the impact of screen size in performing tests with visual stimuli needs to explored in future studies with adequate statistical power. This variability is inherent in the in-home TNP approach where the control on environmental variables (type of device, distractions, interruptions) is reduced; on the other hand, this approach provides information about the feasibility of TNP in the real world and embodies the main advantage of TNP according to users, namely the fact of not having to move, as reported also by the participants in this study. Of course, several factors need to be carefully considered to help ensure that the test findings are valid: patient visibility, lighting, preventive control of audio/video quality and connection, assistance from family members to reduce external distractors or aids (e.g., calendars and notes). The videoconference administration, as opposed to telephone, can mitigate some of these issues, ensuring continuous visual contact between operator and patient.

Second, the satisfaction questionnaire was completed by people who agreed to participate in the study and who were probably more inclined towards telemedicine. Among the patients assessed for eligibility, about a fifth declined to participate to the study: future research investigating also the specific reasons for refusal might provide new interesting information.

Finally, our results may not be generalizable to people with moderate to severe cognitive impairment or with other neurological or neuropsychiatric disorders, since the neuropsychological battery was administered to adult people with subjective or mild cognitive disorders (specifically, SCD, Mild NCD and mild dementia). Future studies including different population are warranted.

Despite these limitations, our results address to the use of the in-home TNP, suggesting that it represents a valid, reliable, and well-accepted way of performing the neuropsychological assessment. Integrated in the multidisciplinary plan for dementia care and management, TNP may be helpful to complement the regular health care services even after the pandemic for both clinical (early diagnosis and follow-up) and research purposes, improving care for patients and their caregivers. With these promising results, future efforts should focus on the creation of formal guidelines and standards for the practice of TNP to help psychologists provide a high-quality standardized neuropsychological assessment, comparable to in-person assessment. Our study aimed to be a first step in the implementation of TNP in an Italian clinical setting.

Footnotes

Acknowledgments

The authors have no acknowledgments to report.

ORCID iD

Chiara Zucchella

Author contributions

Miriana Maria Bressan (Conceptualization; Data curation; Formal analysis; Investigation; Methodology; Writing – original draft; Writing – review & editing); Anna Maria Musso (Conceptualization; Investigation; Methodology; Supervision; Writing – review & editing); Tommaso Bovi (Conceptualization; Investigation; Methodology; Writing – review & editing); Bruno Bonetti (Conceptualization; Funding acquisition; Investigation; Methodology; Supervision; Writing – review & editing); Chiara Zucchella (Conceptualization; Data curation; Formal analysis; Investigation; Methodology; Project administration; Writing – original draft; Writing – review & editing).

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is funded by Brain Research Foundation Verona – BRFVr.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability

The data supporting the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

References

Alzheimer Europe. Dementia in Europe Yearbook 2019: estimating the prevalence of dementia in Europe. Senningerberg, Luxembourg: Alzheimer Europe, 2019.

Colucci

Bosco

Fasanaro

, et al. Alzheimer's disease costs: what we know and what we should take into account. J Alzheimers Dis 2014; 42: 1311–1324.

Grosch

Gottlieb

Cullum

. Initial practice recommendations for teleneuropsychology. Clin Neuropsychol 2011; 25: 1119–1133.

Benaque

Gurruchaga

Abdelnour

, et al. Dementia care in times of COVID-19: experience at Fundacio ACE in Barcelona, Spain. J Alzheimers Dis 2020; 76: 33–40.

Adams

Myers

Waddell

, et al. Telemedicine: a valuable tool in neurodegenerative diseases. Curr Geriatr Rep 2020; 9: 72–81.

Corallo

Maresca

Bonanno

, et al. Importance of telemedicine in mild cognitive impairment and Alzheimer disease patients’ population during admission to emergency departments with COVID-19. Medicine (Baltimore) 2023; 102: e32934.

Hammers

. Teleneuropsychology and applications of technology. In: Bush

Yochim

(eds) A handbook of geriatric neuropsychology, practice essentials. New York: Routledge, 2022, pp.267–285.

Miller

Barr

. The technology crisis in neuropsychology. Arch Clin Neuropsychol 2017; 32: 541–554.

Bilder

Postal

Barisa

, et al. Inter organizational practice committee recommendations/guidance for teleneuropsychology in response to the COVID-19 pandemic. Clin Neuropsychol 2020; 34: 1314–1334.

10.

Geddes

O’Connell

Fisk

, et al. Remote cognitive and behavioral assessment: report of the Alzheimer society of Canada task force on dementia care best practices for COVID-19. Alzheimers Dement (Amst) 2020; 12: e12111.

11.

Postal

Bilder

Lanca

, et al. Inter organizational practice committee guidance/recommendation for models of care during the novel coronavirus pandemic. Clin Neuropsychol 2021; 35: 81–98.

12.

Hildebrand

Chow

Williams

, et al. Feasibility of neuropsychological testing of older adults via videoconference: implications for assessing the capacity for independent living. J Telemed Telecare 2004; 10: 130–134.

13.

Cullum

Weiner

Gehrmann

, et al. Feasibility of telecognitive assessment in dementia. Assessment 2006; 13: 385–390.

14.

Loh

Donaldson

Flicker

, et al. Development of a telemedicine protocol for the diagnosis of Alzheimer’s disease. J Telemed Telecare 2007; 13: 90–94.

15.

McEachern

Kirk

Morgan

, et al. Reliability of the MMSE administered in-person and by telehealth. Can J Neurol Sci 2008; 35: 643–646.

16.

Cullum

Hynan

Grosch

, et al. Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment. J Int Neuropsychol Soc 2014; 20: 1028–1033.

17.

Harrell

Wilkins

Connor

, et al. Telemedicine and the evaluation of cognitive impairment: the additive value of neuropsychological assessment. J Am Med Dir Assoc 2014; 15: 600–606.

18.

Wadsworth

Galusha-Glasscock

Womack

, et al. Remote neuropsychological assessment in rural American Indians with and without cognitive impairment. Arch Clin Neuropsychol 2016; 31: 420–425.

19.

Brearly

Shura

Martindale

, et al. Neuropsychological test administration by videoconference: a systematic review and meta-analysis. Neuropsychol Rev 2017; 27: 174–186.

20.

Lindauer

Seelye

Lyons

, et al. Dementia care comes home: patient and caregiver assessment via telemedicine. Gerontologist 2017; 57: e85–e93.

21.

Wadsworth

Dhima

Womack

, et al. Validity of teleneuropsychological assessment in older patients with cognitive disorders. Arch Clin Neuropsychol 2018; 33: 1040–1045.

22.

Parikh

Grosch

Graham

, et al. Consumer acceptability of brief videoconference-based neuropsychological assessment in older individuals with and without cognitive impairment. Clin Neuropsychol 2013; 27: 808–817.

23.

Appleman

O'Connor

Boucher

, et al. Teleneuropsychology clinic development and patient satisfaction. Clin Neuropsychol 2021; 35: 819–837.

24.

Elbaz

Cinalioglu

Sekhon

, et al.

A systematic review of telemedicine for older adults with dementia during COVID-19: an alternative to in-person health services?

Front Neurol 2021; 14: 761965.

25.

Nanda

Sharma

. A review of patient satisfaction and experience with telemedicine: a virtual solution during and beyond COVID-19 pandemic. Telemed J E Health 2021; 27: 1325–1331.

26.

Parsons

Gardner

Sherman

, et al. Feasibility and acceptance of direct-to-home tele-neuropsychology services during the COVID-19 pandemic. J Int Neuropsychol Soc 2022; 28: 210–215.

27.

Sumpter

Camsey

Meldrum

, et al. Remote neuropsychological assessment: acceptability and feasibility of direct-to-home teleneuropsychology methodology during the COVID-19 pandemic. Clin Neuropsychol 2023; 37: 432–447.

28.

Angelopoulou

Papachristou

Bougea

, et al. How telemedicine can improve the quality of care for patients with Alzheimer's disease and related dementias? A narrative review. Medicina (Kaunas) 2022; 58: 1705.

29.

Martínez-Alcalá

Pliego-Pastrana

Rosales-Lagarde

, et al. Information and communication technologies in the care of the elderly: systematic review of applications aimed at patients with dementia and caregivers. JMIR Rehabil Assist Technol 2016; 3: e6.

30.

Strauss

Sherman

EMS

Spreen

. Norms selection in neuropsychological assessment. In: Strauss

Sherman

EMS

Spreen

(eds) A compendium of neuropsychological tests: administration, norms, and commentary. 3rd edition. Oxford: University Press, 2006, pp.44–54.

31.

Zanin

Aiello

Diana

, et al. Tele-neuropsychological assessment tools in Italy: a systematic review on psychometric properties and usability. Neurol Sci 2022; 43: 125–138.

32.

Marra

Hamlet

Bauer

, et al. Validity of teleneuropsychology for older adults in response to COVID-19: a systematic and critical review. Clin Neuropsychol 2020; 34: 1411–1452.

33.

Sperling

Acheson

Fox-Fuller

, et al. Tele-neuropsychology: from science to policy to practice. Arch Clin Neuropsychol 2024; 39: 227–248.

34.

Alegret

Espinosa

Ortega

, et al. From face-to-face to home-to-home: validity of a teleneuropsychological battery. J Alzheimers Dis 2021; 81: 1541–1553.

35.

Barcellos

Horton

Shao

, et al. A validation study for remote testing of cognitive function in multiple sclerosis. Mult Scler 2021; 27: 795–798.

36.

Chapman

Gardner

Ponsford

, et al. Comparing performance across in-person and videoconference-based administrations of common neuropsychological measures in community-based survivors of stroke. J Int Neuropsychol Soc 2021; 27: 697–710.

37.

Harder

Hernandez

Hague

, et al. Home-based pediatric teleneuropsychology: a validation study. Arch Clin Neuropsychol 2020; 35: 1266–1275.

38.

Worhach

Boduch

Zhang

, et al. Remote assessment of pediatric patients with daytime sleepiness and healthy controls: a pilot study of feasibility and reliability. Neurol Open 2021; 8: 2329048X–211048064.

39.

Binng

Splonskowski

Jacova

. Distance assessment for detecting cognitive impairment in older adults: a systematic review of psychometric evidence. Dement Geriatr Cogn Disord 2020; 49: 456–470.

40.

Hunter

Jenkins

Dolan

, et al. Reliability of telephone and videoconference methods of cognitive assessment in older adults with and without dementia. J Alzheimers Dis 2021; 81: 1625–1647.

41.

ISTAT. https://www.istat.it/it/files/2023/12/Cittadini-e-ICT-2023.pdf (2023, accessed 19 August 2024).

42.

Sala

Gaia

. Older people’ s use of information and communications technologies in Europe. The Italian case. Autonomie Locali Serv Soc 2019; 2: 163–183.

43.

Jessen

Amariglio

van Boxtel

, et al. A conceptual framework for research on subjective cognitive decline in preclinical Alzheimer's disease. Alzheimers Dement 2014; 10: 844–852.

44.

Magni

Binetti

Bianchetti

, et al. Mini-Mental state examination: a normative study in Italian elderly population. Eur J Neurol 1996; 3: 198–202.

45.

Katz

. A.D.L. Activities of daily living. JAMA 1963; 185: 914–919.

46.

Lawton

Brody

. Assessment of older people: self-maintaining and instrumental activities of daily living. Gerontologist 1969; 9: 179–186.

47.

Carlesimo

Caltagirone

Gainotti

. The mental deterioration battery: normative data, diagnostic reliability and qualitative analyses of cognitive impairment. The group for the standardization of the mental deterioration battery. Eur Neurol 1996; 36: 378–384.

48.

Monaco

Costa

Caltagirone

, et al. Erratum to: forward and backward span for verbal and visuo-spatial data: standardization and normative data from an Italian adult population. Neurol Sci 2015; 36: 345–347. Erratum for: Neurol Sci 2013; 34: 749–754.

49.

Nocentini

Giordano

Di Vincenzo

, et al. The symbol digit modalities test - oral version: Italian normative data. Funct Neurol 2006; 21: 93–96.

50.

Aiello

Esposito

Gramegna

, et al. The frontal assessment battery (FAB) and its sub-scales: validation and updated normative data in an Italian population sample. Neurol Sci 2022; 43: 979–984. Erratum in: Neurol Sci 2022; 43: 6621. Erratum in: Neurol Sci 2023; 44: 425.

51.

Mondini

Mapelli

Vestri

, et al. Esame neuropsicologico breve. Milano, IT: Raffaello Cortina Editore, 2003.

52.

Caffarra

Vezzaini

Dieci

, et al. A short version of the Stroop test: normative data in an Italian population sample. Nuova Riv Neurol 2022; 12: 111–115.

53.

Novelli

Papagno

Capitani

, et al. Tre test clinici di ricerca e produzione lessicale. Taratura su soggetti normali. Arch Psicol Neurol Psichiatr 1986; 4: 477–506.

54.

Spinnler

Tognoni

. Standardizzazione e taratura italiana di test neuropsicologici. Ital J Neurol Sci 1987; 8: 1–120.

55.

Cohen

. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. New York, NY: Routledge Academic, 1988.

56.

Lin

. A concordance correlation coefficient to evaluate reproducibility. Biometrics 1989; 45: 255–268.

57.

Lin

. A note on the concordance correlation coefficient. Biometrics 2000; 56: 324–325.

58.

Altman

. Practical statistics for medical research. New York: Chapman and Hall/CRC, 1990.

59.

Bland

Altman

. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 8: 307–310.

60.

Timpano

Pirrotta

Bonanno

, et al. Videoconference-based mini mental state examination: a validation study. Telemed J E Health 2013; 19: 931–937.

61.

Kim

Jhoo

Jang

. The effect of telemedicine on cognitive decline in patients with dementia. J Telemed Telecare 2017; 23: 149–154.

62.

Carotenuto

Rea

Traini

, et al. Cognitive assessment of patients with Alzheimer's disease by telemedicine: pilot study. JMIR Ment Health 2018; 5: e31.

63.

Vaccaro

Aglieri

Rossi

, et al. Remote testing in Abbiategrasso (RTA): results from a counterbalanced cross-over study on direct-to-home neuropsychology with older adults. Aging Clin Exp Res 2023; 35: 699–710.

64.

Jacova

McGrenere

Lee

, et al. C-TOC (cognitive testing on computer): investigating the usability and validity of a novel self-administered cognitive assessment tool in aging and early dementia. Alzheimer Dis Assoc Disord 2015; 29: 213–221.

65.

Luxton

Pruitt

Osenbach

. Best practices for remote psychological assessment via telehealth technologies. Prof Psychol Res Pr 2014; 45: 27–35.

66.

Booth

Murray

Muniz-Terrera

. Are we measuring the same thing? Psychometric and research considerations when adopting new testing modes in the time of COVID-19. Alzheimers Dement 2021; 17: 251–254.

67.

Gnassounou

Defontaines

Denolle

, et al. Comparison of neuropsychological assessment by videoconference and face to face. J Int Neuropsychol Soc 2022; 28: 483–493.