From Face-to-Face to Home-to-Home: Validity of a Teleneuropsychological Battery

Abstract

Background:

Over the last decade, teleneuropsychology has increased substantially. There is a need for valid neuropsychological batteries to be administered home-to-home. Since 2006, the neuropsychological battery of Fundació ACE (NBACE) has been administered face-to-face in our clinical settings. Recently, we adapted the NBACE for teleneuropsychology use to be administered home-to-home (NBACEtn).

Objective:

The aims of the present study are: 1) to determine the home-to-home NBACE equivalence compared to its original face-to-face version; and 2) to examine home-to-home NBACE discriminant capacity by differentiating among cognitively healthy, mild cognitive impairment, or mild dementia subjects and comparing it with the face-to-face version.

Methods:

Data from 338 individuals assessed home-to-home (NBACEtn) were contrasted with 7,990 participants assessed with its face-to-face version (NBACE). Exploratory and confirmatory factorial structure, and invariance analysis of the two versions of the battery were performed.

Results:

Exploratory and confirmatory factor analysis supported the four-factor model (attention, memory, executive, and visuospatial/constructional functions). Configural, metric, and scalar measurement invariance was found between home-to-home and face-to-face NBACE versions. Significant differences in most of the neuropsychological variables assessed were observed between the three clinical groups in both versions of administration. No differences were found between the technological devices used by participants (computer or tablet and mobile devices).

Conclusion:

For the first time, invariance analysis findings were addressed by determining a teleneuropsychological battery’s equivalence in comparison with its face-to-face version. This study amplifies the neuropsychological assessment’s applicability using a home-to-home format, maintaining the original measure’s structure, interpretability, and discriminant capacity.

Keywords

Alzheimer’s disease cognitive impairment construct validity face-to-face home-to-home invariance neuropsychological assessment telemedicine teleneuropsychology

INTRODUCTION

Teleneuropsychology, as a part of telemedicine, uses videoconference technology to deliver specialized health-care services for people who cannot attend medical appointments in person for any reason, such as mobility problems or the novel coronavirus disease (COVID-19) pandemic. This procedure facilitates earlier diagnosis and management of chronic diseases [1, 2].

Over the last decade, teleneuropsychology has increased substantially. However, neuropsychologists have often been reluctant to change from face-to-face to home-to-home assessments, which may be due to the person-to-person tradition, confidentiality issues, and the need to change the administration instructions and procedures of some tests [3, 4]. The potential effects of these modifications need to be validated and should not be underestimated [1, 5]. Despite this, several studies have reported similar results across face-to-face and videoconference-administered tests [4 –7], being well received by people with and without cognitive impairment [5 , 9]. Some studies about teleneuropsychology have included measures of global cognition, such as the Mini-Mental State Exam (MMSE) [6 , 8–10]; attention, such as digit span forward [5 , 10]; verbal learning and memory such as Hopkins Verbal Learning test [6, 10]; language such as Boston Naming test [5 , 10]; and executive functions such as letter fluency or Trail Making test [5 , 10]; and others have used existing neuropsychological batteries such as the Repeated Battery for the Assessment of Neuropsychological Status (RBANS) [5] and the Rowland Universal Dementia Assessment Scale [11], all of them in reduced samples with and without cognitive impairment [5 , 8–12].

The videoconference has been demonstrated to be a valid procedure for brief neuropsychological assessments in a controlled scenario, for example a large screen, a high-speed connection, and a hospital testing office [6 , 9–14]. However, there exists a need for home-to-home teleneuropsychological studies assessing their validity and availability in real-life conditions [15]. Further research is needed on neuropsychological batteries, including praxis and visual tests to achieve a complete teleneuropsychological assessment by a wide range of devices, mainly computers and tablets, but also mobile applications.

Elderly and male patients as well as those with underlying health conditions have an increased risk of developing COVID-19 [16]. Moreover, patients with mild cognitive impairment (MCI) or dementia may have incomplete public health information about the COVID-19 pandemic as well as difficulties in remembering and organizing safeguard actions, such as the need of confinement at home, the proper use of masks, or the maintaining of a safe social distance [17]. In other words, telemedicine can be an opportunity to attend positive or negative COVID-19 patients with cognitive impairment, avoiding the chance of infection. In fact, the COVID-19 pandemic has accelerated the rise of telemedicine, and it might be taken as an excellent opportunity to improve teleneuropsychological assessment procedures, which became the starting point of this study. While teleneuropsychological assessment is increasingly showing clinical potential, providers may need to be mindful of its strengths, limitations, and appropriate uses for brief cognitive assessments [15].

The neuropsychological battery in use in Fundació ACE (NBACE) is a relatively brief and easy-to-administer test battery able to detect cognitive impairment in adulthood [18] and predict the conversion to dementia and Alzheimer’s disease [19 –21]. It has been administered, for first/baseline and follow-up face-to-face visits, from January 2006 to March 2020 at the Memory Unit of Fundació ACE, with standardized and cut-off values to be used in the clinical practice [18, 22]. Recently, the NBACE was adapted to be administered home-to-home from the neuropsychologist home office instead of in-person visits to ensure the continuity of Fundació ACE’s specialty health-care services [23] to people during COVID-19 confinement [2] and, as recommended, guaranteeing that the videoconferencing system was as similar as possible to face-to-face condition [1]. A home-to-home version of the NBACE administration for teleneuropsychology (NBACEtn) was performed using available videoconferencing technological devices (tablets and desktop computers by means of the Skype, FaceTime, or Google Duo applications; and mobile devices by means of the WhatsApp application for individuals who did not have tablets or computers) to ensure the continuity of health care.

However, whether the NBACEtn preserves its theoretical structure, interpretability, and discriminant capacity in comparison with its original face-to-face version (NBACE) remains unclear. Thus, it was hypothesized that neuropsychological test performances obtained with the home-to-home and face-to-face NBACE versions would be similar in these different psychometric properties when assessing and differentiating cognitively healthy (CH) individuals and patients with MCI or dementia, allowing accurate clinical diagnoses. One crucial point when considering the correspondence between these two forms of the same measure is to ensure that both ways of administration provide the needed guaranty of an equal interpretation. What is proposed in this study is to determine this property under a systematic and structured assessment, determining the invariance of the NBACEtn.

Measurement invariance allows to assess the equivalence of latent variables or constructs across groups of people in the context of a prospective design [24]. The main idea justifying this approach is to determine whether the construct of interest has the same structure and meaning independently of the group assessed. Something as common as to contrast the difference between two groups of subjects, for example, scores in a neuropsychological test between men and women, or pre-and post-treatment in the same participants, require having been explored invariance as previous and necessary step. Unless measurement invariance has been demonstrated, it is not possible to execute group comparisons. Despite the enormous relevance of assessing invariance given the potential dramatic impact of not doing so, it has received poor attention in studies of the validation or adaptation of psychological measures. To the best of our knowledge, no published literature so far uses the invariance measure to determine the equivalence of the home-to-home and face-to-face versions of the same neuropsychological battery. The aims of the present study are the following: 1) to determine the home-to-home NBACE (NBACEtn) equivalence compared to its original face-to-face version (NBACE) regarding its structure and interpretation; and 2) to examine home to-home NBACE discriminant capacity by differentiating among CH, MCI, or mild dementia subjects and comparing it with its face-to-face NBACE version.

METHODS

Participants

From all individuals assessed with the NBACEtn and who completed the Fundació ACE diagnostic procedure [23] from March 18 to May 29, 2020, those with a diagnosis of CH, MCI, or mild dementia were selected for analyses. The comparative face-to-face group was taken from people who were assessed with the original NBACE in person from January 2006 to March 2020 at the Memory Unit of Fundació ACE with a final clinical diagnosis of CH, MCI, or mild dementia at a daily consensus meeting with the multidisciplinary team [23].

The diagnosis for the CH group were as follows: the absence of objective cognitive impairment, with average or above-average scores on the NBACE [18, 22]; normal general cognition (MMSE score≥27) [25, 26]; a Clinical Dementia Rating (CDR) [27] of 0; and no history of functional impairment caused by declining cognition, as reported by an informant in the neurological interview, and with a score below 4 on the Blessed Dementia Rating Scale (BDRS) [28, 29].

The diagnosis for the MCI group were: preserved general cognition (MMSE score≥24); relatively normal performance in activities of daily living (as reported by an informant and a BDRS < 4); the absence of dementia; a CDR of 0.5; an objectively measurable impairment in memory (in the word list learning test from the Wechsler Memory Scale, third edition (WMS-III) [30], without interference list [18]) or another cognitive function [18, 31]; and the absence of prescribed symptomatic treatment for dementia (i.e., acetylcholinesterase inhibitors or memantine).

The diagnosis for the dementia group were as follows: diagnosis of dementia [32]; a severity of dementia determined by a total CDR≥1; an MMSE score < 24; and impaired activities of daily living (as reported by an informant and a BDRS≥4). For the purpose of this study, only patients with mild dementia (CDR = 1 and MMSE > 19) were included in the analyses.

Procedure

As detailed elsewhere [2], the coordinated multidisciplinary team of Fundació ACE migrated from a face-to-face model of visits to one mainly based on telemedicine, in an adaptive and individualized manner so no one was left behind. Fundació ACE’s Human Resources team ensured that all professionals could safely and effectively deliver care while home-based. A secure remote access to medical records through a virtual private network was habilitated. Moreover, neurological and neuropsychological visits were adapted to be carried out by videoconference. Thus, neuropsychologists had a computer and a tablet, apart from the connection to the eHealth platform of Fundació ACE to access protocols and enter data, to ensure the correct home-to-home procedure.

Depending on available resources on the patients’ side, our team began doing consultations using videoconference platforms such as Skype, FaceTime, GoogleDuo, or WhatsApp. Usually when patients did not know how to use it, a close person helped them, being allowed to be near them during the visit without interfering. Prior to the evaluation, informed verbal consent to perform the Fundació ACE’s teleneuropsychological visit was obtained from all individuals (and caregivers in the cases of dementia); and we wrote down the name of people who gave their permission.

Then, the subjects were seated in front of the technological device screen and were greeted and introduced by the remote examiner to receive a home-to-home teleneuropsychological assessment (NBACEtn). The study was conducted in accordance with the Declaration of Helsinki and under Spanish biomedical laws (Law 14/July 3, 2007, about biomedical research; Royal Decree 1716/November 18, 2011). The study was approved by the Fundació ACE Research Ethics Committee.

The NBACEtn

From March 18 to May 20, 2020, a home-to-home teleneuropsychological visit that lasted approximately fifty minutes was carried out by the Fundació ACE neuropsychology team [2]. It resulted from an NBACE adaptation that had been administered in person from January 2006 to March 2020 at the Memory Unit of Fundació ACE. Apart from our deep knowledge of using the NBACE, it has the advantage of having standardized and cut-off values to be used in the clinical practice [18, 22].

Similar to the NBACE [22], the NBACEtn includes tests sensitive to information processing speed, orientation, attention, memory, language, visuoperception, praxis, and visuospatial and executive functions. As described elsewhere [2], the NBACEtn includes the following tests: temporal, spatial and personal orientation; digit span (forward and backward) and similarities (abbreviated to the 10 first items) subtests of the Wechsler Adult Intelligence Scale, third edition (WAIS–III) [33]; the word list learning test from the WMS–III [30], without interference list [22]; verbal comprehension (to correctly execute two simple, two semi-complex, and two complex commands extracted from ADAS-cognitive subscale [34] and the Barcelona test battery [35]; repetition (two words and two sentences) [36]; an abbreviated fifteen-item naming test from the Boston Naming Test [37]; two Poppelreuter-type overlap figures [38]; Luria’s clock test [39]; the automatic inhibition subtest of the Syndrom-Kurztest (SKT) [40]; letter fluency (words beginning with “p” in one minute) [41]; category fluency (“animals” in one minute) [36]; the clock test [42]; and the 15-Objects Test [43, 44]. Finally, depressive and anxiety symptoms were measured with the Spanish version of the Hospital Anxiety and Depression Scale [45, 46].

Considering the impossibility to administer the WAIS-III block as the test material (the cubes) was not available for the participants, the figure copy of RBANS [47, 48] was selected instead. Thus, the WAIS-III block design was the only test of the original NBACE not included in the NBACEtn, but it had two additional tests: 1) the RBANS figure copy, which allows one to assess constructional praxis and long-term visual memory using the figure recall of RBANS; and 2) verb fluency to assess executive function [49, 50]. The tests added in the NBACEtn were not entered in the analyses planned here because this requires the NBACEtn and NBACE variables to be exactly the same, in addition to the complete administration of the neuropsychological battery.

Apart from the procedural differences intrinsic to technological devices, the only other differences for home-to-home administration were those for the scoring and registration of nonverbal tests, such as the RBANS figure copy and the clock test; the participants were asked to hold up their drawings to the camera to be conventionally scored by the neuropsychologist, ensuring accuracy. The scoring of the drawings was done in real time while the subjects held up their products to the camera. Moreover, they were strictly warned not to write or copy during the memory tests. It has to be mentioned, that the visit started after the neuropsychologist verified that videoconference and environmental conditions were optimal. However, computer experience for the patients was not required.

Statistical analysis

The home-to-home group was contrasted with a sample of 7,990 face-to-face participants extracted from the historical neuropsychological records. This sample was selected by applying the same inclusion criteria presented above for the home-to-home group. The sample was randomly divided into two subsamples, 70% (n = 5,588) versus 30% (n = 2,402).

The biggest subsample (70%), here called the exploratory face-to-face group, was used to determine the factorial structure of neuropsychological variables in an exploratory way using principal component analysis. Varimax was used as rotation strategy to identify factorials solutions. This strategy is commonly used when the aim of the exploration is to simplify the interpretation of the resulting factors. Moreover, Varimax rotation has been also identified as the best rotation approach when ensuring factorial invariance [51]. The aim of this first step was to find the most optimal factorial model. Total explained variance and discriminant factorial loadings in the rotated matrix, higher in one factor and lower in the rest, were the main criteria to select the final factorial solution. Models of three, four, and five factors were explored.

The second subsample (30%) was analyzed under a confirmatory factorial analysis (CFA) approach, testing the quality of the models with an acceptable result in the previous exploratory analysis. This subsample was identified as the confirmatory face-to-face group. The same confirmatory analyses, and for the same factorial solutions than in the face-to-face group, were explored in the home-to-home group. Standard goodness-of-fit indices were considered: the comparative fit index (CFI), the Tucker–Lewis Index (TLI), and the root mean square error of approximation (RMSEA). Considering the propensity to reject good models when large samples are analyzed [52], the chi-square (χ²) statistic was not included here as a model fit index when making decisions about the quality of the resulting models, although it was reported along with estimated degrees of freedom. According to Marsh et al. (2004) [53] an acceptable model can be inferred with RMSEA values below 0.080 combined with CFI or TLI values greater than 0.90 [53].

To ensure the equivalence of the best-fitting model identified, measurement invariance was assessed. Invariance was explored contrasting the two procedures under the best factorial model obtained in the previous confirmatory section across NBACE versions, home-to-home (n = 338) and confirmatory face-to-face (n = 2,402) groups, imposing the same constraints to the two procedures in its parameters and in a hierarchical way. Thus, configural, metric and scalar invariance were assessed [54]. The first step was to test configural invariance, establishing a baseline model. Here, no restriction was imposed and factor loadings and item intercepts used were freely estimated by the model. The second step was to estimate metric invariance, forcing factor loadings to be equal across groups. Finally, a third model with a new restriction was imposed, in this case constraining factor loadings and item intercepts to assess scalar invariance. These steps are described elsewhere [55]. The overall fit of each model was assessed, considering a CFI or TLI > 0.90 and an RMSEA < 0.080 as evidence of an acceptable model fit. One can assume that an increased invariance is supported if the model fit indices are not significantly diminished when compared to the previous restricted model. Significant differences between two nested models were then determined using the change in CFI (ΔCFI) supplemented by the change in RMSEA (ΔRMSEA) [56]. To contrast the neuropsychological variables of the home-to-home and face-to-face procedures and according to the three clinical conditions of the participants, a multivariant analysis was performed including the ten neuropsychological variables. The NBACE version (home-to-home and face-to-face) and diagnosis (CH, MCI, and mild dementia) were included in the model as independent factors. As the confirmatory face-to-face group consisted of a big sample of participants in contrast to the NBACEtn group and under the aim of not generating over-significant results as a consequence of the sample size, a random selection of participants of the confirmatory face-to-face group was obtained according to the distribution of diagnoses observed in the home-to-home version. Thus, the resulting comparative NBACE sample for this multivariant analysis had 66 CH, 192 MCI, and 80 dementia subjects. In this multivariate analysis, age, sex and years of formal education were used as adjusting variables. Mplus 8.1 was used to assess the confirmatory and invariance results, and SPSS v26 was used to execute the rest of the statistical analyses.

RESULTS

A total of 461 individuals were visited home-to-home by the Fundació ACE neuropsychology team from March 18 to May 29, 2020. Of those, a final sample of 338 participants was analyzed for the purpose of this study, comprising 66 (19.5%) CH individuals, 192 (56.8%) subjects with MCI, and 80 (23.7%) patients with mild dementia. Those patients with a GDS > 1 (n = 69) or missing data on a neuropsychological test (n = 52) were discarded for analysis. All of these 52 latest subjects had missing data on the inhibition subtest of SKT, and two of them also on the Luria’s clocks.

The home-to-home group was contrasted with a sample of 7,990 face-to-face participants assessed with the original NBACE in person from January 2006 to March 2020 at the Memory Unit of Fundació ACE. The distribution of clinical conditions was as follows: 722 (9.0%) CH individuals, 4,599 (57.6%) patients with MCI, and 2,669 (33.4%) patients with mild dementia. Demographic and clinical characteristics of the face-to-face and home-to-home groups are reported in Table 1.

Table 1

Demographic and clinical characteristics of the exploratory face-to-face and home-to-home groups

	Face-to-face (n = 7,990)	Home-to-home (n = 338)	t-test/^*χ²	p
Age (mean/SD/min-max)	74.3 (9.2) [46–93]	74.0 (9.3) [46–93]	0.66	0.512
Sex (% female)	63.1	60.1	^*1.2	0.275
Education (in years) (mean/SD/min-max)	7.7 (4.4) [2–25]	9.0 (4.8) [2–25]	4.7	< 0.005
MMSE (mean/SD/min-max)	25.7 (2.9) [20–30]	26.8 (2.9) [20–30]	6.1	< 0.005
Diagnosis (%)
CH	9.0	19.5
MCI	33.4	23.7
Dementia	57.6	56.8	^*47.1	< 0.005
AD	68.9	62.7
Vascular	17.3	25.3
Lewy Body	4.8	7.2
Frontotemporal	4.2	2.4
Others	4.9	2.4	5.88	0.208

SD, standard deviation; min, minimum; max, maximum; MMSE, Mini-Mental State Examination; CH, cognitively healthy; MCI, mild cognitive impairment; AD, Alzheimer’s disease.

Factorial structure

Rotated factorial loadings of the exploratory face-to-face group (n = 5,588) are reported in Table 2 for solutions of three, four, and five factors. Although the solution of five factors was the output with the biggest explained variance, in the last factor only one variable (Luria’s clock test) showed a relevant score. The four-factor solution explained close to 77% of the total variance. In terms of the discriminability of the factorial loadings, all the higher loadings in a factor were > 0.66 (mean = 0.78), while none of the second higher loadings in the other factors were > 0.45 (mean = 0.29). Every factor showed two or three variables. The three-factorial solution, explaining 71% of the total variance, had higher scores comprising between 0.50 and 0.87 (mean = 0.63), and the second higher loadings in the other factors had a mean of 0.27. Luria’s clock test was not correctly discriminated in only one factor. According to these exploratory results, the four-factor model seems to fit the best with the data.

Table 2

Rotated Factorial Loadings for Three-, Four-, and Five-Factorial solutions in the exploratory face-to-face group

	Three-factorial solution			Four-factorial solution				Five-factorial solution
Total explained variance	70.5%			76.9%				82.5%
Factors	1	2	3	1	2	3	4	1	2	3	4	5
Recognition memory	0.87	0.14	0.19	0.87	0.10	0.04	0.19	0.87	0.06	0.05	0.19	0.09
Delayed recall	0.82	–0.01	0.17	0.83	0.33	0.08	0.17	0.83	0.33	0.08	0.14	0.13
Verbal learning	0.80	0.35	0.21	0.72	0.45	0.24	0.18	0.72	0.44	0.24	0.16	0.13
Letter fluency	0.42	0.62	0.26	0.16	0.82	0.27	0.15	0.16	0.81	0.27	0.20	0.11
Category fluency	0.63	0.40	0.29	0.42	0.70	0.12	0.20	0.41	0.74	0.11	0.10	0.22
Similarities	0.48	0.50	0.36	0.29	0.68	0.23	0.27	0.30	0.55	0.26	0.51	0.02
Digit span (forward)	0.00	0.85	0.04	0.06	0.14	0.92	0.07	0.06	0.12	0.92	0.07	0.05
Digit span (backward)	0.22	0.75	0.26	0.17	0.39	0.67	0.24	0.17	0.36	0.68	0.2	0.18
Luria’s clock test	0.17	0.14	0.86	0.20	0.12	0.16	0.87	0.19	0.18	0.14	0.21	0.92
Clock test	0.32	0.24	0.72	0.22	0.4	0.10	0.67	0.24	0.19	0.14	0.86	0.25

Note: The highest factorial loadings of every factorial solution are in boldface.

Confirmatory and invariance analysis

For the three- and four-factor solutions, confirmatory analyses were calculated in the face-to-face (confirmatory face-to-face group) and home-to-home groups (Table 3). The goodness-of-fit indices indicated an acceptable model fit for the two strategies of administration under the four-factor model. The three-factor model showed, in the face-to-face and home-to-home procedures, an RMSEA > 0.09 and, in the case of the home-to-home procedure, a TLI < 0.90. As a consequence of these results, this last factorial solution was definitively rejected.

Table 3

Confirmatory Analysis: Goodness-of-Fit Indices for the Baseline Model for the confirmatory face-to-face and home-to-home groups and for the Three- and Four-Factor solutions of the NBACE

								90% CI for RMSEA
Group	Factor solution	χ ²	df	CFI	TLI	SRMR	RMSEA	LL	UL
Face-to-face	3F	909.18	32	0.92	0.89	0.05	0.107	0.101	0.113
	4F	465.54	29	0.96	0.94	0.03	0.079	0.073	0.086
Home-to-home	3F	130.72	32	0.95	0.93	0.05	0.096	0.079	0.113
	4F	70.44	29	0.98	0.97	0.03	0.065	0.046	0.084

CFI, comparative fit index; TLI, Tucker–Lewis Index; SRMR, standardized root mean square residual; RMSEA, root mean square error of approximation; CI, confidence interval; LL, lower limit; UL, upper limit.

According to the distribution of the ten neuropsychological variables in the four-factor model, the latent variables that were estimated were as follows: attention, memory, executive and visuospatial/constructional functions. The standardized factor loadings for these latent variables are provided in Table 4.

Table 4

Standardized Factor Loadings and Standard Errors for the confirmatory face-to-face and home-to-home groups of the NBACE

	Face-to-face	Home-to-home
	n = 2,402	n = 338
Memory
Recognition memory	0.71 (0.03)	0.72 (0.01)
Delayed recall	0.90 (0.01)	0.87 (0.01)
Verbal learning	0.95 (0.01)	0.90 (0.01)
Executive Functions
Letter fluency	0.77 (0.03)	0.72 (0.01)
Category fluency	0.85 (0.02)	0.79 (0.01)
Similarities	0.80 (0.02)	0.76 (0.01)
Attention
Digit span (forward)	0.62 (0.04)	0.58 (0.02)
Digit span (backward)	0.85 (0.03)	0.89 (0.02)
Visuospatial/Constructional
Luria’s clock test	0.59 (0.05)	0.64 (0.02)
Clock test	0.72 (0.05)	0.75 (0.02)

Standard errors in parenthesis. All factor loadings are statistically significant (p < 0.001).

Invariance analyses was finally executed contrasting the profile of the two procedures of administration under the solution of 4 factors. Configural invariance was supported because all the fit indices were adjusted to the corresponding thresholds (Table 5). This result means, in an applied sense, that the relations between the 4 latent factors and external variables can be compared across groups because a one-unit change in one group would be equal to a one-unit change in any other group. When the factor loadings were constrained to be equal (metric invariance), the fit indices obtained comparable results. Moreover, ΔCFI and ΔRMSEA were both < 0.01. These results indicate that neuropsychological variables can be interpreted in a comparable way regardless of the administration procedure. Finally, the imposition of a new restriction based on assuming comparable variable intercepts (scalar invariance) did not reduce the fit of indices (ΔCFI and ΔRMSEA < 0.01). This last step indicated that scores on the four latent variables can be interpreted in the same way in both administration procedures.

Table 5

Measurement Invariance Analysis contrasting face-to-face and home-to-home groups for the Four-Factor Model of the NBACE

	Goodness-of-fit indices
Home-to-home (n = 338)	χ ²	df	CFI	TLI	RMSEA (CI90%)	ΔCFI	ΔRMSEA
Face-to-face (n = 2,402)
Configural invariance	535.98	58	0.96	0.95	0.078 (0.072–0.084)
Metric invariance	552.53	64	0.96	0.95	0.075 (0.069–0.080)	0.000	0.003
Scalar invariance	663.69	70	0.96	0.94	0.079 (0.073–0.084)	0.000	0.004

ΔCFI less than or equal to 0.010 is considered non-significant.

A stratified random sample of the confirmatory face-to-face group was obtained according to the distribution of diagnoses in the home-to-home group (CH: n = 66, 19.5%; MCI: n = 192, 56.8%; mild dementia: n = 80, 23.7%). When comparing the distributions of age, sex, and years of formal education between the face-to-face and home-to-home groups in the context of a logistic regression, age and education obtained a statistically significant result (p = 0.018 and p = 0.013, respectively), and sex presented a marginal effect (p = 0.074). Older participants and more educated participants were found in the home-to-home group, while more women tended to be observed in the face-to-face group.

The CH and dementia participants presented no statistically significant differences neither in the MMSE nor in any of the neuropsychological variables when comparing the face-to-face and home-to-home groups. However, MCI group showed statistically significant better scores in the MMSE (almost 1 point) and all variables in the home-to-home procedure except for letter verbal fluency and digit span forward (Table 6). When the diagnoses were compared, most of the results were discriminant. Significant differences also appeared between pairs of diagnoses and for the two administration procedures except for letter verbal fluency when comparing MCI and dementia in the face-to-face version, in digit span (forward and backward), where some pairs of comparisons appeared non-significant or above the threshold imposed by the Bonferroni correction in the two administration versions, and in the case of the clock test, where in the home-to-home procedure, CH and MCI were presented with a comparable mean.

Table 6

Multivariant Analysis Contrasting the face-to-face and home-to-home Groups of the NBACE and the CH, MCI, and Dementia Conditions

	Mean (SE)^*			Post hoc contrasts
	CH	MCI	Dementia	F-T-F versus H-T-H CH/MCI/dementia	CH versus dementia/MCI versus dementia/CH versus MCI
	n = 132	n = 384	n = 160
	(66/66)	(192/192)	(80/80)	p^**	p^***
Memory
Recognition memory	22.63 (0.37)	19.99 (0.20)	17.11 (0.32)	0.347/ < 0.001/0.976	< 0.001/ < 0.001/ < 0.001
	23.09 (0.36)	21.13 (0.20)	17.12 (0.32)		< 0.001/ < 0.001/ < 0.001
Delayed recall	6.38 (0.27)	2.85 (0.15)	1.08 (0.23)	0.030/ < 0.001/0.210	< 0.001/ < 0.001/ < 0.001
	7.17 (0.26)	4.47 (0.15)	1.48 (0.23)		< 0.001/ < 0.001/ < 0.001
Verbal learning	26.89 (0.71)	19.59 (0.39)	4.05 (0.19)	0.252/ < 0.001/0.198	< 0.001/ < 0.001/ < 0.001
	27.94 (0.69)	22.14 (0.39)	4.19 (0.19)		< 0.001/ < 0.001/ < 0.001
Executive Functions
Letter fluency	15.44 (0.53)	10.07 (0.29)	8.33 (0.46)	0.168/0.616/0.293	< 0.001/ < 0.001/0.001
	16.41 (0.52)	9.87 (0.29)	7.67 (0.46)		< 0.001/ < 0.001/ < 0.001
Category fluency	19.36 (0.57)	12.96 (0.31)	9.79 (0.49)	0.686/ < 0.001/0.805	< 0.001/ < 0.001/ < 0.001
	19.66 (0.55)	14.51 (0.31)	9.95 (0.49)		< 0.001/ < 0.001/ < 0.001
Similarities	11.97 (0.35)	8.99 (0.19)	7.52 (0.30)	0.418/ < 0.001/0.596	< 0.001/ < 0.001/ < 0.001
	12.34 (0.34)	10.07 (0.19)	7.30 (0.30)		< 0.001/ < 0.001/ < 0.001
Attention
Digit span (forward)	8.10 (0.21)	6.99 (0.11)	7.19 (0.18)	0.031/0.124/ < 0.001	< 0.001/0.329/0.001
	7.51 (0.20)	6.74 (0.11)	6.33 (0.18)		< 0.001/0.050/0.001
Digit span (backward)	5.17 (0.18)	3.58 (0.09)	3.45 (0.16)	0.917/ < 0.001/0.875	< 0.001/.0462/ < 0.001
	5.20 (0.18)	4.26 (0.09)	3.41 (0.16)		< 0.001/ < 0.001/ < 0.001
Visuospatial/Constructional
Luria’s clock test	3.77 (0.12)	2.87 (0.07)	2.30 (0.10)	0.956/ < 0.001/0.039	< 0.001/ < 0.001/ < 0.001
	3.78 (0.12)	3.36 (0.07)	2.60 (0.11)		< 0.001/ < 0.001/ < 0.001
Clock test	6.50 (0.22)	5.50 (0.12)	4.05 (0.19)	0.734/0.001/0.582	< 0.001/ < 0.001/ < 0.001
	6.40 (0.21)	6.08 (0.12)	4.19 (0.19)		< 0.191/ < 0.001/ < 0.001

The results are adjusted by age, sex, and education. ^*means and standard errors (in parenthesis) for, respectively, the face-to-face and home-to-home procedures. ^**p-value of the contrast between the face-to-face (normal letter) and home-to-home (in italics) procedures in, respectively, CH/MCI/dementia conditions. According to the aim of these analyses, no Bonferroni correction was applied.

After the discriminant capacity of the home-to-home version was assessed, and as a measure of control, the technological device used during the assessment was analyzed. A multivariant analysis was also performed here, including the ten neuropsychological variables as dependent factors and the device (computer/tablet or mobile devices, n = 90 and n = 248, respectively) as an independent factor, adjusted by age, sex, education, and the three diagnoses (the CH, MCI, and dementia conditions, including two dummy variables). Differences between mobile and tablet/computer were not statistically significant. Only the digit span backward showed a tendence of p = 0.068. The rest of neuropsychological variables showed a p > 0.114. However, it has to be mentioned that from the 52 missing data, the 86.5% (n = 45) was produced in the mobile group, and only 7 in the computer/tablet one.

DISCUSSION

Teleneuropsychology assessed with the NBACEtn seems to be a valid, viable, and available tool. In its transition from traditional face-to-face to home-to-home testing, the NBACE has been demonstrated as a valid tool providing the diagnostic and clinical needs of a clinical setting. Up to now, a few studies have been made on teleneuropsychology, most of them using only verbal tests, with reduced samples and carried out in a controlled scenario instead of on a home-to-home basis, assessing its availability.

A confirmatory and invariance analysis of the obtained results has shown an acceptable goodness-of-fit index for the face-to-face and home-to-home versions under the four-factor model, comprising attention, memory, executive and visuospatial/visuoconstructional functions. It has to be mentioned that, following the reasoning of the original NBACE [22], we named executive function to the factor comprising verbal fluency (letter and category) and Similarities subtests because they assess problem solving skills. Verbal fluency requires initiation and maintenance of word production, cognitive flexibility, mental processing speed, and inhibition of certain responses [57] and Similarities measures abstract reasoning and the ability to find the likenesses, instead of saying the differences, between two semantically related words [33]. The four cognitive composites obtained in the CH, MCI, and mild dementia groups by the home-to-home version were similar to those obtained in the original face-to-face NBACE. Our findings are consistent with other studies reporting that teleneuropsychology may be a valid resource for cognitive assessments [5 , 12].

The CH individuals and patients with mild dementia showed similar performances on all the neuropsychological tests when comparing the face-to-face and home-to-home groups. However, the MCI individuals showed better performances in most tests in the home-to-home procedure. This result cannot be explained in terms of a systematic bias where the home-to-home is associated to a better performance (or it is finally applied to participants with a better cognitive status), since only MCI participants are showing this differential outcome. In fact, MCI patients under home-to-home protocol showed a slightly higher general cognitive performance (a mean of 1 point in the MMSE) than their face-to-face counterparts. One possible explanation could be that in the case of MCI group, in which participants were autonomous in their daily life, those who accepted videoconference visits had higher-level skills in their instrumental activities of daily living (i.e., use of technological devices) than those who refused it, which could be translated in better neuropsychological performances in the home-to-home than in the face-to-face group. However, it could also be due to the fact that the MCI entity is a heterogeneous condition between cognitively healthy aging and early dementia [20 , 58].

As far as we know, this is also the first teleneuropsychology study carried out in a home-to-home scenario implemented during the COVID-19 pandemic by an experienced team of a Memory Unit, as detailed elsewhere [2], with a neuropsychological battery with normative data [22], cut-off scores [18], and demonstrated to be a valid measure for the diagnosis and prediction of conversion from MCI to dementia in its original face-to-face version [20, 21]. Up to now, most studies about teleneuropsychology have reduced samples [5 , 12] and hospital-to-home attendance [12] or brief neuropsychological assessments without visual material [9]. Additionally, this data represents the largest study to date of teleneuropsychological assessment. In this study, a comprehensive neuropsychological assessment was administered in a real home-to-home scenario. Measurement invariance is a crucial step when determining the equality of a measure administered in different ways, and to the best of our knowledge, this is also the first study exploring the measurement invariance of face-to-face and home-to-home neuropsychological batteries. Our results provide evidence that remote testing may be an adequate tool for cognitive assessment. We recognize that remote testing may not allow to administrate some cognitive tests and may limit their clinical outcome. However, the results from this study and from our experience, we are convinced about the effectiveness of the teleneuropsychological visits.

According to previous studies regarding the validity of teleneuropsychology [6, 8], our results confirm that the home-to-home NBACE procedure is as valid as its original NBACE administered in person. The vast majority of comparisons among the clinical conditions obtained consistent and expected differences across the neuropsychological variables in the same way as the face-to-face procedure.

Moreover, the device used with the home-to-home battery emerged as non-relevant. All the neuropsychological variables obtained homogeneous results, whether computer/tablet or mobile devices were used. In addition, this observation, to the best of our knowledge, had not been thoroughly explored and reported in the process of validation of a neuropsychological battery administered in a home-to-home manner. Assessments using computer, tablets or mobile devices, undifferentiated, is a substantial advantage for teleneuropsychology, taking into account the higher diffusion of mobile devices in comparison with computers, especially in older populations. The results of this study, however, must be considered as only the first step of others that have to provide more conclusive results in this sense, beyond simple comparisons of means between technological devices. The subsample size of participants under the computer assessment in this study was not necessary to provide consistent results [59], but invariance analysis could also be proposed here, contrasting devices, and conducted to determine the replicability between them. However, it should be noted that the use of the mobile produced more data missing on the inhibition subtest of the SKT.

As mentioned previously [10], some classical neuropsychological tests are not feasible for teleneuropsychology. For this reason, in the conversion from the face-to-face to the home-to-home NBACE versions, to assess the constructional praxis, the block design of WAIS-III was replaced by the RBANS figure copy because the patients had no access to stimuli to complete the task at home. Nowadays, alternative oral versions of some classical tests could be useful for brief teleneuropsychological use, such as the oral Trail Making A and B Tests [60] and the Face Memory Test®. However, the NBACEtn has maintained visual and praxis tests from the original NBACE version, such as visuoperceptual (Poppelreuter-type overlap figures and the 15-Objects Test), and imitational and ideomotor praxis tests, which have proven to be viable and valid neuropsychological tests, reinforcing the viability of teleneuropsychology [6, 8].

Some classical tests do not need instructions or material modifications from their face-to-face to home-to-home versions, such as the word list from the WMS-III, the Boston Naming Test, and the letter and category fluency tests. However, other tests require some modifications to ensure the correct execution of the task, mainly when visual material is present, such as the inhibition subtest of the SKT and the RBANS figure copy. As recommended previously [1], in this study a great effort was made to ensure that the NBACEtn was similarly administrated as the NBACE. Finally, some important practical and ethical issues should be remarked before performing teleneuropsychology. This study was done by a multidisciplinary team from the Fundació ACE institution, with great experience in normal aging, MCI, and dementias. Therefore, the practice of telemedicine requires professionals to provide quality, ethical, and safe services to patients, apart from adequate technological devices and connectivity.

Teleneuropsychological assessment has demonstrated to be a suitable healthcare option. People will be able to choose between face-to-face and home-to-home on their convenience. It will allow to visit patients even if they are far away, temporary in another residence, or if they prefer to be followed up after moving of residence. In fact, our experience during this short period of implementation of telemedicine and teleneuropsychology has allowed us to open a new exploratory field and consider that e-tech can be used to support these patients and facilitate their follow up.

While teleneuropsychological assessment has increasingly shown clinical potential, providers should be mindful of its limitations, strengths, and appropriate uses for cognitive assessment. One limitation is consequence of being a unicentric approach. Although this data represents the largest sample to date of teleneuropsychological testing, results observed in this study, and according to the analytical approach here proposed, have to be interpreted as the first of others conducted to ensure the equivalence of measures administered in different formats and replicated in other centers, in order to determine its generalization. The second limitation is the design applied, where the home-to-home version was contrasted with a historical dataset of neuropsychological assessments. This last subsample was not explicitly designed to meet the objectives of the present study. A classical experimental design would have been recommended here. One scenario would be assigning participants at random to the two versions of the measure, creating a between-subjects design. The second experimental option would be a within-subject (repeated measures) design, contrasting both formats in the same sample of participants. However, the present study emerged from the need of providing continuous attention in a Memory Clinic during a lockdown, and experimental design was not possible to apply. Moreover, a within-subject design could be applied if it is possible to ensure that no learning or memory process is present when the second form of administration is explored, which is extremely difficult when cognitive variables are explored.

When analyzing differences between the two forms of administration, as reported above, some statistical discrepancies were observed. Home-to-home participants had more years of formal education, presented better MMSE and were more frequently CH cases (and less MCI) than their counterparts. This unbalanced distribution of factors could be interpreted as a potential bias when interpreting results. However, invariance analysis results evidenced that these differences are irrelevant when determining the homogeneity of the two forms of administration, both in structure as in interpretation. Finally, we were not able to measure feasibility of the home-to-home procedure, due to an impending and crucial need to focus all the efforts on validating the NBACEtn battery. Further studies incorporating acceptance, adherence, and motivational aspects with regard to home-to-home assessments in these participants could be considered. The results of this study could be applied only to the tests that were used and the population who participated in this study. Future studies should be done to see if teleneuropsychology can be used with other tests and different populations, such as those with some characteristics of this study, traumatic brain injury, epilepsy or intellectual disability. Thus, further research is needed to assess the reliability and validity of teleneuropsychology using the home-to-home version of the NBACE in larger groups and other clinical settings.

In conclusion, this study has demonstrated the validity of the NBACEtn, which may be a suitable neuropsychological assessment method similar to the NBACE, useful for the cognitive assessment of people with and without cognitive impairment. From now on, the NBACEtn may be a valid option for delivering neuropsychological visits to people who cannot attend medical appointments in person for any reason, without modifying the psychometric quality of the measure.

Footnotes

ACKNOWLEDGMENTS

We acknowledge all the individuals who participated in this study and their families for their involvement in our day-to-day clinical tasks, especially during the acute pandemic period; without their collaboration, this work would not have been possible. We also want to thank all the participating investigators and personnel from Fundació ACE for their close collaboration and continuous intellectual input.

Authors’ disclosures available online ().

References

Grosch

, Gottlieb

, Cullum

(2011) Initial practice recommendations for teleneuropsychology. Clin Neuropsychol 25, 1119–1133.

Benaque

, Gurruchaga

, Abdelnour

, Hernandez

, Cañabate

, Alegret

, Rodriguez

, Rosende-Roca

, Tartari

, Esteban

, Lopez

, Gil

, Vargas

, Mauleon

, Espinosa

, Ortega

, Sanabria

, Perez

, Alarcon

, Gonzalez-Perez

, Marquie

, Valero

, Tarraga

, Ruiz

, Boada

, Morera

, Guitart

, Gailhajanet

, Tantinya

, Moreno

, Preckler

, Lafuente

, Aguilera

, Buendia

, Pancho

, Ibarria

, Diego

, Jofresa

, Roberto

, De Rojas

, Moreno-Grau

, Montrreal

, Martin

(2020) Dementia care in times of COVID-19: Experience at Fundacio ACE in Barcelona, Spain. J Alzheimers Dis 76, 33–40.

Soto-Pérez

, Franco-Martín

(2019) Atención psicológica y tecnologías: Oportunidades y conflictos. Rev Iberoam Psicol 11, 114–124.

Bernardo-Ramos

, Franco-Martín

, Soto-Pérez

(2012) Cyber-Neuropsychology: Application of new technologies in neuropsychological evaluation. Actas Esp Psiquiatr 40, 308–314.

Galusha-Glasscock

, Horton

, Weiner

, Cullum

(2016) Video teleconference administration of the repeatable battery for the assessment of neuropsychological status. Arch Clin Neuropsychol 31, 8–11.

Munro Cullum

, Hynan

, Grosch

, Parikh

, Weiner

(2014) Teleneuropsychology: Evidence for video teleconference-based neuropsychological assessment. J Int Neuropsychol Soc 20, 1028–1033.

Wadsworth

, Dhima

, Womack

, Hart

, Weiner

, Hynan

, Cullum

(2017) Validity of teleneuropsychological assessment in older patients with cognitive disorders. Arch Clin Neuropsychol 33, 1040–1045.

Parikh

, Grosch

, Graham

, Hynan

, Weiner

, Shore

, Cullum

(2013) Consumer acceptability of brief videoconference-based neuropsychological assessment in older individuals with and without cognitive impairment. Clin Neuropsychol 27, 808–817.

Loh

P-K

, Donaldson

, Flicker

, Maher

, Goldswain

(2007) Development of a telemedicine protocol for the diagnosis of Alzheimer’s disease. J Telemed Telecare 13, 90–94.

10.

Harrell

, Wilkins

, Connor

, Chodosh

(2014) Telemedicine and the evaluation of cognitive impairment: The additive value of neuropsychological assessment. J Am Med Dir Assoc 15, 600–606.

11.

Wong

, Martin-Khan

, Rowland

, Varghese

, Gray

(2012) The Rowland Universal Dementia Assessment Scale (RUDAS) as a reliable screening tool for dementia when administered via videoconferencing in elderly post-acute hospital patients. J Telemed Telecare 18, 176–179.

12.

Barton

, Morris

, Rothlind

, Yaffe

(2011) Video-telemedicine in a memory disorders clinic: Evaluation and management of rural elders with cognitive impairment. Telemed J E Health 17, 789–793.

13.

Wong

, Anand

, Chapman

, Rackley

, Zientz

(2009) When nouns and verbs degrade: Facilitating communication in semantic dementia. Aphasiology 23, 286–301.

14.

Brearly

, Shura

, Martindale

, Lazowski

, Luxton

, Shenal

B V.

, Rowland

(2017) Neuropsychological test administration by videoconference: A systematic review and meta-analysis. Neuropsychol Rev 27, 174–186.

15.

Hantke

, Gould

(2020) Examining older adult cognitive status in the time of COVID-19. J Am Geriatr Soc 68, 1387–1389.

16.

Jin

, Bai

, He

, Wu

, Liu

, Han

, Liu

, Yang

(2020) Gender differences in patients with COVID-19: Focus on severity and mortality. Front Public Health 8, 1–6.

17.

Wang

, Li

, Barbarino

, Gauthier

, Brodaty

, Molinuevo

, Xie

, Sun

, Yu

, Tang

, Weidner

, Yu

(2020) Dementia care during COVID-19. Lancet 395, 1190–1191.

18.

Alegret

, Espinosa

, Valero

, Vinyes-Junqué

, Ruiz

, Hernández

, Rosende-Roca

, Mauleón

, Becker

, Tárraga

, Boada

(2013) Cut-off scores of a Brief Neuropsychological Battery (NBACE) for Spanish individual adults older than 44 years old. PLoS One 8, e76436.

19.

Alegret

, Cuberas-Borrós

, Vinyes-Junqué

, Espinosa

, Valero

, Hernández

, Roca

, Ruíz

, Rosende-Roca

, Mauleón

, Becker

, Castell-Conesa

, Tárraga

, Boada

(2012) A two-year follow-up of cognitive deficits and brain perfusion in mild cognitive impairment and mild Alzheimer’s disease. J Alzheimers Dis 30, 109–120.

20.

Espinosa

, Alegret

, Valero

, Vinyes-Junqué

, Hernández

, Mauleón

, Rosende-Roca

, Ruiz

, López

, Tárraga

, Boada

(2013) A longitudinal follow-up of 550 mild cognitive impairment patients: Evidence for large conversion to dementia rates and detection of major risk factors involved. J Alzheimers Dis 34, 769–780.

21.

Alegret

, Cuberas-Borrós

, Espinosa

, Valero

, Hernández

, Ruíz

, Becker

, Rosende-Roca

, Mauleón

, Sotolongo

, Castell-Conesa

, Roca

, Tárraga

, Boada

(2014) Cognitive, genetic, and brain perfusion factors associated with four year incidence of Alzheimer’s disease from mild cognitive impairment. J Alzheimers Dis 41, 739–748.

22.

Alegret

, Espinosa

, Vinyes-Junqué

, Valero

, Hernández

, Tárraga

, Becker

, Boada

(2012) Normative data of a brief neuropsychological battery for Spanish individuals older than 49. J Clin Exp Neuropsychol 34, 209–219.

23.

Boada

, Tárraga

, Hernández

, Valero

, Alegret

, Ruiz

, Lopez

, Becker

(2014) Design of a comprehensive Alzheimer’s disease clinic and research center in Spain to meet critical patient and family needs. Alzheimers Dement 10, 409–415.

24.

Putnick

, Bornstein

(2016) Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Dev Rev 41, 71–90.

25.

Folstein

, Folstein

, McHugh

(1975) A practical state method for. J Psychiatr Res 12, 189–198.

26.

Blesa

, Pujol

, Aguilar

, Santacruz

, Bertran-Serra

, Hernández

, Sol

, Peña-Casanova

, Soler

, Zabay

, Riera

, Castellví

, Torner

, Charques

, Toirán

, Manero

, Peter Böhm

, Martí

, Meza

, Crespo

(2001) Clinical validity of the “mini-mental state” for Spanish speaking communities. Neuropsychologia 39, 1150–1157.

27.

Morris

(2012) The Clinical Dementia Rating (CDR): Current version and scoring rules. Neurology 43, 2412.

28.

Aguilar

, Santacruz

, Hernández

, Insa

, Pujol

, Sol

, Blesa

, Normacodem

(1997) Normalización de instrumentos cognitivos y funcionales para la evaluación de la demencia (NORMACODEM) (I): Objetivos, contenidos y población. Neurologia 12, 61–68.

29.

Blessed

, Tomlinson

, Roth

(1968) The association between quantitative measures of dementia and of senile change in the cerebral grey matter of elderly subjects. Br J Psychiatry 114, 797–811.

30.

Wechsler

, Pereña

(2004) WMS-III: Escala de Memoria de Wechsler-III. TEA Ediciones SA, Madrid.

31.

Petersen

(2004) Mild cognitive impairment as a clinical entity and treatment target. Arch Neurol 62, 1160–1163.

32.

Asociación Estadounidense de Psiquiatría (2005) Manual Diagnostico y Estadístico de los Trastoronos Mentales. Masson SA, Barcelona.

33.

Weschler

(2002) Escala de Inteligencia deWeschler para Adultos-III. TEA Ediciones SA, Madrid.

34.

Peña-Casanova

, Aguilar

, Santacruz

, Bertran-Serra

, Hernández

, Sol

, Pujol

, Blesa

(1997) [Adaptation and normalization of the Alzheimer’s disease Assessment Scale for Spain (NORMACODEM) (II)]. Neurologia 12, 69–77.

35.

Peña-Casanova

, Meza

, Bertran-Serra

, Manero

, Espel

, Martí

, Böhm

, Blesa

, Guardia

(1997) [Shortened version of the Barcelona test (III): Criterion validity with the ADAS-Cog]. Neurologia 12, 117–119.

36.

Goodglass

, Kaplan

(1983) The Assessment of Aphasia and Related Disorders (2nd edition). Lea and Febiger, Philadelphia, PA.

37.

Kaplan

, Goodglass

H WS

(1983) The Boston Naming Test. Lea Febiger, Philadelphia.

38.

Della Sala

, Laiacona

, Trivelli

, Spinnler

(1995) Poppelreuter-Ghent’s overlapping figures test: Its sensitivity to age, and its clinical use.. Arch Clin Neuropsychol 10, 511–534.

39.

Golden

(1980) In reply to Adams’s “In search of Luria’s battery: A false start.”. J Consult Clin Psychol 48, 517–521.

40.

Erzigkeit

(1989) The SKT — a short cognitive performance test as an instrument for the assessment of clinical efficacy of cognition enhancers. In Diagnosis and Treatment of Senile Dementia, Bergener M., Reisberg B, eds. Springer, Berlin, Heidelberg.

41.

Piatt

, Fields

, Paolo

, Koller

, Tröster

(1999) Lexical, semantic, and action verbal fluency in Parkinson’s disease with and without dementia. J Clin Exp Neuropsychol 21, 435–443.

42.

del Ser Quijano

, Sánchez Sánchez

, García de Yébenes

, Otero Puime

, Zunzunegui

M V

, Muñoz

(2004) [Spanish version of the 7 Minute screening neurocognitive battery. Normative data of an elderly population sample over 70]. Neurologia 19, 344–358.

43.

Alegret

, Vinyes-Junqué

, Boada

, Martínez-Lage

, Cuberas

, Espinosa

, Roca

, Hernández

, Valero

, Rosende-Roca

, Mauleón

, Becker And

, Tárraga

(2010) Brain perfusion correlates of visuoperceptual deficits in mild cognitive impairment and mild Alzheimer’s disease. J Alzheimers Dis 21, 557–567.

44.

Pillon

, Dubois

, Bonnet

, Esteguy

, Guimaraes

, Vigouret

, Lhermitte

, Agid

(1989) Cognitive slowing in Parkinson’s disease fails to respond to levodopa treatment: The 15-objects test. Neurology 39, 762–768.

45.

Zigmond

, Snaith

(1983) The hospital anxiety and depression scale. Acta Psychiatr Scand 67, 361–370.

46.

De las Cuevas Castresana

, Garcia-Estrada Perez

, Gonzalez de Rivera

(1995) “ Hospital anxiety and depression scale” y psicopatología afectiva. An Psiquiatr 11, 126–130.

47.

Randolph

, Tierney

, Mohr

, Chase

(1998) The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): Preliminary clinical validity. J Clin Exp Neuropsychol 20, 310–319.

48.

Muntal-Encinas

, Gramunt-Fombuena

, Badenes-Guia

, Casas-Hernanz

, Aguilar-Barbera

(2012) Traducción y adaptación española de la batería neuropsicológica Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) forma A en una muestra piloto. Neurologia 27, 531–546.

49.

Piatt

, Fields

, Paolo

, Koller

, Tröster

(1999) Lexical, semantic, and action verbal fluency in Parkinson’s disease with and without dementia. J Clin Exp Neuropsychol 21, 435–443.

50.

Alegret

, Peretó

, Pérez

, Valero

, Espinosa

, Ortega

, Hernández

, Mauleón

, Rosende-Roca

, Vargas

, Rodríguez-Gómez

, Abdelnour

, Berthier

, Bak

, Ruíz

, Tárraga

, Boada

(2018) The role of verb fluency in the detection of early cognitive impairment in Alzheimer’s disease. J Alzheimers Dis 62, 611–619.

51.

Ricolfi

, Testa

(2020) Factorial invariance and orthogonal rotation. Multivariate Behav Res, doi: 10.1080/00273171.2020.1770571.

52.

Barrett

(2007) Structural equation modelling: Adjudging model fit. Pers Individ Dif 42, 815–824.

53.

Marsh

, Wen

, Hau

K-T

(2004) Structural equation models of latent interactions: Evaluation of alternative estimation strategies and indicator construction. Psychol Methods 9, 275–300.

54.

Español-Martín

, Pagerols

, Prat

, Rivas

, Sixto

, Valero

, Artigas

, Ribasés

, Ramos-Quiroga

, Casas

, Bosch

(2020) Strengths and difficulties questionnaire: Psychometric properties and normative data for Spanish 5- to 17-year-olds. Assessment, doi: 10.1177/1073191120918929.

55.

Dimitrov

(2010) Testing for factorial invariance in the context of construct validation. Meas Eval Couns Dev 43, 121–149.

56.

Chen

(2007) Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct Equ Model A Multidiscip J 14, 464–504.

57.

Bishop

(1999) Handbook of normative data for neuropsychological assessment; Mitrushina

, Boone

, D’Elia LF

, eds. Oxford University Press, New York.

58.

Petersen

(2004) . Mild cognitive impairment as a diagnostic utility. J Intern Med 256, 183–194.

59.

Meade

, Lautenschlager

(2004) A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organ Res Methods 7, 361–388.

60.

Mrazik

, Millis

, Drane

(2010) The oral trail making test: Effects of age and concurrent validity. Arch Clin Neuropsychol 25, 236–243.

61.

Alegret

, Muñoz

, Roberto

, Rentz

, Valero

, Gil

, Marquié

, Hernández

, Riveros

, Sanabria

, Perez-Cordon

, Espinosa

, Ortega

, Mauleón

, Abdelnour

, Rosende-Roca

, Papp

K V.

, Orellana

, Benaque

, Tarraga

, Ruiz

, Boada

(2020) A computerized version of the Short Form of the Face-Name Associative Memory Exam (FACEmemory®) for the early detection of Alzheimer’s disease. Alzheimers Res Ther 12, 1–11.