Cognitive Stimulation of Elderly Individuals with Instrumental Virtual Reality-Based Activities of Daily Life: Pre-Post Treatment Study

Abstract

As the demographic structure in western societies ages, the prevalence and impact of cognitive decline rises. Thus, new solutions to tackle this problem are required. The use of Information and Communication Technologies (ICT)-based cognitive exercises has emerged in the last few decades, though with inconsistent results. Hence, we conducted a pre-post treatment study to further investigate this approach. We designed a set of virtual reality exercises that mimic activities of daily living by which the patient can train different cognitive domains. Twenty-five participants, ages 65–85, underwent 12 training sessions between the pre-treatment and post-treatment assessments. Significant increases were seen between the two assessments for some of the neuropsychological measures: visual memory, attention, and cognitive flexibility. Results also suggest that participants with lower baseline cognitive performance levels improved most after these sessions.

Introduction

As the population ages in developed countries, namely in Europe (in 2030 ∼ 25 percent of EU citizens will be over 65 years old¹), new challenges arise from the growing burden of age-related conditions and diseases on national health systems and consequent needs to develop better treatments for those conditions. The negative impact of aging on cognitive functioning, which is our focus, is well known.²

Studies testing traditional techniques to improve cognitive functioning have shown some therapeutic success. A systematic review³ found that cognitive stimulation, particularly in dementia patients, promotes gains in cognitive function as well as in both general wellbeing and quality of life (QoL). Similar results were found in another review.⁴

One alternative to traditional strategies and exercises is the use of ICT (Information and Communication Technologies) programs. The evidence to support ICT-based solutions to overcome age-related cognitive decline is, nevertheless, not yet robust⁵: modest improvements were found in studies involving computerized cognitive training (CCT). Cognitive domain and design choice were the primary factors behind the efficacy of the specific technique used.⁶ A set of seven randomized controlled trial studies (RCTs) tested a variety of techniques to enhance memory, executive reasoning, and processing speed; a meta-analysis of its results showed that cognitive training had some long-lasting effects on performance.⁷

In a literature review including 10 RCTs, patients in only half of the trials reviewed showed cognitive improvement, with no visible long-lasting or crossover effects in domain functionality; the authors of the review concluded that there was no evidence that structured cognitive interventions are effective.⁸ Another systematic review on cognitive and memory training in adults at risk of dementia highlighted some promising results but suggested that more RCTs are required to provide robust support for the success of these techniques.⁹ Another review highlighted the significant effect of such therapies on working memory, with clearer results in patients with less prior training; furthermore, the results show promising effects in the transfer of skills from the training context to everyday living.¹⁰ Authors of a review devoted specifically to computer-based approaches⁶ also found interesting results to support these solutions, but they warned against using these solutions outside of a broader rehabilitation program supervised by specialists. A more positive outlook is presented in a review in which the authors conclude that the computerized training is at least as effective as traditional solutions.¹¹ In that same review, the authors found that most of the issues raised when addressing computer-based solutions are not confirmed by the results. Furthermore, these solutions were found to often provide swifter, less labor-intensive, and more effective alternatives to traditional approaches.

It is now common to find serious games (SG) allied to virtual reality (VR). SGs have all the properties of their less serious cousins. However, these games are designed not for the entertainment of players, but to produce a positive impact on the player's mental and physical health.¹² VR/SG apps simulate real-life environments. Unlike traditional paper-and-pencil instruments, these apps provide ecological validity to assessment and stimulation. They also provide behavioral indicators to the therapist by monitoring and recording participants' performance during the VR activities.¹³ Larson et al. provide a review of the advantages and disadvantages of ICT-based procedures for mental health purposes.¹⁴

A meta-analysis¹⁵ was conducted specifically on the use of VR as a therapy for adults with brain trauma. The systems ranged in quality, from fully immersive systems to cheap online games or videogames. In before-after comparisons, improvements in balance emerged in four case studies and two small RCTs. In between-group comparisons in these RCTs, no differences were seen between VR and traditional therapy. While the authors found that the VR applications improved cognitive function and that participants rated them more positively than traditional techniques, they found no difference in success rates between VR-based and traditional techniques. A more specific review of VR rehabilitation shows, however, very promising results with this approach, with significant clinical improvements in both memory and attention outcomes.¹⁶ Authors of a review found that these solutions showed long-term, significantly positive effects on cognitive performance and psychological outcomes, but they indicated the need for larger samples, quality of life (QoL) assessment, and longitudinal design.¹⁷ These concerns were also voiced in another review,¹⁸ which further pointed out the lack of clinical guidelines and standardized treatment protocols; these deficiencies, in turn, limit the potential for a broader implementation of these therapies. Most reviews, however, agree on the potential for effective use of VR/SG solutions. Moreover, compared to traditional therapies, VR/SG techniques demonstrate faster and better results for cognitive rehabilitation.¹⁹

In the current study, to contribute to the development of a structured program for cognitive intervention focusing on the elderly, we explored the effects of a cognitive stimulation using VR/SG within a multidomain program that also includes the promotion of social skills and informatics training for elderly people.

Methods

Sample

Twenty-five participants (21 women) of the Day Care Centre of the Benfica Parish, Lisbon, between the ages of 65 and 74 (M = 74; standard deviation [SD] = 5.27; ranging from 65 to 85 years) were recruited at the center and participated in this study. Their average education level was 6 years of school (SD = 2.42; ranging from 4 to 12 years). Exclusion criteria were as follows: (a) having been clinically diagnosed with a psychiatric or neurological disorder and (b) a history of alcohol or substance abuse. Two participants were excluded because of depression, and one participant, because of transient ischemic attack.

Measures

The neuropsychological measures used in this study are all well-established measures with good psychometric properties: the Montreal Cognitive Assessment (MoCA),²⁰ used for cognitive screening; the Frontal Assessment Battery (FAB),²¹ a set of six subtests that assess executive functioning; the Wisconsin Card Sorting Test (WCST),²² commonly used to assess executive functions and cognitive flexibility; the Rey Complex Figure,²³ used to assess visuo-constructive abilities and visual memory; and d2,²⁴ used to measure attention/concentration. Depression and functionality were assessed with self-report scales: The Beck Depression Inventory II (BDI-II)²⁵ for depressive symptoms, and the Everyday Competence Questionnaire²⁶ for functionality.

Procedure

This study was carried out between April 2017 and June 2017. The participants were selected for cognitive stimulation after enrollment in the multi-domain Memória Ativa Senior (MAS) program. The MAS is a project of the Psychology Department of the Junta de Freguesia de Benfica (one of the 24 civil parishes in Lisbon, Portugal). The MAS has three main activities: (a) informatics—teaching participants basic computer skills and training them to use social networks to chat with their relatives; (b) cognitive stimulation—promoting cognitive functioning through training in daily life activities in VR/SG-based scenarios; (c) social skills—teaching participants skills for recognizing and managing emotions, for communication, and for solving problems.

After their initial neuropsychological assessments (two 1-hour sessions), the participants were familiarized with the desktop VR system (non-immersive) during two 20-minute sessions. The same system used for training was used for the intervention. A desktop system is better for older patients, whose tendency for weaker eyesight leaves them more prone to cybersickness when using head-mounted display (HMD) VR headsets. The virtual environments were developed with Unity3d. The application was run on a 15.6′′ Asus X555L laptop with 2GB Geforce 920M graphic board. One training scenario consisted of a maze in which the participants had to find their way to the exit while collecting several items placed throughout the virtual environment. In the other scenario, the participants had to navigate a virtual city by following directions for specific locations. The cognitive intervention was performed twice a week for a total of twelve 30-minute sessions. One of two trained psychologists assisted each participant throughout the sessions.

The cognitive stimulation in the VR environment included several daily life activities designed to train cognitive functions, such as attention tasks (e.g., selecting ingredients to bake a cake), working memory tasks (e.g., shopping at a grocery store), auditory memory tasks (e.g., listening to and remembering news on TV), executive functions (e.g., selecting the appropriate clothes to wear, arranging shoes in a shoe closet), as shown in Figure 1—from top-left clockwise, the bathroom, living room in the apartment, virtual kitchen test, shoe closet test, bedroom, outdoor environment, pharmacy, and art gallery test. The level of difficulty for each type of task was gradually increased throughout the sessions, so that, for instance, during the first session the participants had to complete simple tasks, and during the last session they had to complete more complex tasks (e.g. to buy several items in the grocery store). The detailed description of the intervention protocol is presented in Supplementary Table S1; Supplementary Data are available online at www.liebertpub.com/cyber).

FIG. 1.

Screenshots depicting some of the tasks used in VR/SG cognitive stimulation. VR/SG, virtual reality/serious games.

Statistical analysis

To uncover significant changes in actual individual outcomes, we calculated Reliable Change Indices (RCIs) individually for each neuropsychological outcome.²⁷ The RCI approach standardizes individual changes according to the standard error of difference, which accounts for test-retest stability in the calculation of a standardized change between test scores.²⁸

Test-retest scores were then calculated for each neuropsychological variable to estimate the stability over time of each measure used in the calculation of the RCI. This approach was developed as an alternative to the limitations of comparisons between subjects and of situations in which cases cannot easily be matched with controls in their individual characteristics.²⁹

The resulting Z-score in RCI measures the performance of the participant during post-treatment assessment relative to their baseline for each of the measures. Accordingly, a change exceeding Z ± 1.96 (p < 0.05) reflects a reliable neuropsychological change between test scores for a 95 percent confidence interval (CI) (2.5 percent in a positive direction and 2.5 percent in a negative direction for a two-tailed test − [α/2 at 95 percent CI]).

The RCI is given by the following expression²⁵: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} RC = { \frac { x2 - x1 } { { S_ { diff } } } } \end{align*} \end{document}

where x 2 and x 1 are the scores in each neuropsychological measure respectively for post- and pre-treatment assessments, where S_diff reflects the standard error of difference for test scores: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} {S_{diff}} = \sqrt {2{{ \left( {{S_E}} \right) }^2}} \end{align*} \end{document}

which accounts for the standard error of measurement (S_E): \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} {S_E} = {S_1} \sqrt {1 - {r_{xx}}} \end{align*} \end{document}

where the S₁ is the SD at pre-treatment, whereas r_xx represents the test-retest reliability.

The RCIs were calculated individually for each neuropsychological measure. On the basis of 1,000 bootstrap samples, the correlations between the RCI and variables such as age, education and global cognitive ability at pre-treatment were estimated with 95 percent CIs, a level chosen to account for the small sample size.

Results

Mean change in each neuropsychological domain for the overall sample

Means and SDs were calculated for both the pre-treatment and the post-treatment assessments to provide an understanding of the mean change in the overall sample (Table 1). The changes in the mean values for the neuropsychological measures suggested that most of these changes occurred in the expected direction. The mean scores of the screening measures (i.e., MoCA and FAB) increased slightly although non-significantly between the pre-treatment and post-treatment assessments, specifically in the FAB, in which the mean total score changed from a clinical score at the time of pre-treatment assessment to a normative score (i.e., above 15 points) at the time of post-treatment assessment. The differences in all the mean scores were tested with a paired sample t-test, which showed significant differences between pre-treatment and post-treatment assessments for the number of perseverative errors of WCST (t[21] = 2.493; p = 0.021), the score in the memory trial of RCF (t[22] = −3.991; p = 0.001), error percentage (t[16] = 3.463; p = 0.003), and concentration index of d2 (t[16] = −2.926; p = 0.010).

Table 1.

Descriptive Statistics and Test-Retest Scores for the Neuropsychological Variables

	Pre-treatment		Post-treatment
	M₁	S₁	M₂	S₂	t	r_xx
MoCA	22.88	3.05	23.00	4.00	−0.244	0.79
FAB	14.68	2.28	15.48	1.27	−1.528	0.33
WCST trials	125.42	7.70	119.83	13.83	1.687	0.36
WCST errors	57.21	20.19	47.00	21.82	1.762	0.18
WCST P errors	32.21	20.15	23.87	13.26	2.493^*	0.51
WCST categories	2.96	2.01	3.87	1.76	−1.622	0.35
RCF copy score	28.88	5.42	28.74	5.16	0.763	0.67
RCF copy time	5.71	3.07	5.96	3.50	0.775	0.57
RCF memory score	11.98	5.24	14.98	5.74	−3.991^**	0.80
RCF memory time	3.01	1.77	3.35	1.97	−0.609	0.49
d2 targets processed	337.95	77.13	316.70	79.63	1.106	0.18
d2 errors, %	14.57	14.42	4.23	5.20	3.463^**	0.60
d2 global performance	286.32	55.72	303.91	66.32	−1.087	0.41
d2 concentration index	86.84	52.74	118.52	23.67	−2.926^*	0.57
ECQ	35.20	5.16	33.56	3.35	1.459	0.42
BDI II	10.28	6.31	10.78	8.31	0.048	0.79

M₁, mean values at pre-treatment; S₁, SD at pre-treatment; M₂, mean values at post-treatment; S₂, SD at post-treatment; r_xx, test-retest reliability.

p < 0.05.

p < 0.01.

BDI-II, Beck Depression Inventory-II; ECQ, Everyday Competence Questionnaire; FAB, Frontal Assessment Battery; MoCA, Montreal Cognitive Assessment; RCF, Rey complex figure; RCF memory score, Rey complex figure memory trial total score; SD, standard deviation; WCST, Wisconsin Card Sorting Test (values are percentages, except the number of categories which is the raw score); WCST P errors, perseverative errors in the WCST in percentage.

Test-retest reliability scores

Participants in this sample were retested 3 months after their initial tests. The same neuropsychological instruments were used in both assessments. These instruments, which are well established, have been extensively studied in the geriatric population. Table 1 depicts the test-retest scores (r_xx) for our sample. These scores were below acceptable (<0.50) for some neuropsychological measures. The significant outcomes had acceptable scores for test-retest reliability ranging from 0.51 (number of P errors of WCST) to 0.80 (score in memory trial of RCF).

Reliable change index

The RCIs were calculated for the significant outcomes in accordance with the statistical approach used to measure change.²⁷ The resulting variables in which the better treatment outcomes were given by a decrease in test scores (i.e., error percentages) were reverse scored, so that positive scores would describe improvements while negative scores would describe deteriorations.

Figure 2 shows an asymmetry in the distribution of positive and negative changes across measures, reflecting a higher proportion of positive change across the outcomes of the study. The proportion of positive and negative changes that were unlikely to occur by chance at a 95-percent CI is depicted in Table 2. Changes that were positive (Z > 1.96) or negative (Z < 1.96) in more than 2.5 percent (α/2 for two-tailed predictions) would probably be due to a treatment effect, because only the extreme outcomes were considered significant changes. Table 2 shows that improvements (Z > 1.96) were observed in 9 percent of the sample for WCST (perseverative errors), 22 percent for the RCF (score in the memory trial), 12 percent in error percentage, and 13 percent in the concentration index of d2. None of the cases showed a significant negative change in neuropsychological outcomes.

FIG. 2.

Percentages of cases for positive and negative change (RCI). RCI, Reliable Change Index.

Table 2.

Reliable Change Index

	(+) Change Z > 1.96	(−) Change Z < 1.96
WCST P errors	9	0
RCF memory score	22	0
d2 errors	12	0
d2 concentration index	13	0

Values are in percentages.

Correlates of cognitive performance

We also explored whether these improvements were related to individual characteristics such as age, education, or general cognitive ability at baseline. Bivariate correlations were used with bootstrap sampling. The results show a significant correlation between the improvements in the error percentage of d2 and the total score of the MoCA ( r = −0.44; bootstrapped 95 percent CI [−0.23 to −0.74]; SE = 0.13), which suggests that the magnitude of improvements is highest for individuals with lower levels of cognitive functioning at baseline.

Discussion

Our study was carried out with elderly participants who had not been clinically diagnosed with cognitive deficits. Since the elderly are generally vulnerable to cognitive decline, we tested whether a 3-month cognitive stimulation program would help improve cognitive functioning in this population. Participants showed statistically significant improvements in attention (d2), in visual memory (Rey Complex Figure), and on two indicators of cognitive flexibility (WCST), as shown by group analysis.

A control group would have allowed us to discern the effects of confounding variables (e.g., practice effects) on treatment outcomes. Because we lacked a control group, we adopted a more conservative statistical approach based on RCI. This approach has been used in a variety of clinical research studies because of its ability to detect how much each individual has changed between the pre-treatment and post-treatment assessments while controlling for the standard error of difference between test and retest scores.^28,29 This change results in a standardized score, in which scores above or below Z = 1.96 are less likely to occur by chance (i.e., only in 5 percent of the sample for a 95 percent CI). A higher percentage than 2.5 percent (α/2) of improved cases was observed on cognitive flexibility (9 percent), attention/concentration (12 percent to 13 percent) and visual memory (22 percent). These results show that there were strong improvements in main domains involved in the VR/SG program.

The tasks that were used in the cognitive stimulation program are general tasks that mimic daily life activities but are not specific, focused tasks for any of these domains. However, the specific functions in which significant differences were observed, that is, attention, memory, and cognitive flexibility, are required in daily activities, such as shopping at the supermarket, preparing food, and choosing the right clothes to wear; in most sessions, these activities were used to train participants. Conversely, no significant differences were observed among the results of the screening tests (MoCA and FAB), which assess general functioning; this finding suggests that these improvements were restricted to specific domains of cognitive functioning and are consistent with the results of recent reviews showing improvements in attention and memory as specific effects of ICT-based interventions.¹⁶

This finding contrasts with a previous study, in which a similar VR/SG approach was used but in which we used a clinical sample diagnosed with cognitive deficits. In that study, we found significant improvements in executive functioning, which was assessed with the FAB.³⁰ However, in that study, the sample consisted of patients undergoing community treatment for alcohol use disorder, which has a different pattern of cognitive deficits from the sample in our study.

We explored the data set further to determine the relationships between the RCIs and individual characteristics such as age, education, and general cognitive functioning; the results suggest that the improvements found in attention (through the decreases in error percentage in d2) were negatively associated with general cognitive functioning at baseline. This suggests that if improvements in attention are the main aim, the most efficient use of this program is to target patients showing some signs of cognitive decline. No effects of age or education were found on RCIs within the limited range of both in our sample.

Our results should be interpreted with some caution, since our study lacked a control group. We sought to minimize this problem by transforming the outcomes to RCIs, which is a more conservative approach to the analysis of pre-post-treatment effects. This does not allow us to discern the effects of practice on repeated test scores. We should note, however, that practice effects are more likely to be seen in measures that involve problem-solving abilities;³¹ problem-solving was not required in the evaluations of memory and attention, which were assessed using the RCF and the d2, respectively. Also, the 3-month post-treatment interval may have reduced the practice effects of repeated exposure to these tests. One way of overcoming this limitation would be to use a standard regression-based change score approach from an independent normative group drawn from a different study³² that would account for practice effects; however, there were no data available for most of the tests used in the current study.

The results of our cognitive flexibility, attention, and memory tests are consistent with recent reviews^17,18 and highlight the advantages of using VR/SG for cognitive intervention, even with populations that may be unfamiliar with the use of such technology, such as the elderly. As technology keeps progressing, there is great potential for the improvement and increased ecological validity of VR/SG.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

References

Giannakouris

. (2010) Regional population projections EUROPOP2008: Most EU regions face older population profile in 2030. Statistics in Focus, 1/2010. Luxembourg: Publications Office of the European Union.

Hedden

, Gabrieli

. Insights into the ageing mind: A view from cognitive neuroscience. Nature Reviews Neuroscience, 2004; 5:87–96.

Aguirre

, Woods

, Spector

, et al. Cognitive stimulation for dementia: A systematic review of the evidence of effectiveness from randomised controlled trials. Ageing Research Reviews, 2013; 1:253–262.

Elliott

, Parente

. Efficacy of memory rehabilitation therapy: A meta-analysis of TBI and stroke cognitive rehabilitation literature. Brain Injury, 2014; 28:1610–1616.

Ballesteros

, Kraft

, Santana

, et al. Maintaining older brain functionality: A targeted review. Neuroscience and Biobehavioral Reviews, 2015; 55:453–477.

Lampit

, Hallock

, Valenzuela

. Computerized cognitive training in cognitively healthy older adults: A systematic review and meta-analysis of effect modifiers. PLoS Medicine, 2014; 11:e1001756.

Valenzuela

, Sachdev

. Can cognitive exercise prevent the onset of dementia? Systematic review of randomized clinical trials with longitudinal follow-up. The American Journal of Geriatric Psychiatry, 2009; 17:179–187.

Papp

, Walsh

, Snyder

. Immediate and delayed effects of cognitive interventions in healthy elderly: A review of current literature and future directions. Alzheimer's and Dementia, 2009; 5:50–60.

Gates

, Sachdev

, Fiatarone Singh

, et al. Cognitive and memory training in adults at risk of dementia: A systematic review. BMC Geriatrics, 2011; 11:55.

10.

Johansson

, Tornmalm

. Working memory training for patients with acquired brain injury: effects in daily life. Scandinavian Journal of Occupational Therapy, 2011; 19:176–183.

11.

Kueider

, Parisi

, Gross

, et al. Computerized cognitive training with older adults: A systematic review. PLoS One, 2012; 7:e40588.

12.

Gamito

, Oliveira

, Morais

, et al. (2011) Serious games for serious problems: from Ludicus to Therapeuticus. In Kim

, ed. Virtual reality. London, UK: IntechOpen, pp. 527–548.

13.

Gamito

, Oliveira

, Alghazzawi

, et al. The art gallery test: A preliminary comparison between traditional neuropsychological and ecological VR-based tests. Frontiers in Psychology, 2017; 8:1911.

14.

Larson

, Feigon

, Gagliardo

, et al. Virtual reality and cognitive rehabilitation: A review of current outcome research. NeuroRehabilitation, 2014; 34:759–772.

15.

Pietrzak

, Pullman

, McGuire

. Using virtual reality and videogames for traumatic brain injury rehabilitation: A structured literature review. Games for Health Journal, 2014; 3:202–214.

16.

Shin

, Kim

. Virtual reality for cognitive rehabilitation after brain injury: A systematic review. Journal of Physical Therapy Science, 2015; 27:2999–3002.

17.

Coyle

, Traynor

, Solowij

. Computerized and virtual reality cognitive training for individuals at high risk of cognitive decline: Systematic review of the literature. The American Journal of Geriatric Psychiatry, 2015; 23:335–359.

18.

Bogdanova

, Yee

, Ho

, et al. Computerized cognitive rehabilitation of attention and executive function in acquired brain injury. Journal of Head Trauma Rehabilitation, 2016; 31:419–433.

19.

Rosa

, Sousa

, Faustino

, et al. (2017) The effect of virtual reality-based serious games in cognitive interventions: A meta-analysis study. In Proceedings of the 4th Workshop on ICTs for improving Patients Rehabilitation Research Techniques. Lisbon, Portugal: ACM, pp. 113–116.

20.

Freitas

, Simo∼es

, Alves

, et al. Montreal Cognitive Assessment (MoCA): Normative study for the Portuguese population. Journal of Clinical and Experimental Neuropsychology, 2011; 33:989–996.

21.

Lima

, Meireles

, Fonseca

, et al. The Frontal Assessment Battery (FAB) in Parkinson's disease and correlations with formal measures of executive functioning. Journal of Neurology, 2008; 255:1756–1761.

22.

Heaton

, Chelune

, Talley

, et al. (1993) Wisconsin card sorting test (WCST)—Manual revised and expanded. Odessa, FL: Psychological Assessment Resources.

23.

Osterrieth

. Filetest de copie d'une figure complex: contribution a l'etude de la perception et de la memoire [The test of copying a complex figure: a contribution to the study of perception and memory]. Archives de Psychologie, 1944; 30:286–356.

24.

Brickenkamp

. (2002). The d2 test of attention. Gottingen, Germany: Hogrefe Verlag GmbH & Co.

25.

Beck

, Steer

, Brown

. (1996) Manual for the Beck Depression Inventory II. San Antonio, TX: Psychological Corporation.

26.

Kalisch

, Richter

, Lenz

, et al. Questionnaire-based evaluation of everyday competence in older adults. Clinical Interventions in Aging, 2011; 6:37–46.

27.

Jacobson

, Truax

. Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 1991; 59:12–19.

28.

Merritt

, Bradson

, Meyer

, et al. Evaluating the test-retest reliability of symptom indices associated with the ImPACT post-concussion symptom scale (PCSS). Journal of Clinical and Experimental Neuropsychology, 2018; 40:377–388.

29.

Chelune

, Naugle

, Luders

, et al. Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology, 1993; 7:41–52.

30.

Gamito

, Oliveira

, Lopes

, et al. Executive Functioning in Alcoholics Following an mHealth Cognitive Stimulation Program: Randomized Controlled Trial. Journal of Medical Internet Research 2014, 16:e102.

31.

Dikmen

, Heaton

, Grant

, et al. Test-retest reliability and practice effects of Expanded Halstead-Reitan Neuropsychological Test Battery. Journal of the International Neuropsychological Society. 1999; 5:346–356.

32.

Cysique

, Franklin

, Abramson

, et al. Normative data and validation of a regression based summary score for assessing meaningful neuropsychological change. Journal of Clinical and Experimental Neuropsychology, 2011; 33:505–522.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB