A Videogame-Based Approach to Measuring Information Processing Speed in Multiple Sclerosis Patients

Abstract

Objective:

Slowing information processing speed (IPS) is a biomarker of neuronal damage in patients with multiple sclerosis (pwMS). A focus on IPS might be the ideal solution in the perspective of promptly detecting cognitive changes over time. We developed a tablet-based home-made videogame to test the sensitivity of this device in measuring subclinical IPS in pwMS.

Materials and Methods:

Forty-three pwMS without cognitive impairment and 20 healthy controls (HCs) were administered the videogame task with a tablet. Response times (RTs) and accuracy were recorded.

Results:

PwMS (mean RTs = 505.5 ± 73.9 ms) were significantly slower than HCs (mean RTs = 462.3 ± 40.3 ms, P = 0.014) on the videogame task. A moderate but significant correlation (r = −0.35, P = 0.03) between mean RTs and the Symbol Digit Modalities Test was observed.

Conclusion:

Our videogame showed good sensitivity in measuring IPS in apparently cognitive normal pwMS. Computerized testing might be useful in screening initial cognitive dysfunction that should be monitored as a marker of underlying disease progression.

IRB approval Number is 2332CESC.

Introduction

Multiple sclerosis (MS) is a chronic immune-mediated disease of the central nervous system (CNS), characterized by multifocal areas of inflammatory demyelination within both the white matter (WM) and the gray matter (GM), neurodegeneration, and axonal injury.¹

In addition to physical disability, cognitive impairment (CI) is common in people with multiple sclerosis (pwMS), with frequencies ranging from 43% to 70%.² CI can occur from the early stages of the disease,^3,4 playing a critical prognostic role in predicting disability progression and accumulation of cortical atrophy in the long term.^5,6 Furthermore, CI results in a considerable disruption to personal, occupational, and social functioning of pwMS, with a consequent detrimental effect on patients' quality of life.^7,8 Despite the paramount importance to promptly identify the presence of CI, initial changes in cognitive functioning are difficult to be detected, especially in the early stages of the disease.⁴

The gold standard for cognitive assessment in pwMS consists of paper-and-pencil tests that, might be relatively less sensitive in detecting initial cognitive difficulties in those individuals who do not manifest overt signs of cognitive impairment.^4,9

By contrast, computer-based behavioral assessment can offer numerous advantages in terms of sensitivity with respect to paper-and-pencil tests. Several studies provided evidence on behalf of the adoption of computerized tasks to improve the detection of subtle signs of cognitive dysfunctions in different clinical populations.^10–12 The major advantage is that computer-based tasks offer the possibility to simultaneously manipulate and control several variables, thus providing the opportunity to design multifaceted and flexible conditions that better simulate complex real-life demands. Furthermore, computer-based tasks enable to record response times (RTs) within a millisecond precision, a parameter potentially highly sensitive to capture small changes in processing speed capacities.

Slowing in information processing speed (IPS) is one of the most common behavioral signs in pwMS,^13–15 whose impairment can exert a disturbing effect on the effective functioning of other cognitive abilities. The neuronal underpinnings of IPS have been widely investigated and a consensus has been reached over the relationship between reduced IPS and both structural and functional brain damages related to MS.^16–18

In the context of an underlying disease that determines WM and GM damage with consequent functional reorganization,¹⁹ IPS might account for the first marker of more global cognitive inefficiency.¹⁵ It has been argued that a decline in processing speed is likely one of the main and direct consequences of demyelination caused either by normal aging or by an underlying disease of the CNS.^20,21

Recently, the high potential of measurement of basic response speed with computerized testing has been recognized also in pwMS, with promising results.^18,22,23 Moreover, computerized testing could potentially be used as a practical tool to monitor cognitive changes of pwMS over time.²³

In this perspective, we developed a task for measuring IPS running on a videogame format. Videogaming is a popular form of leisure activity that in recent years has attracted scientific interests, mainly with reference to the field of rehabilitation.^24,25 Designing a novel videogame allowed us to implement tasks based on strong theoretical background but still characterized by high technological appeal. Furthermore, by designing our own videogame, it has been possible to implement a fine control over the stimuli, the graphical interface, the task's difficulty, as well as the outcome measures of interest. Our long-term aim is to develop a videogame to be used for a remote monitoring of pwMS cognitive performance over time, which could be easily self-administered at home.

With this preliminary study, we aimed primarily at exploring the ability of the videogame-based task to detect IPS in a group of pwMS classified as having no CI with respect to paper-and-pencil neuropsychological tests, investigating whether (1) the RTs recorded with the videogame task were a sensitive measure to detect signs of slower IPS in pwMS with respect to a group of healthy controls (HCs); (2) the RTs recorded with the videogame task correlated with the gold standard for IPS assessment in pwMS, that is, the Symbol Digit Modalities Test (SDMT).²⁶

Method

Participants

Seventy participants were enrolled for this study, consisting of 49 pwMS, recruited at the MS Centre of the Verona University Hospital (Verona, Italy), and 21 neurologically HCs.

Inclusion criteria for pwMS were diagnosis of MS,²⁷ no concomitant neurological conditions (other than MS), no psychiatric or other pathological health conditions, no substance abuse, no hearing or visual impairment, no relapses within the previous 6 months from enrollment, and absence of global CI. A neuropsychological assessment, composed of the Brief Repeatable Battery of neuropsychological tests²⁸ and the Stroop Test,²⁹ was available for all patients. The median time between neuropsychological assessment and videogame testing was 6.5 months. Neuropsychological assessment was carried out by experienced neuropsychologists. To be classified as having CI, pwMS had to fail three or more neuropsychological tests.³⁰

Inclusion criteria for HCs consisted of absence of neurological or psychiatric conditions, no substance abuse or medications, and normal or corrected-to-normal vision and hearing.

Owing to software technical problems during data collection, six pwMS and one HC were excluded before proceeding to statistical analyses. Hence, the final study sample was composed of 43 pwMS and 20 HCs.

Physical disability of pwMS was measured with the Expanded Disability Status Scale (EDSS).³¹ Functioning of upper limbs was assessed by an experienced neurologist as part of the neurological assessment; none of the patients had motor difficulties of the upper limbs.

At the time of videogame testing, all pwMS presented with a relapsing remitting (RR) form; 9 pwMS were not treated with specific disease-modifying therapy for MS, whereas 23 were treated with dimethyl fumarate, 5 with fingolimod, 1 with natalizumab, 2 with teriflunomide, 1 with interferon beta-1a, 1 with cladribine, and 1 with ocrelizumab.

The study was approved by the local ethic committee and all participants provided written informed consent before participation in the study.

The videogame

Graphics

All the software graphics were created using the Gimp software (www.gimp.org), a cross-platform, open-source image editor. Stimuli were designed by an expert designer and were composed of three frames each, which has been proven to be the optimal compromise between fluidity and reactivity with respect to user interaction.

Programming

The software was written in C and Java programming languages, to have a working prototype to adapt to different interfaces and videogame levels. After several tests, the software architecture was replaced by a skeleton source code, solid and almost unchangeable, which supported an external layer of easily modifiable functions. This structure permitted to manipulate basic functions, such as managing inputs, writing and uploading files, viewing characters, and managing animations.

The Gamemaker (https://www.yoyogames.com/gamemaker) videogame engine was used, an integrated development environment specialized in the display of high-speed two-dimensional graphics. At the same time, the external layer of software's own functions was created by converting part of the previous C and Java code into JavaScript scripts. JavaScript permitted to translate effectively the mathematical algorithms that allow processing of all the data received from user interactions. After that, to improve the videogame usability, JavaScript has been converted to GML that allows to extend the app to multiple platforms (the videogame runs currently on Android operative system).

Hardware

The videogame was administered through a touch-screen tablet (Nvidia Shield) with the following technical specifications: dimensions = 221 × 126 × 9.2 mm; weight = 390 g; display type = capacitive touchscreen, 16M colors; display size = 185.6 cm² (∼66.6% screen-to-body ratio); display resolution = 1920 × 1200 pixels, 16:10 ratio (∼283 ppi density); memory storage = 16 GB; RAM = 2 GB; processor = Nvidia Tegra K1; operative system = Android.

Stimuli and procedure

The scenario consisted of an apartment building with nine windows (Fig. 1). A total of 80 trials have been implemented, with each trial starting with one out of the nine windows opening; behind the window immediately appeared the target (i.e., a figure representing a prisoner—Fig. 1). The window remained open for a maximum duration of 1000 ms. Participants were asked to touch on the prisoner as quickly as possible as it appeared on the screen. As the target was touched, it disappeared behind the closing roll-up shutter. After a variable time period (200 or 800 ms), a new window opened, and a new target appeared. The position of the target (i.e., the window in which it appeared) was randomized across trials. The RTs, measured in milliseconds, and the number of correct responses were recorded.

FIG. 1.

Examples of the videogame scenario.

With the purpose of making the task more engaging for the players, a score was assigned to each response (i.e., +3 points for each correct response, −3 points for each error; −1 point for each omission) and displayed on the upper left corner of the screen (Fig. 1). Furthermore, different audio feedbacks were given in response to correct responses and errors.

Each participant was tested individually in a quiet room. The administration started with a practice session (warm-up) to familiarize with the videogame setting. After the warm-up, participants were administered the videogame task, which overall lasted ∼2 minutes.

Together with the videogame, pwMS were administered the SDMT, to rule out the possibility of the presence of IPS impairment at the time of videogame testing. The SDMT was chosen for this purpose as it is a widely recognized test of visual IPS that might serve as the gold standard for the assessment of IPS in pwMS.³² In the SDMT, patients are presented with a series of nine symbols, each of which is paired with a single digit (1 to 9) in a key at the top of a sheet. A pseudo-randomized sequence of the symbols is presented to the patient, who is instructed to respond, as quickly as possible, with the digit associated with each symbol. The final score is the total number of correct responses provided within 90 seconds. The SDMT was considered impaired with z scores smaller than −1.5 standard deviation (SD) compared with the normative data.²⁸

Statistical analyses

Linear regression analyses (stepwise method) were used to assess whether demographic characteristics (i.e., age, education, and gender) had an impact on the dependent variables (i.e., mean RTs and accuracy). Based on the results of the regression analyses, two separate analysis of covariances (ANCOVAs) were performed to compare videogame's mean RTs (calculated on correct responses only) and accuracy between the HC and pwMS groups.

To offer a measure of concurrent validity of the videogame RTs as a specific measurement of IPS, correlation analysis between videogame mean RTs and the SDMT total score was performed.

All the statistical analyses were executed with the software JASP (Version 0.9.0.1; JASP Team, 2019).

Results

Demographic and clinical characteristics of the study sample are listed in Table 1.

Table 1.

Demographic and Clinical Characteristics of People with Multiple Sclerosis and Healthy Controls

	pwMS (n = 43)	HCs (n = 20)
Gender (M/F)	11/32	13/7
Age (years)	36.4 ± 8.6	34.4 ± 12.3
Education (years)	14.1 ± 3.1	17.5 ± 2.5
EDSS	1.5 (0–3.5)	—
Disease duration (years)	7.6 ± 5.5	—

Means ± SD were provided for continuous variables. Median (range) was provided for EDSS.

EDSS, Expanded Disability Status Scale; HCs, healthy controls; pwMS, people with multiple sclerosis; SD, standard deviation.

Cognitive and clinical characteristics of pwMS

The median EDSS score was 1.5 (range: 0–3.5) and the mean disease duration was 7.6 ± 5.5 years.

According to the inclusion criteria, all the patients have been classified as having no CI.³⁰ Furthermore, none of the pwMS showed evidence of IPS impairment, as assessed with the SDMT (all z-scores > −1.5), at the time of videogame testing.

Videogame results

The linear regression analysis run on mean RTs evidenced age as the only significant predictor (adjusted R² = 0.21, P < 0.001). Similarly, the linear regression analysis run on the total number of correct responses showed only a significant effect of age (adjusted R² = 0.08, P = 0.03). Based on these results, the subsequent ANCOVAs were performed including age as covariate.

A significant difference in mean RTs was found between pwMS and HCs (F(1,60) = 5.463; P = 0.023; η²_p = 0.08), with pwMS being significantly slower than HCs (mean ± SD pwMS: 505.5 ± 73.9 ms; mean ± SD HCs: 462.3 ± 40.3; Fig. 2). By contrast, no significant difference between the two groups was observed on accuracy (F(1,60) = 1.835; P = 0.181; η²_p = 0.03): both pwMS and HCs showed high accuracy at the videogame task (pwMS: 97.2% of correct responses; HCs: 98.4% of correct responses).

FIG. 2.

Difference between the two groups in the mean RTs at the videogame task. *P < 0.05. HCs, healthy controls; pwMS, patients with multiple sclerosis; RTs, response times.

The correlation analysis between videogame mean RTs and the SDMT total score showed a significant negative correlation (r = −0.35; P = 0.03; CI [−0.60 to −0.04]): the higher the SDMT, the faster the patients' response on the videogame task.

Discussion

As far as we know, this is the first study in which it has been investigated the efficacy of a videogame task as a screening tool for evaluating IPS in a group of pwMS as compared with a group of HCs.

The results showed that our tablet-based videogame is indeed an effective tool to test IPS and it is sensitive in discriminating a group of pwMS, identified as without CI on the paper-and-pencil neuropsychological tests, from a group of HCs. Specifically, a significant difference between pwMS and HCs was observed on the videogame task in reference to mean RTs, with pwMS being on average slower than HCs. By contrast, no significant difference in the number of correct responses between the two groups was observed, even though pwMS were slightly less accurate than HCs.

These results are in accordance with previous studies in pwMS,^18,23 in which it has been highlighted that speed of response (i.e., RTs), rather than accuracy, might be considered as the ideal measurement to focus on. IPS is the key cognitive deficit in MS:¹⁴ when allowed adequate time to process the required information, pwMS succeed in reaching performance accuracy with rates comparable with those of HCs,^33,34 suggesting that even difficulties observed in other cognitive domains (e.g., working memory, executive functions, and verbal memory) are actually a predominant matter of speed. Furthermore, IPS has been identified as the first marker of cognitive difficulties due to neuronal and axonal damage.¹⁵

Promptly detecting initial IPS inefficiencies is of paramount importance considering that slowed IPS might be suggestive of an underlying progression of the disease that should require further clinical consideration. However, the initial detection of slowed IPS could be challenging by using only paper-and-pencil tests, as they offer measures mainly related to accuracy parameters. By contrast, recording RTs through computerized paradigms might reveal initial IPS inefficiencies that might not be captured by paper-and-pencil tests. Accordingly, the results of this study fit well within this framework: videogame mean RTs, but not accuracy, have been found sensitive to slowed IPS in a group of pwMS identified as without IPS impairment by the gold standard of paper-and-pencil IPS tests, that is, the SDMT.

The SDMT is a reliable brief cognitive screening measure that has been recommended for use in both screening and monitoring perspectives in pwMS.³⁵ In this study, we found a significant correlation between the videogame mean RTs and the SDMT total score, a result that provides evidence in support of the concurrent validity of the videogame task as a reliable measure of IPS in pwMS. According to this result, our videogame task might potentially reveal itself as a valid substitute of the SDMT in screening procedures.

The potential of the videogame format is outstanding. A videogame is a very usable and friendly tool that ideally allows for higher compliance, especially in youngest patients, in comparison with traditional tests. Videogame format might bring the same advantages as computerized paradigms with respect to the rigorous implementation of sensitive tasks and the high degree of accuracy in recording IPS within a millisecond precision; in addition, videogames offer an inherent playfulness not comparable with the setting of laboratory-based computer tasks. Videogames are easygoing, amusing, dynamic, and engaging, thus offering the opportunity to dramatically reduce performance anxiety and to allow for potential high acceptance rate by patients.

Despite the encouraging results, this study is not free from limitations. First, we did not control for psychological factors, such as depression and anxiety, which might have had an impact on videogame performance. Second, the study sample is characterized by the presence of relatively young pwMS. Even though this kind of testing might be particularly suitable for the youngest patients, future studies should include also older individuals to evaluate whether the results are generalizable to a broader range of pwMS. Similarly, the generalizability of the present results to different MS phenotypes beyond the RR type has to be further investigated.

An interesting aspect that might be investigated in future studies should foresee the inclusion of measures of every-day life functioning to evaluate whether computerized tools (such as our videogame) might indeed capture real-life demands to a greater extent than paper-and-pencil tests.

Lastly, it is worth noting that computerized testing might come with some limits. Fundamental questions about psychometric obstacles, inadequate normative data, reliability, and validity have been raised³⁶ and it has been argued that cautiousness is needed when adopting computerized assessment tools in neuropsychological practice.³⁷ With specific reference to our videogame, then, further research over its psychometric qualities, reliability, and usability is needed before extensively releasing it to the clinical practice.

Conclusion

CI in pwMS is extremely difficult to assess and monitor with paper-and-pencil tests considering that various forms of compensation (i.e., neural plasticity, cognitive reserve, practice and learning effects) might contribute in covering initial cognitive inefficiency. Analyzing patients' RTs might boost the potential of assessing actual IPS and monitoring individuals' variances in IPS over time, thus allowing to detect warning signs (red flags) of potential disease progression to be further investigated with a more extensive and comprehensive clinical evaluation. We expect that computerized testing will have an essential role in clinical practice in the near future, especially in screening and monitoring procedures.³⁸

Footnotes

Acknowledgments

We thank Agnese Tamanti for helping us with the data management and Dr. Davide Massidda for his useful suggestions about the statistical analyses. We also thank all the participants who took part in this study for their time and effort. We extend thanks to Biogen Idec for the support provided (grant for the best research project at XLV Italian Congress of Neurology).

Author Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was directly addressed for the completion of the study.

References

Lassmann

Multiple sclerosis pathology: Inflammation versus neurodegeneration. In: Translational Neuroimmunology in Multiple Sclerosis. R Arnon and A Miller (eds); Academic Press; 2016: pp 3–13.

Grzegorski

, Losy

. Cognitive impairment in multiple sclerosis—A review of current knowledge and recent research. Rev Neurosci, 2017; 28:845–860.

Amato

, Portaccio

, Goretti

, et al. Cognitive impairment in early stages of multiple sclerosis. Neurol Sci, 2010; 31(Suppl 2): S211–S214.

Pitteri

, Ziccardi

, Dapor

, et al. Lost in classification: Lower cognitive functioning in apparently cognitive normal newly diagnosed RRMS patients. Brain Sci, 2019; 9:321.

Calabrese

, Gajofatto

, Gobbin

, et al. Late-onset multiple sclerosis presenting with cognitive dysfunction and severe cortical/infratentorial atrophy. Mult Scler, 2015; 21:580–589.

Pitteri

, Romualdi

, Magliozzi

, et al. Cognitive impairment predicts disability progression and cortical thinning in MS: An 8-year study. Mult Scler, 2017; 23:848–854.

Campbell

, Rashid

, Cercignani

, Langdon

. Cognitive impairment among patients with multiple sclerosis: Associations with employment and quality of life. Postgrad Med J, 2017; 93:143–147.

Bobholz

, Rao

. Cognitive dysfunction in multiple sclerosis: A review of recent developments. Curr Opin Neurol, 2003; 16:283–288.

Wilken

, Kane

, Sullivan

, et al. The utility o f computerized neuropsychological assessment of cognitive dysfunction in patients with relapsing remitting multiple sclerosis. Mult Scler, 2003; 9:119–127.

10.

Bonato

, Deouell

. Hemispatial neglect: Computer-based testing allows more sensitive quantification of attentional disorders and recovery and might lead to better evaluation of rehabilitation. Front Hum Neurosci, 2013; 7:1–4.

11.

Kane

, Roebuck-Spencer

, Short

, et al. Identifying and monitoring cognitive deficits in clinical populations using Automated Neuropsychological Assessment Metrics (ANAM) tests. Arch Clin Neuropsychol, 2007; 22(SUPPL. 1):115–126.

12.

Wojcik

, Beier

, Costello

, et al. Computerized neuropsychological assessment devices in multiple sclerosis: A systematic review. Mult Scler J, 2019; 25:1848–1869.

13.

Chiaravalloti

, DeLuca

. Cognitive impairment in multiple sclerosis. Lancet Neurol, 2008; 7:1139–1151.

14.

Costa

, Genova

, Deluca

, Chiaravalloti

. Information processing speed in multiple sclerosis: Past, present, and future. Mult Scler J, 2017; 23:772–789.

15.

Van Schependom

, D'Hooghe

, Cleynhens

, et al. Reduced information processing speed as primum movens for cognitive decline in MS. Mult Scler J, 2015; 21:83–91.

16.

Dineen

, Vilisaar

, Hlinka

, et al. Disconnection as a mechanism for cognitive dysfunction in multiple sclerosis. Brain, 2009; 132:239–249.

17.

Manca

, Sharrack

, Paling

, et al. Brain connectivity and cognitive processing speed in multiple sclerosis: A systematic review. J Neurol Sci, 2018; 388:115–127.

18.

De Meijer

, Merlo

, Skibina

, et al. Monitoring cognitive change in multiple sclerosis using a computerized cognitive battery. Mult Scler J Exp Transl Clin, 2018; 4:1–9.

19.

Calabrese

, Magliozzi

, Ciccarelli

, et al. Exploring the origins of grey matter damage in multiple sclerosis. Nat Rev Neurosci, 2015; 16:147–158.

20.

Denney

, Lynch

, Parmenter

, Horne

. Cognitive impairment in relapsing and primary progressive multiple sclerosis: Mostly a matter of speed. J Int Neuropsychol Soc, 2004; 10:948–956.

21.

Miller

. Intelligence and brain myelination: A hypothesis. Pers Individ Dif, 1994; 17:803–832.

22.

Golan

, Wilken

, Doniger

, et al. Validity of a multi-domain computerized cognitive assessment battery for patients with multiple sclerosis. Mult Scler Relat Disord, 2019; 30:154–162.

23.

Merlo

, Darby

, Kalincik

, et al. The feasibility, reliability and concurrent validity of the MSReactor computerized cognitive screening tool in multiple sclerosis. Ther Adv Neurol Disord, 2019; 12:1756286419859183.

24.

Kühn

, Gallinat

, Mascherek

. Effects of computer gaming on cognition, brain structure, and function: A critical reflection on existing literature. Dialogues Clin Neurosci, 2019; 21:319–330.

25.

Latham

, Patston

LLM

, Tippett

. The virtual brain: 30 years of video-game play and cognitive abilities. Front Psychol, 2013; 4:1–10.

26.

Smith

. Symbol Digit Modalities Test: Manual. Los Angeles: Wester Psychological Services; 1982.

27.

Polman

, Reingold

, Banwell

, et al. Diagnostic criteria for multiple sclerosis: 2010 revisions to the McDonald criteria. Ann Neurol, 2011; 69:292–302.

28.

Amato

, Portaccio

, Goretti

, et al. The Rao’ s Brief Repeatable Battery and Stroop test: Normative values with age, education and gender corrections in an Italian population. Mult Scler, 2006; 12:787–793.

29.

Caffarra

, Vezzadini

, Dieci

, et al. Una versione abbreviata del test di Stroop: dati normativi nella popolazione italiana. [A short version of the Stroop test: Normative data in an Italian population sample.] Riv Neurol, 2002; 12:111–115.

30.

Amato

, Zipoli

, Goretti

, et al. Benign multiple sclerosis: Cognitive, psychological and social aspects in a clinical cohort. J Neurol, 2006; 253:1054–1059.

31.

Kurtzke

. Rating neurologic impairment in multiple sclerosis: An expanded disability status scale (EDSS). Neurology, 1983; 33:1444–1452.

32.

Benedict

RHB

, Deluca

, Phillips

, et al. Validity of the Symbol Digit Modalities Test as a cognition performance outcome measure for multiple sclerosis. Mult Scler, 2017; 23:721–733.

33.

DeLuca

, Gaudino

, Diamond

, et al. Acquisition and storage deficits in multiple sclerosis. J Clin Exp Neuropsychol, 1998; 20:376–390.

34.

Demaree

, DeLuca

, Gaudino

, Diamond

. Speed of information processing as a key deficit in multiple sclerosis: Implications for rehabilitation. J Neurol Neurosurg Psychiatry, 1999; 67:661–663.

35.

Benedict

RHB

, Amato

, Boringa

, et al. Brief International Cognitive Assessment for MS (BICAMS): International standards for validation. BMC Neurol, 2012; 12:55.

36.

Gates

, Kochan

. Computerized and on-line neuropsychological testing for late-life cognition and neurocognitive disorders: Are we there yet?. Curr Opin Psychiatry, 2015; 28:165–172.

37.

Schmand

. Why are neuropsychologists so reluctant to embrace modern assessment techniques?. Clin Neuropsychol, 2019; 33:209–219.

38.

Rao

. Role of Computerized screening in healthcare teams: Why computerized testing is not the death of neuropsychology. Arch Clin Neuropsychol, 2018; 33:375–378.