Abstract
Keywords
INTRODUCTION
Worldwide over 46 million people are estimated to have dementia and this number is expected to rise by almost threefold in 2050 [1]. Early detection and diagnosis of dementia is therefore of paramount importance. It is a prerequisite for arranging adequate care, for taking safety precautions, and for anticipating on future care [2]. In this regard, the detection of mild cognitive impairment (MCI) is especially important in view of the recently found high conversion rates from MCI to dementia [3]. However, although many senior and older people worry about their memory [4], many of them are also reluctant to discuss the possibility of cognitive impairment and dementia with their general practitioner [5].
An easily accessible, anonymous, and validated automated self-test of cognitive function may provide a means for people to test their memory function in the privacy of their own homes. The potential value of such a self-test is likely to be twofold. On the one hand, people with suspicious cognitive decline will be advised to consult their general practitioner who in turn may refer them to a memory clinic. On the other hand, people who are unnecessarily worried about their memory will be reassured.
However, most existing cognitive tests such as the Mini-Mental Status Examination (MMSE) [6] or the Mini-Cog [7] are unsuitable because they require administration by a clinician. Furthermore, they often lack precision in the screening for MCI [8, 9]. Although several self-administered cognitive tests were developed in the past [10–12], it remains unknown which mode of administration is more suitable, e.g., a telephone-based or computer-based self-test. A self-administered cognitive test by telephone [13, 14] would obviously preclude an assessment of the visual modality of cognitive function, or the administration of (cued) recall memory tasks, which are sensitive to early Alzheimer’s disease. Yet, operating a telephone might be easier than operating a computer. Also, research about the diagnostic accuracy of self-administered cognitive tests in the screening for dementia tends to be limited [11, 12]. Moreover, most tests do not automatically provide users with feedback on their results, thereby making them unsuitable for the home setting [10–12].
Accordingly, the aims of this study were to develop automated online and telephone-based self-tests of cognitive function, and to examine their feasibility and validity. We predicted the self-tests to have adequate diagnostic accuracy in the screening for MCI and dementia, and to have strong convergent validity with widely used neuropsychological tests.
MATERIALS AND METHODS
Participants
We conducted a prospective diagnostic accuracy study according to the STARD criteria [15]. Participants were patients from the Amsterdam Dementia Cohort who were enrolled between February and October 2015 at the Alzheimer Center of the VU University Medical Center, an outpatient memory clinic. Patients were referred by either their general practitioner or by a medical specialist for an examination of their cognitive complaints. All participants underwent a diagnostic work-up for dementia. For the present study, patients were eligible if they were ≥50 years, had an MMSE score ≥17, and have either a diagnosis of subjective cognitive decline (SCD) (i.e., memory complaints that could not be objectively verified), MCI, or dementia. Participants were excluded in case of neurological or psychiatric co-morbidity (i.e., psychosis, depression, or alcohol abuse), insufficient command of Dutch, impaired vision and hearing, functional problems of the dominant hand or focal damage, and if they were unable to operate a computer or a telephone.
The self-tests (index tests) were independently administered from the diagnostic work-up (reference standard). Participants received instructions on how to access the self-tests at home in two separate email messages one week apart. To decrease order effects, a random half of the participants completed the telephone self-test before the online self-test and the other random half completed them vice versa. After a week, participants who had not completed the self-tests were sent reminders through e-mail or were contacted by telephone. The medical ethical committee of the VU University Medical Center approved the study [protocol number: 2014-265]. All participants were capable of providing consent by and for themselves and did so accordingly.
Dementia assessment
All participants underwent a dementia assessment that has been described elsewhere [16]. It included a clinical history, a medical and neurological examination, screening laboratory tests, a neuropsychological assessment (see below), an MRI of the brain, EEG, and the 15-item Geriatric Depression Scale (GDS) [17]. Diagnoses of probable AD and MCI were made according to the core clinical criteria of the National Institute on Aging-Alzheimer’s Association (NIA-AA) workgroups [18, 19]. Clinicians were kept blind to the results from the self-tests.
The neuropsychological assessment consisted of the MMSE, and tests to assess functioning in the following three cognitive domains (tests between parentheses): Attention (digit span test forward from the Wechsler Adult Intelligence Scale–III [20], Trail making test part A [21], and Color Reading and Naming of Color Patches from the Stroop test [22], Executive function (Digit span test backward from the Wechsler Adult Intelligence Scale–III [20], the trail making test part B [21] and Color-Word interference of the Stroop test [22], Memory (Visual association test [23], the Dutch version of the Auditory Verbal Learning Test including the recognition condition [24, 25], and Global Cognitive Function or a composite measure of all of these tests.
Telephone-based and online self-tests
Both the online and the telephone cognitive self-tests were developed based on the Telephonic Remote Evaluation of Neuropsychological Deficits (TREND) [14]. The content of the online self-test was expanded by including visual and visuospatial tasks. In addition to the cognitive tests described below, both self-tests included a five item short form of the GDS [26]. For both self-tests, we aimed for a duration of about 20 minutes. In the development process of both self-tests, preliminary versions of the self-tests were critically reviewed by neuropsychologists and research fellows from the Alzheimer Center of the VU University Medical Center for face validity. To examine the user-friendliness of the self-tests, four patients without and three with dementia were provided access to the self-tests and both were shown to be user-friendly.
Telephone self-test
The telephone self-test contained eight tasks that required touch-tone input for all responses.
Orientation: Participants were asked which year, season, month, day of the month, and day of the week it was (score range 0–5). Participants were explicitly instructed not to use aids, e.g., a clock or a calendar.
Digit-sequence learning: On 3 successive trials, participants heard a sequence of 5 digits (e.g., 5-7-6-8-1) and were asked to enter the sequence in the presented order. Each trial was scored as correct or incorrect (score range 0–3).
Immediate word recognition: Participants heard a list of 10 words and were asked to remember them. Subsequently, participants heard again a list of 10 words of which 5 were targets from the previously presented list and 5 new words that acted as distractors. Participants were asked to distinguish the targets from the distractors (score range 0–10).
Directed key pressing: Participants were asked to press particular keys a specific number of times, e.g., “Press the ‘9’ key three times”. The task consisted of 5 trials (score range 0–5). This task was meant as a distractor between immediate and delayed word recognition.
Delayed word recognition: Testing and scoring was the same as for immediate word recognition. Participants heard the remaining 5 target words and 5 new words (distractors) and were again asked to distinguish targets from distractors.
Auditory spatial relations: Participants heard descriptions of key locations relative to a 3×3 keypad matrix (1 to 3 top, 4 to 6 middle, and 7 to 9 bottom row) and were asked to press the corresponding keys. For example, “TOP RIGHT” corresponded to the “3” key. There were three trials (score range 0–3).
Semantic comprehension: Participants heard 6 declarative statements and were asked to indicate whether each statement did or did not make sense (e.g., “We wanted to cut down a tree in our front garden, so we went to the garage to get our hammers”). Three sentences made sense and three did not (score range, 0–6).
Backward digit span: Participants heard different 4-digit sequences on 3 successive trials. They were asked to enter the digits in the reverse order. Each trial was scored as correct or incorrect (score range 0–3).
Online self-test
The online self-test consisted of ten tasks. Some of the tasks of the telephone self-test were replaced by visual tasks. Participants used the computer mouse and keyboard to provide their responses.
Orientation: Identical to telephone self-test but visually presented.
Digit-sequence learning: Participants were shown three consecutive trials of five digit numbers. Each number appeared separately on the screen. Participants were then required to enter these digits from a displayed set of numbers (1 to 9) by clicking on the numbers in the correct order.
Immediate word recall: Participants were shown 10 words and were asked to remember them. Subsequently, they were asked to type as many words from the list that they remembered into a text box.
Connecting numbered dots: Participants were asked to connect a series of numbers in the correct increasing order by clicking on them as fast as possible (score range 0–300 indicating the amount of time it took to complete the task).
Letter number alternation: Participants were asked to connect numbers and letters in the correct increasing order while alternating between numbers and letters by clicking on them as fast as possible, i.e., 1-A-2-B, etc. (score range 0–300 indicating the amount of time it took to complete the task).
Delayed word recall: Participants were again asked to type as many words as they remembered from the word list presented earlier into a text box.
Delayed word recognition: Participants were presented with 10 targets from the previous word list as well as 10 distractors and were asked to discriminate between targets and distractors (score range 0–20).
Word pairs immediate recall: Participants were presented with 10 word pairs of which 4 were semantically related, and were asked to remember them. Subsequently, they were presented with one word of each of the pairs and were asked to type in the matching word (2 successive trials, score range 0–10 per trial).
Word pairs recognition: Participants were shown one word of the word pair and were presented with four possible matches from which they had to choose. One was the correct target, the other three were distractors of which one was strongly semantically related, one less strongly semantically related, and one was semantically unrelated but had the same lexical frequency (based on film subtitles [27]) (score range 0–10).
Semantic comprehension: Identical to telephone self-test but visually presented.
Data analysis
Patients with SCD, MCI, or dementia were compared on relevant demographic and clinical characteristics using one-way ANOVA, Kruskal Wallis, and χ2-tests. For the telephone and online self-test separately, z-scores were calculated for the individual tasks because they had different scoring ranges. Subsequently, these z-scores were averaged to obtain a mean z-score for the telephone self-test and a mean z-score for the online self-test. Z-scores were also calculated for the neuropsychological domains of attention, executive function, memory, and global cognition.
Convergent validity of the self-tests was examined by calculating Spearman’s rank correlations between the self-tests and the MMSE, as well as the neuropsychological domains of attention, executive function and memory, and global cognition. Spearman’s rank correlations were also calculated between the short version of the GDS included in both self-tests and the 15-item GDS from the dementia assessment.
We examined the diagnostic accuracy of the telephone and online self-test for the following two reference standards: (a) MCI + dementia versus SCD and (b) MCI alone versus SCD (thus excluding the participants with a diagnosis of dementia). The Area under the curve (AUC), as well as the sensitivity (i.e., the proportion of people with MCI and dementia or MCI alone with a score below the cut-off value on the self-tests) and the specificity (i.e., the proportion of people with SCD with a score above the cut-off value on the self-tests) were calculated. Cut-off scores were derived by inspecting where the ROC curves had optimal sensitivity and specificity.
Two additional analyses were conducted. First, using backwards logistic regression, we identified which tasks of the online and telephone self-test were most sensitive in the screening for the two reference standards. AUCs were then recalculated. Second, we examined the influence of depression on the diagnostic accuracy of both self-tests by including the short form of the GDS into the total average z-scores in a separate ROC analysis. Data were analyzed using SPSS version 22.
RESULTS
A total of 117 patients were included of whom 100 (86%) had results for the online cognitive self-test and 93 (80%) had results for the telephone cognitive self-test (see Flowchart in Fig. 1). A total of 76 patients completed both self-tests. Few patients dropped out of the study because of technical difficulties (n = 5, 4%) or because of experiencing the self-tests as too confrontational (n = 7, 6%). Furthermore, three patients felt they needed some time to become accustomed to the self-tests, e.g., the use of the keyboard in case of the online test. One patient explicitly mentioned that the instructions were not clear, whereas another one stated that the speed of the test was too fast. Four patients expressed a clear preference for the online cognitive self-test over the telephone self-test, e.g., because of problems with the touch-tones. The mean completion time for the telephone self-test was 16 minutes (range 13–23). The mean completion time for the computer self-test was 24 minutes (range 13–48). Altogether, both the online and telephone cognitive self-tests seemed to be feasible.
Of the participants, 34 (29.1%) had SCD, 30 (25.6%) had MCI, and 53 (45.3%) had dementia (32 had probable Alzheimer’s disease, 1 had vascular dementia, 5 had Lewy body dementia, 7 had frontotemporal dementia, 3 had primary progressive aphasia, and 5 had dementia not otherwise specified). As expected, compared to those with SCD, MCI patients had lower MMSE scores, performed worse on the online and telephone self-tests, had worse global cognitive function, and also performed worse in the cognitive domains of executive function and memory. In turn, patients with dementia performed worse on all of these measures than MCI patients (see Table 1). The median time interval between the administration of both self-tests was 2.1 weeks (±1.8 weeks). The median time interval between the neuropsychological assessment and the first self-test was 13.8 weeks (±9.3 weeks).
Convergent validity
Strong associations were found between thez-score of the online cognitive self-test and the MMSE as well as with the global cognition measure of the neuropsychological assessment (see Fig. 2). Correlations between the z-score of the telephone cognitive self-test and the MMSE and global cognition were moderate (see Fig. 3). The online self-test was moderately correlated with the domain scores of ‘attention’ and ‘executive function” and strongly correlated with the domain score of ‘memory’. The correlations between the telephone self-test and the domains of ‘attention’ ‘executive function’ and ‘memory’ were moderate. Moderate correlations were observed between the short form of the GDS from the self-tests and the GDS from the dementia assessment (rs = 0.56, p < 0.001 and rs = 0.40, p < 0.01).
Diagnostic accuracy
Table 2 shows the AUCs for the online and telephone cognitive self-test as well as the MMSE along with sensitivity and specificity values. It also shows the true positive and true negative values, that indicate the actual number of patients that were correctly classified (according to the reference standard) by the cognitive self-tests.
The online self-test (full administration) had adequate diagnostic accuracy in the screening for MCI and dementia versus SCD. Based on backwards logistic regression, the following tasks of the online self-test were sensitive to the screening of MCI and dementia versus SCD: ‘orientation’, ‘letter number alteration’, and ‘word pairs immediate recall trial 2’. After only including these sensitive tasks, the AUC of the online self-test improved somewhat. The AUC of the MMSE was 0.82 (95% CI: 0.74–0.89). By contrast, the telephone self-test had lower diagnostic accuracy. Sensitivity analyses including only the most sensitive tasks of the telephone self-test did not improve the AUCs to at least the level of the AUC of the MMSE. Diagnostic accuracy was not influenced when the short form of the GDS was included in the total scores of the self-tests (results available on request).
DISCUSSION
The online automated cognitive self-test had adequate diagnostic accuracy in distinguishing MCI and dementia from SCD. The online self-test including only the most sensitive subtests had a sensitivity of 90% and a specificity of 79%. This indicates that 9 out of 10 people who have MCI or dementia will be accurately classified by the online self-test as having MCI or dementia and that 8 out of 10 people with SCD will be accurately classified as not having MCI or dementia. The self-tests are not diagnostic instruments however. They are meant to provide people advice on whether they should see their general practitioner. In case someone is incorrectly classified, they will still be encouraged to visit their general practitioner if their worries persist. Furthermore, we found evidence for good convergent validity of the online self-test with the MMSE and the neuropsychological measures of global cognition, attention, memory, and executive function. By contrast, the telephone self-test had lower diagnostic accuracy in detecting MCI and dementia versus subjective cognitive decline. Furthermore, compared to those from the online self-test, the correlations of the telephone self-test with the MMSE total score and the neuropsychological measures of global cognition and memory were lower. Lastly, the presence of depressive symptoms did not seem to influence the diagnostic accuracy of both self-tests. Taken together, these findings favor the online self-test of cognitive function over the telephone-based one. The potential clinical utility of the online self-test for people who worry about their cognitive function is twofold. On the one hand, it is likely to reassure people who are unnecessarily worried about benign forgetfulness that is normal for one’s age. On the other hand, in case of suspicious cognitive complaints, it has the potential to expedite the help-seeking process and an adequate referral of people to a memory clinic. It has therefore the potential to increase the early detection of MCI. The recently found high conversion rates from MCI to dementia [3], especially among the amnestic subtype of probable MCI, underline the importance of instruments such as the online cognitive self-test. The online cognitive self-test may also encourage earlier interventions through lifestyle guidance and treatment of lifestyle-related diseases [28, 29] as well as pharmacological treatment of MCI with underlying AD pathology [30]. This in turn may prevent or delay further cognitive decline and conversion to dementia. More research in future is needed to answer these questions and the potential of cognitive self-tests in these respects.
The online self-test is likely to be useful in clinical practice for other reasons as well. It is more feasible than the MMSE, because unlike the MMSE, it does not require the routine presence of a clinician to conduct or score the test. In addition, general practitioners may also recommend the online cognitive self-test for patients who are worried about their memory. Given that the majority of our study participants were able to complete the online cognitive self-test, it seems also feasible in this regard.
At the same time, a caveat to the use of the online self-test as a screening instrument of cognitive function that deserves attention, is the risk of unnecessary worry and upheaval caused by test results as was argued before [31]. Although adequate test feedback and advice was formulated and built into our cognitive self-tests, more research is needed about how to provide adequate test advice as a crucial part of completing the self-tests. This would truly make them ready for practical use in everyday life.
This study had several strengths. The sample was sufficiently large and likely to resemble the patients who are typically seen in clinical practice. Spectrum bias was unlikely as both MCI and dementia patients were included and the differences in performance on the cognitive self-tests between the groups were probably small enough to prevent inflation of the diagnostic accuracy. A strength of the reference standard was that it was based on a thorough diagnostic workup in accordance with diagnostic consensus criteria. Furthermore, its extensive neuropsychological examination allowed us to examine associations between the self-tests and more formal neuropsychological tests.
This study had also limitations. Our sample was not large enough to compare the accuracy of the self-tests with that of well-known cognitive screeners such as the MMSE. Future research with larger samples should be aimed at such comparisons. There may be an incorporation bias for the MMSE, as this screener was also part of the diagnostic work-up. However, examination of cognitive function is an essential part of the diagnostic work-up. Excluding a measure such as the MMSE would result in an incomplete dementia assessment. Because our sample consisted of patients from a memory clinic, ideally these larger samples should be recruited in primary care settings. A final limitation is that we included relatively few people with other forms of dementia than Alzheimer’s disease. In everyday life, all people with memory problems will potentially use our self-tests, including people with possible Lewy body or vascular dementia. This underlines the need to replicate the present study in patients who have other types of dementia as well.
Longitudinal studies aimed at examining the responsiveness of the online self-test to cognitive decline over time are also warranted. For that goal, future developments should also be aimed at constructing parallel forms of the online cognitive self-test and investigating their test-retest reliability. The development of alternate forms would also enable people to re-take the self-test if they feel they were not able to perform optimally due to fatigue or other factors.
Yet another step would be to construct a shorter version of the online self-test and to examine its diagnostic accuracy in an independent sample of people to confirm the present findings. A possibility in this regard would be to develop a computerized adaptive testing version of the online self-test [32]. This would enable tailored testing by selecting only items of appropriate difficulty depending on a patient’s global cognitive ability and would thereby shorten the administration time in a more flexible manner.
In conclusion, the online cognitive self-test is a promising instrument to distinguish MCI and dementia from subjective cognitive decline that can be easily self-administered by people who worry about their memory. Future developments may involve further modifications of the online cognitive self-test to shorten its administration time and to build in test-advice which can be provided as feedback to users, as well as investigations of the merits of cognitive self-tests in terms of improving the early screening of cognitive impairment followed by timely and appropriate interventions.
Footnotes
ACKNOWLEDGMENTS
We are grateful to all participants for spending their time. We would kindly thank the research fellows from the Alzheimer Center of the VU University Medical Center for critically reviewing preliminary versions of the self-tests. This study was funded by the Stichting Stoffels-Hornstra.
