Using Self-Administered Game-Based Cognitive Assessment to Screen for Degenerative Dementia: A Pilot Study

Abstract

Background:

The earlier detection of dementia is needed as cases increase yearly in the aging populations of Taiwan and the world. In recent years, the global internet usage rate has gradually increased among older people. To expand dementia screening and provide timely medical intervention, a simple self-administrated assessment tool to assist in easily screening for dementia is needed.

Objective:

The two-part goal of this pilot study was, first, to develop a Game-Based Cognitive Assessment (GBCA) tool, and then, to evaluate its validity at early screening for patients with cognitive impairment.

Method:

The researchers recruited 67 patients with neurocognitive disorders (NCDs) and 57 healthy controls (HCs). Each participant underwent the GBCA and other clinical cognitive assessments (CDR, CASI, and MMSE), and filled out a questionnaire evaluating their experience of using the GBCA. Statistical analyses were used to measure the validity of the GBCA at screening for degenerative dementia.

Results:

The average GBCA scores of the HC and NCD groups were 87 (SD = 7.9) and 52 (SD = 21.7), respectively. The GBCA correlated well with the CASI (r² = 0.90, p < 0.001) and with the MMSE (r² = 0.92, p < 0.001), indicating concurrent validity. The GBCA cut-off of 75/76 corresponded to measurements of sensitivity, specificity, and area under curve of 85.1%, 91.5%, and 0.978, respectively. The positive predictive value was 91.9%, and the negative predictive value was 84.4%. The results of the user-experience questionnaire for the HC and NCD groups were good and acceptable, respectively.

Conclusion:

The GBCA is an effective and acceptable tool for screening for degenerative dementia.

Keywords

Computerized cognitive assessment dementia screening validity

INTRODUCTION

There is an urgent need to be able to detect dementia in its early stages because of the yearly growth in the number of cases among the aging populations of Taiwan and elsewhere in the world. This number of people with dementia almost doubles every 20 years and is expected to reach 131.5 million by 2050 [1]. Consequently, the overall social burden of dementia is bound to increase, affecting various aspects of life including household finances, interpersonal relationships, mood, and health. There is a positive relationship between the severity of the dementia and the costs of caring for dementia patients in Taiwan, with a strong association between functional decline and a higher total cost [2]. Early detection and treatment, with the goal of maintaining the patients’ levels of cognitive and physical functioning, may improve their quality of life and reduce the overall cost of dementia. Early detection may also make it possible for patients with certain reversible cognitive disorders to get early treatment and to restore their cognitive functioning. This would also allow their families to gain an overall understanding of future treatment plans and to plan their lives appropriately. Early detection comes with other potential benefits as well (e.g., the early identification and multifaceted reduction of risk factors, the active management of depression, greater engagement in cognitive activity and physical exercise, and the promotion of better nutrition) which together may help to slow some forms of cognitive decline associated with dementia and its progression [3].

However, fewer than half of elderly patients who meet the diagnostic criteria for degenerative dementia are identified, and under-diagnosis continues to be a major issue in the management of primary care for dementia [4, 5]. There is a high rate of undiagnosed dementia in Taiwan, with 73% of cases left undetected in 2016, according to data from the National Development Council of Taiwan.

The validity of the early diagnosis of dementia is highly dependent on its severity. The detection of very mild or mild stages of dementia has been associated with clinical diagnostic sensitivity measurements of only 0.09 to 0.41 [6]. The AD8 questionnaire, which has been widely adopted to screen for dementia, shows high sensitivity (0.97), but poor specificity (0.17) [7, 8]. A number of other common brief neurocognitive tests, such as the Mini-Mental State Examination (MMSE), the Clock Drawing Test (CDT), and the Montreal Cognitive Assessment (MoCA), have been used in the evaluation of cognitive impairment. Different studies have reported discordant results with regards to the discriminative ability of these tests [9 –13]. These cognitive tools, as well as the AD8, lack classification accuracy in detecting very mild dementia (Clinical Dementia Rating (CDR) 0.5) [14]. In addition, a trained technician or clinician is required to administer these objective tests, and patients may be unwilling to spend time or money to go to the hospital for related examinations.

In recent years, the global rate of internet use has been gradually increasing. In 1995, only 1% of the world population used computer networks, whereas by 2020 the proportion had reached nearly 60%. The rate of seniors using the internet in Canada increased from 32% in 2007 to 68% in 2016 [15]. The situation was similar in the UK in 2016, where it was reported that the proportion of people aged 75 and over who had used the internet in the prior 3 months nearly doubled in the five years leading up to 2016 (from 20% to 39%). Recent years have seen a rapidly growing rate of use of online resources by the elderly, which could help to increase the opportunities to screen for dementia and educate the people involved about this disease.

Computerized psychological assessment tools may be uniquely well-suited to achieve the early detection of changes in cognition in the elderly and so to assist with early diagnosis, evaluation, and treatments [16, 17]. Recent systematic reviews and meta-analyses have shown that computerized memory tests and paper-and-pencil memory tests had comparable diagnostic performances with regards to the detection of mild cognitive impairment and dementia [18]. Computerized cognitive tests have several strengths such as the standardization of the test administration, the accurate measurement of many variables, the automated keeping of records, and the economizing of time and costs [16 , 20]. Progress is being made in the development and validation of computerized cognitive tests. In addition to tests developed for the evaluation of an individual’s cognitive state, one particular study reported that at least 14 computerized cognitive screening tests have been designed to detect possible cognitive impairment in the preclinical phase and to serve as markers for referral for further, more specialized testing [20]. Most of them exhibit good reliability and validity. However, there are some limitations to these screening tests that preclude their widespread use as screening tools for dementia. The limitations include 1) the long duration (i.e., taking more than half an hour to complete) of such tests as the CNS Vital Signs (CNSVS), the Cognitive Stability Index (CSI), the Computer-Administered Neuropsychological Screen for Mild Cognitive Impairment (CANS-MCI), the MicroCog, and the Mild Cognitive Impairment (MCI) screening test; 2) the requirement that a technician administer such tests as the CSI, the CogState, the MCI Screen, the CogniScreen, and the untitled tests developed by Kluger et al. and Maki et al.; 3) the relatively limited capability to detect various domains of cognitive function of such tests as the CANS-MCI, the CANTAB, the Cognitive Function Test (CFT), the MCI Screen, the CogniScreen, and the untitled tests developed by Inoue et al. and Maki et al.; and 4) the limited age range examined by such tests as the CFT [21 –32].

In addition to these screening tests, the ‘COGselftest’, designed by Dougherty Jr et al., is a short self-administered test battery which includes measures of visuospatial/executive processing, working memory, verbal fluency, attention, orientation, and processing speed. It demonstrates a high degree of sensitivity and specificity and is capable of accurately identifying cognitive impairment in patients with variable degrees of cognitive abnormality [33]. However, this test does not provide any diagnostic information to the patient/test taker in the results section. The author noted that the sample population in this study consisted mostly of Caucasian individuals with a high level of education. Therefore, a more diversified sampling is needed.

A recent study on the use of a self-administered computerized test, the Computerized Cognitive Screen (CoCoSc), was conducted by Wong [34]. The test, administered on a touchscreen computer, is a 15-minute computerized cognitive assessment covering memory, executive functions, orientation, attention and working memory, and prospective memory. This test showed a sensitivity of 0.78 and a specificity of 0.69. However, it does not evaluate language skills or visuospatial perception. The author concludes that this assessment tool is feasible for testing individuals with high or low education levels.

Another recently designed cognitive test battery for detecting mild cognitive impairment and early-stage Alzheimer’s disease showed good validity in the cognitive evaluation of elderly people [35]. The battery includes eight tests with 13 subscores and was designed to evaluate visual attention, auditory attention, information processing speed, visual memory, motor control, and visuospatial perception. It does not assess abilities related to orientation, language, or executive function, and it cannot be used on home computers as it was designed for use on tablet computers.

Although the use of electronic products is quite common in Taiwan, no widely available, computer-based, Chinese-language assessment tool for detecting multiple domains of cognition is available yet. To partially fill this gap with regards to the earlier detection of dementia, we have developed a computerized screening test, the Game-Based Cognitive Assessment (GBCA) tool, which is self-administered, was designed to detect possible cognitive impairment, and can be conveniently used on a tablet computer. The aim of this study was to assess the validity of the GBCA in screening for dementia.

MATERIALS AND METHODS

In this study, purposive sampling was used. All the participants were aged between 55 years old and 90 years old. The institutional review board at the National Cheng Kung University Medical Center approved the study protocol before the study was initiated.

Participants

Degenerative neurocognitive disorder (NCD) group

The participants in the degenerative NCD group were patients recruited from the neurologic or geropsychiatric clinics in a general hospital. Diverse studies (including examinations of patient history and of the physical, neurologic, and mental states of patients, as well as laboratory experiments and brain image studies) have confirmed the diagnoses of degenerative major NCD (previously called dementia) and mild NCD for certain types of degenerative dementia (including Alzheimer’s disease, dementia with Lewy bodies, frontotemporal dementia, and Parkinson’s disease with dementia). All the participants’ diagnoses of major and mild NCD were made using the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) [36]. The participants were all given the CDR scale, MMSE, or Cognitive Abilities Screening Instrument (CASI 2.0) neuropsychological screening tests to corroborate the diagnosis of cognitive dysfunction [37, 38]. The CDR ratings of all participants with mild NCD was 0.5 and either 1 or 2 for those with major NCD. Finally, it should be noted that there is a high rate of missed and delayed diagnoses of dementia. In particular, observed changes in cognition are sometimes mistakenly interpreted as the result of normal aging rather than as signs of the onset of cognitive decline due to dementia [39, 40]. The aim of this self-administrated computerized test is to allow the earlier detection of degenerative dementia. Therefore, we designed the contents of the GBCA to be suitable for patients with a CDR score of 2 or lower.

Control group

The healthy non-demented subjects, hereafter called Healthy controls (HCs), were members of the local community of similar ages, genders, and levels of education as those of the members of the NCD group. They were all given screening questionnaires; physical, neurological, and mental state examinations; the MMSE; the CASI; and the instrumental activities of daily living (iADL) Chinese version. None of these tests showed evidence of neurocognitive deficits.

Neuropsychological tests

Each HC, as well as each mild and major NCD test subject, was given the MMSE, CASI, and CDR at the same time. These three tests are the standard tools for diagnosing dementia in neurological or geropsychiatric clinics in Taiwan. The MMSE is a short and effective test that is extensively used in both clinical and research settings to measure cognitive function. It is used to evaluate abilities related to registration (repeating named prompts), attention and calculation, recall, language, following simple commands, and orientation. It is easy to administer and takes only 5 to 10 minutes to complete. One recent review of a number of community studies shows that the pooled accuracy at a cut point of 24 was sensitivity 0.85 and specificity 0.90 [41].

The CASI is another assessment tool with high sensitivity and specificity for evaluating dementia that is also commonly used in clinical and research settings. Research on the effects of patients’ education levels has shown that, in clinical cases, the cut-off scores of the CASI for differentiating between dementia and normality were 49/50 for people with no formal education; 67/68 for people who had gone to school for 1 to 5 years; and 79/80 for people with more than 6 years of formal schooling [38]. It typically takes 15 to 20 minutes to administer this test. It has a score range of 0 to 100 and provides a quantitative assessment of nine cognitive domains related to attention, concentration, orientation, short-term memory, long-term memory, language abilities, visual construction, list-generating fluency, abstraction, and judgment. The CASI has been shown to have both a sensitivity and a specificity of over 83 % [42] and sufficient test-retest reliability [43].

GBCA

Motivating and engaging the participants was an important aim of our design of the GBCA. Tong et al. (2016) compared traditional paper-and-pencil cognitive assessment tools and serious games, with the latter featuring, in general, a repeatability and higher entertainment value that increased both the motivation of subjects to complete the test and the ease of administering it for non-clinicians [44]. Furthermore, McCallum (2012) distinguished between an “activity for health” and a “game for health” as follows: If a game is not engaging and enjoyable, it becomes merely an activity [45]. Therefore, we designed the GBCA to allow earlier screening for NCD. The flow chart of the GBCA is shown in Fig. 1. This new computerized test was designed using the Unity game engine (version 5.6.5F) as an interactive 2D game mobile application for use on both Android and Windows platforms. It includes a feature for registering the background information of the test-taker and 13 test games covering 5 cognitive domains (visuospatial perception, attention, language, memory, and executive functioning), with a total score of 100. The test games are defined in detail in Table 1. There are two key features of the game items: examples of traditional Taiwanese architecture and the Taiwanese night market culture were included in the test-game stories; and the user interface was designed with large pictures and buttons and with spoken instructions for illiterate study participants to be able to operate the app easily. Sample interfaces are shown in Figs. 2 3. This interactive game was designed as a “Happy Village”, with each participant invited to see and experience the life of the farmers residing there, who play the role of implicit guides. Some story statements are used to link the tests (as in Fig. 2). Some of the pictures appearing in the user interface are shown in Fig. 3.

Fig. 1

Flow chart of Game-Based Cognitive Assessment.

Table 1

Game-Based Cognitive Assessment (GBCA)

Item	Cognitive domain	No of questions	Score (total = 100)
Orientation (visuospatial perception)	Orientation	5	10
Registration	Memory	5	5
Repeat and Reverse the number	Attention	2	4
Serial sevens	Attention (Calculation)	5	10
Recall	Memory	5	10
Recognize the number (Visuospatial perception)	Visual recognition	3	6
Naming	Naming (Language)	6	10
Touch the screen when it displays 1	Selected attention (Attention)	6	6
Repeat after me	Language	3	5
Ten fruits	Verbal fluency (Language)	10	10
Overlapped Pentagons Drawing test	Visuospatial perception	1	4
Clock Drawing test	Executive functioning	1	10
Tower of London test	Execution functioning	3	10

Fig. 2

Story statements that used to link between tests. The English translation of the Chinese words in the pictures are: (a) ‘Hello, I am Ahui. Nice to meet you’. (b) ‘I live in Happy Village. Let’s visit the village together.’ (c) ‘Here we are at a historical site.’ (d) ‘Let’s go to the night market.’ (for the Tower of London test).

Fig. 3

The examples of pictures in the user interface (a) time orientation, (b) calculation, (c) attention, (d) recall, (e) naming, and (f) clock drawing test.

The actions that can be used to operate the GBCA include tapping buttons, long-tapping buttons, speaking into a microphone, and drawing by dragging one’s fingers over the touchscreen. In this app, the speech recognition function uses Google’s Speech API, which currently supports the Chinese language. The scores for each answer were calculated manually during the data collection. The GBCA and all the neuropsychological tests were administered in a quiet room. Assistance to open the GBCA file, to access each item, and to explain the instructions was given if participants needed it. This help did not influence the scoring of the items on the GBCA.

Questionnaire on user experience

A brief questionnaire was administered to determine whether the participants considered the GBCA an acceptable method of cognitive assessment. It includes 1 question about whether it was easy to use a tablet computer, 2 questions about whether the users felt the game stories were interesting and familiar, 7 questions about the users’ reactions to the interface design (typesetting, color, instruction, etc.), and 4 questions aimed at comparing the GBCA with other cognitive assessment tools (clear enough to understand, easy to use, etc.). Responses to each question are recorded on a 5-point scale (5 = very satisfied, 4 = satisfied, 3 = neutral, 2 = unsatisfied, and 1 = very unsatisfied).

Statistical analysis

The performance of the GBCA and each of its items was evaluated in comparison to that of other assessment tools widely used in the clinical setting (the MMSE and the CASI). The chi-square and independent t-test were used to examine the differences between two groups of participants with regards to demographic data, scores on the CASI, MMSE, and GBCA, and responses to the questionnaire. Pearson’s correlations were used to calculate the linear correlations between the MMSE, CASI, and GBCA, and their respective items. A receiver operating characteristic (ROC) curve was created to assess the overall performance of the CASI, MMSE and GBCA at discriminating between NCD subjects and HCs. The maximum value of the Youden Index (sensitivity+specificity –1) was used to select the cut-off value. All statistical analyses were performed with the Statistical Package for Social Science 20.0 (SPSS Inc, Chicago, IL, USA) for Windows.

RESULTS

Subject characteristics and GBCA, CASI, and MMSE performance for each group

There were 67 NCD and 57 HC subjects, who were chosen by means of purposive sampling, included in the data analysis. Similarities in characteristics between the subject groups were found in terms of age distributions, with a mean of 74.78 years old (SD = 7.04) for NCD subjects and 72.31 (SD = 7.35) for HCs; gender proportions, with female subjects comprising 62.7% (39/67) of the NCD group and 69.5% (41/59) of the HC group; and education levels, with 41.8% (28/67) of NCD subjects and 54.2% (32/59) of HCs having fewer than 9 years of formal schooling (see Table 2). Among the NCD subjects, 43 had been diagnosed with Alzheimer’s disease, 19 with dementia with Lewy bodies, 2 with frontotemporal dementia, and 3 with Parkinson’s disease dementia; 14 of them had mild NCD, CDR = 0.5, whereas the rest had major NCD, CDR = 1 (n = 29) and CDR = 2 (n = 24). The average GBCA scores of the HC and NCD groups were 86.95 and 51.97, respectively. The average MMSE scores of the HC and NCD groups were 27.73 and 18.73, respectively. The average CASI scores of the HC and NCD groups were 89.42 and 61.88, respectively. The total score on the MMSE, as well as the total score on the CASI and the scores for the CASI sub-items showed a significant difference between the subject groups. Similarly, the total GBCA score and the scores for most of the items on the GBCA showed a significant difference between the subject groups, with the exception of the “Registration” item.

Table 2

Demographic data and the performance of Mini-Mental State Examination (MMSE), Cognitive Abilities Screening Instrument (CASI), and Game-Based Cognitive Assessment (GBCA) in each group

	NCD (N = 67)		HC (N = 59)		Statistics
	Mean	SD	Mean	SD	t/χ²	p
Age	74.78	7.04	72.31	7.35	–1.93	0.06
Sex (M/F)	25/42		18/41		0.646	0.42
Education level (< 9 y/≥9 y)	28/39		32/27		1.948	0.16
MMSE	18.73	5.96	27.73	1.94	11.09	< 0.001
CASI Total	61.88	20.46	89.42	5.47	10.02	< 0.001
LTM	8.25	2.71	9.86	0.63	4.46	< 0.001
STM	4.47	3.26	10.52	1.35	13.29	< 0.001
ATTEN	6.81	1.54	7.80	0.55	4.68	< 0.001
MENMA	5.99	3.23	8.42	1.65	5.23	< 0.001
ORIEN	8.84	4.52	17.80	8.13	7.76	< 0.001
ABSTR	6.64	2.81	9.68	1.42	7.51	< 0.001
LANG	8.27	2.04	9.63	1.06	4.58	< 0.001
DRAW	7.73	2.96	9.73	0.93	4.97	< 0.001
ANML	4.81	2.76	6.97	2.08	4.91	< 0.001
GABC_Total (100)	51.97	21.71	86.95	7.88	11.71	< 0.001
Orientation (10)	4.18	3.51	9.66	0.92	11.64	< 0.001
Registration (5)	4.85	0.86	5.00	0.00	1.34	0.184
Repeat the number and	1.31	1.65	2.98	1.36	6.14	< 0.001
Reverse the number (4)
Serial sevens (10)	4.00	3.68	8.19	2.42	7.42	< 0.001
Recall (10)	0.69	1.62	5.32	2.99	11.01	< 0.001
Recognize the number (6)	5.31	1.73	6.00	0.00	3.06	0.003
Naming (10)	8.49	2.32	9.61	0.95	3.46	0.001
Touch the screen when	3.04	2.20	5.31	1.45	6.70	< 0.001
it displays 1 (6)
Repeat after me (5)	3.51	1.65	4.42	0.93	3.78	< 0.001
Ten fruits (10)	6.24	2.90	9.34	1.21	7.64	< 0.001
Overlapped Pentagons	2.46	1.65	3.92	0.53	6.45	< 0.001
Drawing test (4)
Clock Drawing test (10)	4.24	4.16	9.10	1.63	8.43	< 0.001
Tower of London test (10)	3.64	4.06	8.10	2.20	7.52	< 0.001

NCD, neurocognitive disorder; HC, healthy control; LTM, long-term memory; STM, short-term memory; ATTEN, attention; MENMA, mental manipulation (concentration and calculation); ORIEN, orientation; ABSTR, abstract thinking and judgment; LANG, language ability; DRAW, drawing (visual construction); ANML, animal-name fluency (categorical verbal fluency).

Correlations among the GBCA, CASI, and MMSE, and intra-GBCA

The correlation analysis showed highly significant positive relationships between the GBCA (including the total score and its subtests), the MMSE, and the CASI (including its subtests). The exceptions to these results were the correlations between the “Registration” item of the GBCA and the “Short-term memory”, “Orientation”, “Language”, and “Animal-name fluency” items of the CASI (see Supplementary Table 1). The psychometric characteristics of the tests in the GBCA battery were similar to the characteristics of the more conventional CASI and MMSE tests. The GBCA was consistent with the MMSE and CASI in detecting degenerative dementia, which suggests that the GBCA and its items produce a high-quality assessment of cognitive function (see Figs. 4 5). Most of the correlations between individual items of the GBCA were highly significant and positive, except for “Registration” (see Supplementary Table 2). Overall, the GBCA shows high internal consistency, except for the “Registration” item. This item was not significantly correlated with the following 5 items: “Repeat the number and reverse the number”, “Recall”, “Recognize the number”, “Naming”, and the “Overlapped pentagons drawing test”.

Fig. 4

The correlations between the Game-Based Cognitive Assessment (GBCA) and Cognitive Abilities Screening Instrument (CASI) with r = 0.898 and p < 0.001. HC, healthy control. Clinical Dementia Rating (CDR) = 0.5 indicates mild neurocognitive disorder (NCD), CDR≥1 indicates major NCD.

Fig. 5

The correlations between the Game-Based Cognitive Assessment GBCA and Mini-Mental State Examination (MMSE) with r = 0.915 and p < 0.001. HC, healthy control. Clinical Dementia Rating (CDR) = 0.5 indicates mild neurocognitive disorder (NCD), CDR > = 1 indicates major NCD.

The sensitivity, specificity, and cut-off point of the GBCA for the NCD group

An ROC curve and the area under the curve (AUC) were calculated to evaluate the performance of the GBCA in distinguishing between NCD subjects and HCs (see Fig. 6). With cut-off points for the NCD group of 75/76, the proposed GBCA produced a sensitivity of 85.1%, and a specificity of 91.5% (PPV: 91.9%; NPV: 84.4%). The AUC of the GBCA was 0.978 (95% CI was 0.954~1.000). The education level is needed so that the cut-off points can be used to evaluate cognitive deficits. After years of officials promoting the importance of education in Taiwan, education levels have steadily increased over time. In this pilot study, educational attainment was classified as either ≤9 years or > 9 years of formal schooling. There are no cut-offs in the CASI or the MMSE for people with more than 9 years of formal schooling. We also employed an ROC curve to distinguish the sensitivity and specificity of the MMSE and CASI tests for the NCD group from those for the HC group. For the MMSE, the cut-off points of NCD from HC were 23/24, with a sensitivity of 80.6% and a specificity of 96.6% (PPV: 96.5%; NPV: 81.4%). The AUC of the MMSE was 0.911 (95% CI was 0.856∼0.967). For the CASI, the cut-off points of NCD from HC were both 80/81, with a sensitivity of 82.1% and a specificity of 96.6% (PPV: 96.5%; NPV: 82.6%). The AUC of the CASI was 0. 898 (95% CI was 0.836∼0.960). Similar to the CASI and MMSE, the GBCA is a good assessment tool for distinguishing the NCD subjects from the HCs.

Fig. 6

The receiver operating characteristic (ROC) curves of the Game-Based Cognitive Assessment (GBCA), Cognitive Abilities Screening Instrument (CASI) and Mini-Mental State Examination (MMSE).

Fig. 7

The distribution of responses on the 14 items of the questionnaire on users’ experience of the GBCA. Vertical Axis – Satisfaction Scale: 5 = very satisfied, 4 = satisfied, 3 = neutral, 2 = unsatisfied, and 1 = very unsatisfied; Horizontal Axis – Number of Persons. Questionnaire Items: 1) Is it easy for you to use a tablet? 2) Was it easy for you to operate the Game-Based Cognitive Assessment app? 3) Are the written instructions in this app clear? 4) Is the arrangement of the text and graphs in this app satisfactory? 5) Is the design of the graphs in this app satisfactory? 6) Do you think the game story in this app is interesting? 7) Do you think the game story in this app closely reflects the Taiwanese culture? 8) Are the oral instructions in this app easy to follow? 9) Overall, how satisfied are you with this app? 10) Based on your experience using this app, would you recommend it to your friends? 11) How interesting is this app compared to other cognitive assessment instruments (such as the MMSE and the CASI)? 12) Other cognitive assessment instruments (such as those mentioned in the previous question) give instructions with voice commands. Are the graphs provided in this app easier to understand? 13) Compared to other cognitive assessment instruments (such as those mentioned in a previous question) which are administered by an examiner, is this self-administered app easier to complete? 14) Compared to other cognitive assessment instruments (such as those mentioned in a previous question) which are administered by an examiner, is this self-administered app more convenient to use?

Results of the questionnaire on user experience

The responses of the HC group on all items of the questionnaire ranged from 4 to 5, which means they were quite satisfied with their experience of using the app, and they thought it was easy to operate, well designed, and useful as a self-assessment tool (Fig. 4 and Supplementary Table 3). The GBCA is a tool that meets their needs. On the other hand, the average response of the NCD group was between 3.3 and 3.7, which puts their level of satisfaction with the app somewhere between “neutral” and “satisfied”. Comparisons between all the questions on the questionnaire showed a significant difference between the HC and NCD groups. Apparently, the HC group was more satisfied with their experience of using the GBCA than was the NCD group.

The data that we personally collected on the associations between questionnaire scores, on the one hand, and CDR severity level, education level and age, on the other, showed correlations that deserve to be discussed further (see Supplementary Table 4). For example, lower scores given by the participants to our questionnaire corresponded to a higher CDR score, a higher age, and a lower education level. More specifically, the severity of cognitive dysfunction adversely affected the participants’ ability to use the tablet to complete the GBCA, as suggested by the poor ratings given to the text and design of the GBCA. Regarding age, it had a negative influence on the participants’ satisfaction with the pictures, text, and story used in the GBCA. Finally, education level influenced, somewhat negatively, the participants’ satisfaction with the pictures and text. For the HC group, age and education level were not significantly correlated with scores on the GBCA items, whereas the opposite was true for the NCD group. When its use with the participants in the HC and mild NCD groups was compared to that with patients with major NCD, the GBCA proved to be more convenient and easier to administer than both the CASI and the MMSE.

DISCUSSION

Reliability and validity

The results of this study show that the GBCA is a promising tool for detecting cognitive impairment. It exhibited a high sensitivity (85.1%) and a high specificity (91.5%) in differentiating between the NCD and HC subject groups. As for currently used assessment tools, the MMSE is a common instrument with a high sensitivity and specificity that was designed to identify the cognitive impairments symptomatic of dementia [41]. In addition, the CASI provides a reliable profile of the abilities included in 9 cognitive domains and provides an overall measure of the underlying cognitive abilities of dementia [46]. There are good correlations between the total scores on the GBCA and scores on the other two more standard neuropsychological tests—the overall score on the MMSE and the domain-specific scores on the CASI. The GBCA was designed to assess general functioning in 5 cognitive domains (visuospatial perception, attention, language, memory, and executive functioning) present in the diagnostic criteria for neurocognitive disorders in the DSM-5. The subtests of the GBCA were highly correlated with sub-items of the CASI, suggesting a consistency between these two tests and their sub-items. Moreover, most items of the GBCA show internal consistency. These data indicate that the GBCA is a valid and sensitive instrument for evaluating functional deficits in both global and specific cognitive domains.

The principal limitation of the GBCA is that the “Registration” item was not significantly correlated with certain items of the CASI (“Short-term memory”, “Orientation”, “Language”, and “Animal-name fluency”, with p-values of 0.077, 0.145, 0.054, and 0.0078, respectively). The “Registration” item was also not significantly correlated with 5 other items of the GBCA (“Repeat the number and reverse the number”, “Recall”, “Recognize the number”, “Naming”, and the “Overlapped pentagons drawing test”). The “Registration” item was designed to evaluate the short-term recollection of 5 different characteristic words through acoustic, visual, and semantic encoding. The process of registration involves sensory memory, and the processing (encoding) and storage of information. The processes of chunking and encoding (visually, acoustically, and semantically) encompass effortful processing through attention, practice, and thought. Hence, poor registration often reflects poor attention or executive dysfunction and may invalidate test results for recall or recognition which evaluate episodic memory [47, 48]. In theory, the “Registration” item should be correlated with all the other items of the GBCA. The automatic computerized scoring system of the “Registration” task lacks limits on the duration that subjects can spend rehearsing the tasks. This shortcoming possibly induced participants to overestimate their registration ability, which in turn caused the inconsistent correlation results in this study. This issue will be addressed in future versions of the GBCA assessment tool.

Clinical practice and applicability

A good cognitive screening tool is practical to use. This feature often requires that the test be short (so that it can be used in a busy clinical setting or as an outcome measure in a trial in order to prevent participants from being overburdened by long interviews) and acceptable to the people involved in its use (so that it does not upset, exhaust, or embarrass the patient or the assessor) [49]. The GBCA takes about 15 to 20 minutes to complete, which the participants felt was acceptable. The results of the questionnaire on user experience showed that levels of satisfaction with using the GBCA were “neutral” to “satisfied” for the NCD group and “satisfied” for the HC group. There are statistically significant differences between both. The CDR score, age, and education level of the participants influenced the degree of satisfaction that they expressed after using the GBCA. More specifically, a higher severity of cognitive dysfunction, a higher age, and a lower education level were associated with lower levels of satisfaction with the use of the GBCA tablet, the operation of the GBCA app, and the text and design of the GBCA. Age and education were significantly associated with the scoring of the GBCA items for the NCD group, but not in the case of the HC group. For the NCD group, being illiterate, seldom using electronic devices, and being unfamiliar with such devices limited their motivation to use the GBCA. These factors, in addition to the stress that accompanies unfamiliar situations and the presence of visual and hearing impairments made participants with NCDs resistant to using mobile apps. Possible ways of reducing the differences in how favorably HC and NCD participants react to using the app in the future include 1) designing more interesting and familiar stories for the GBCA to attract the attention of participants, 2) improving the attractiveness and quality of the pictures, and 3) promoting the public’s familiarity with using computer networks.

A previous study stated that one potential shortcoming of using computerized assessment tools with older patients is the anxiety individuals may experience when confronted with unfamiliar equipment [50]. However, another study showed that state anxiety in older adults is not necessarily deleterious to cognitive performance, has no appreciable negative effect on many cognitive domains, and can even be beneficial [51]. Though the questionnaire in our study did not evaluate anxiety felt while completing the GBCA, the high correlations and consistency found between the GBCA and the other paper-and-pencil tests suggest a limited influence of emotion on GBCA scores compared to scores on the pencil-and-paper tests. Nevertheless, results show that most users like the app proposed in this study and found it interesting. During the administration of the assessment tool, family members of some patients even asked to have it for home use. Overall, the GBCA shows potential.

The benefits and limitations of the GBCA

The use of computer-assisted screening and diagnostic tools in clinical settings is growing in popularity. Particularly noteworthy in this context is the increasing convenience of tablet computers and smartphones, which facilitates the continuous assessment of cognitive function and allows for the unobtrusive collection of data on the auxiliary behavioral markers of cognitive impairment, thus making it possible for test administrators to observe trends and detect acute changes that have traditionally been difficult to identify [52]. The GBCA, as well as other previously published computerized tests for cognitive screening, offer several advantages, including 1) saving time and money, 2) producing notably accurate assessments within milliseconds, 3) providing scores automatically, 4) allowing for greater standardization, 5) precluding the need for a trained administrator (although most patients with moderate dementia require guidance through the test-taking process), 6) using the adaptive presentation of items, and 7) the reduction of examiner bias effects.

The partial limitations of the GBCA and other computerized self-administered tests include 1) the absence of the active participation of a consulting neuro-psychologist who could analyze the qualitative performance of the test-taker; 2) the lack of familiarity with computers and the resulting anxiety of a number of older adults, which can influence both their performance on the test and their willingness to undergo such testing; 3) the presence and possible influence on test-takers of uncontrolled conditions such as noise, distractions, or interruptions; 4) the possible effects of low psychomotor speed and upper-extremity impairments on test results; 5) and the possibility of practice effects [20 , 54]. The GBCA, as a screening tool, provides information on possible cognitive dysfunction but not a diagnosis. Abnormal results of the GBCA are an indication that the participants should receive further assessment and a differential diagnosis in a clinic. To prevent practice effects, alternative versions of the test should be designed for repeated testing. We calculated the ROC curve to distinguish the mild NCD group from the HC group in terms of the sensitivity and specificity of the GBCA, the MMSE, and the CASI. For the GBCA with the cut-off points of 75/76, the sensitivity and specificity were 91% and 43%, respectively. For the MMSE with the cut-off points of 23/24, the sensitivity and specificity were 97% and 21%, respectively. For the CASI with cut-off points of 80/81, the sensitivity and specificity were 95% and 26%, respectively. The GBCA showed a similar validity to that of the MMSE and the CASI in distinguishing the mild NCD group from the HC group. However, only 14 patients with mild NCD were involved in this study, which limits the assessment of the validity of the GBCA in discriminating the mild NCD group from the HC group. In future studies, the inclusion of a larger number of mild NCD cases is necessary in order to be able to assess the capability of the GBCA to detect NCDs early. A larger sampling of different education levels is needed to allow for more precise detection of cognition impairment. Language was another complicating factor in our study. The language used in this version of the GBCA is traditional Mandarin Chinese. It might be useful to design computer speech recognition software that functions with different Chinese dialects. There is still a need to expand the automatic language discrimination and recognition functions of the GBCA to include different languages.

Conclusion

Our results support the validity of the GBCA as a self-administrated screening tool for degenerative dementia.

Footnotes

ACKNOWLEDGMENTS

The authors want to thank Chien-Ting Lin, Feng-Chuan Wang, Chia-Hung Hu, and I-Chun Hung for their excellent technical assistance in this study. We also want to thank Andy Cormier for his English editorial assistance. The study was supported by research grants from the Research Project of Tainan Hospital and Ministry of Health Executive.

Authors’ disclosures available online ().

The supplementary material is available in the electronic version of this article: .

References

Prince

, Bryce

, Albanese

, Wimo

, Ribeiro

, Ferri

(2013) The global prevalence of dementia: A systematic review and metaanalysis. Alzheimers Dement 9, 63–75.

LJE

, Pai

, Shih

(2016) Economic impact of dementia by disease severity: Exploring the relationship between stage of dementia and cost of care in Taiwan. PLoS One 11, e0148779.

Naismith

, Glozier

, Burke

, Carter

, Scott

, Hickie

(2009) Early intervention for cognitive decline: Is there a role for multiple medical or behavioural interventions? Early Interv Psychiatry 3, 19–27.

Ganguli

, Rodriguez

, Mulsant

, Richards

, Pandav

, Bilt

, Dodge

, Stoehr

, Saxton

, Morycz

, Rubin

, Farkas

, DeKosky

(2004) Detection and management of cognitive impairment in primary care: The Steel Valley seniors survey. J Am Geriatr Soc 52, 1668–1675.

Connolly

, Gaehl

, Martin

, Morris

, Purandare

(2011) Underdiagnosis of dementia in primary care: Variations in the observed prevalence and comparisons to the expected prevalence. Aging Ment Health 15, 978–984.

Andrea

, Mark

, Paul

, Susan

, Hardeep

(2009) Missed and delayed diagnosis of dementia in primary care: Prevalence and contributing factors. Alzheimer Dis Assoc Disord 23, 306–314.

Chan

, Xu

, Shaik

, Chong

, Hui

, Chen

, Dong

(2016) Clinical utility of the informant AD8 as a dementia case finding instrument in primary healthcare. J Alzheimers Dis 49, 121–127.

Larner

(2015) AD8 Informant Questionnaire for Cognitive Impairment: Pragmatic diagnostic test accuracy study. J Geriatr Psychiatry Neurol 28, 198–202.

Nasreddine

, Phillips

, Bedirian

, Charbonneau

, Whitehead

, Collin

, Cummings

, Chertkow

(2005) The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. J Am Geriatr Soc 53, 695–699.

10.

Smith

, Gildeh

, Holmes

(2007) The Montreal Cognitive Assessment: Validity and utility in a memory clinic setting. Can J Psychiatry 52, 329–332.

11.

Wen

, Zhang

, Niu

, Li

(2008) The application of Montreal cognitive assessment in urban Chinese residents of Beijing. Chinese J Intern Med 47, 36–39.

12.

Arevalo-Rodriguez

, Smailagic

, Roque

IFM

, Ciapponi

, Sanchez-Perez

, Giannakou

, Pedraza

, Cosp

, Cullum

(2015) Mini-Mental State Examination (MMSE) for the detection of Alzheimer’s disease and other dementias in people with mild cognitive impairment (MCI). Cochrane Database Syst Rev 2015, CD010783.

13.

Park

, Jeong

, Seomun

(2018) The clock drawing test: A systematic review and meta-analysis of diagnostic accuracy. Nurs 74, 2742–2754.

14.

Kan

, Zhang

, Cheng

, Wong

, Venketasubramanian

, Chen

, Xu

(2019) The Informant AD8 can discriminate patients with dementia from healthy control participants in an Asian older cohort. J Am Med Dir Assoc 20, 775–779.

15.

Davidson

, Schimmele

(2019) Evolving Internet use among Canadian seniors, Statistics Canada, Catalogue no. 11F0019M, no. 427.

16.

Wild

, Howieson

, Webbe

, Seelye

, Kaye

(2008) The status of computerized cognitive testing in aging: A systematic review. Alzheimers Dement 4, 428–437.

17.

Buckley

, Sparks

, Papp

, Dekhtyar

, Martin

, Sperling

, Rentz

(2017) Computerized cognitive testing for use in clinical trials: A comparison of the NIH toolbox and Cogstate C3 batteries. J Prev Alzheimers Dis 4, 3–11.

18.

Chan

JYC

, Kwong

JSW

, Wong

, Kwok

TCY

, Tsoi

KKF

(2018) Comparison of computerized and paper-and-pencil memory tests in detection of mild cognitive impairment and dementia: A systematic review and meta-analysis of diagnostic studies. J Am Med Dir Assoc 19, 748–756.

19.

Wesnes

(2014) Moving beyond the pros and cons of automating cognitive testing in pathological aging and dementia: The case for equal opportunity. Alzheimers Res Ther 6, 58.

20.

Zygouris

, Tsolaki

(2015) Computerized cognitive testing for older adults: A review. Am J Alzheimers Dis Other Demen 30, 13–28.

21.

Gualtieri

, Johnson

(2006) Reliability and validity of a computerized neurocognitive test battery, CNS vital signs. Arch Clin Neuropsychol 21, 623–643.

22.

Erlanger

, Kaushik

, Broshek

, Freeman

, Feldman

, Festa

(2002) Development and validation of a web-based screening tool for monitoring cognitive status. J Head Trauma Rehabil 17, 458–476.

23.

Tornatore

, Hill

, Laboff

, McGann

(2005) Self-administered screening for mild cognitive impairment: Initial validation of a computerized test battery. J Neuropsychiatry Clin Neurosci 17, 98–105.

24.

Green

, Green

, Harrison

, Kutner

(1994) Screening for cognitive impairment in older individuals. validation study of a computer-based test. Arch Neurol 51, 779–786.

25.

Trenkle

, Shankle

, Azen

(2007) Detecting cognitive impairment in primary care: Performance assessment of three screening instruments. J Alzheimers Dis 11, 323–335.

26.

Hammers

, Spurgeon

, Ryan

, Persad

, Heidebrink

, Barbas

, Albin

, Frey

, Darby

, Giordani

(2011) Reliability of repeated cognitive assessment of dementia using a brief computerized battery. Am J Alzheimers Dis Other Dement 26, 326–333.

27.

Kingsbury

, Pachana

, Humphreys

, Tehan

, Byrne

GJA

(2010) Utility of a computerised cognitive screen in MCI and depression in an older population. Aust J Rehabil Counsel 16, 14–26.

28.

Kluger

, Saunders

, Hou

, Garvan

, Kirli

, Efros

, Chau

, Crucian

, Finney

, Meador

, Heilman

(2009) A brief computerized selfscreen for dementia. J Clin Exp Neuropsychol 31, 234–244.

29.

Maki

, Yoshida

, Yamaguchi

(2010) Computerized visuo-spatial memory test as a supplementary screening test for dementia. Psychogeriatrics 10, 77–82.

30.

Robbins

, James

, Owen

, Sahakian

, McInnes

, Rabbitt

(1994) Cambridge Neuropsychological Test Automated Battery (CANTAB): A factor analytic study of a large sample of normal elderly volunteers. Dementia 5, 266–281.

31.

Trustram Eve

, de Jager

(2014) Piloting and validation of a novel self-administered online cognitive screening tool in normal older persons: The cognitive function test. Int J Geriatr Psychiatry 29, 198–206.

32.

Inoue

, Jinbo

, Nakamura

, Taniguchi

, Urakami

(2009) Development and evaluation of a computerized testbattery for Alzheimer’s disease screening in community-based settings. Am J Alzheimers Dis Other Dement 24, 129–135.

33.

Dougherty

Jr. , Cannon

, Nicholas

, Hall

, Hare

, Carr

, Dougherty

, Janowitz

, Arunthamakun

(2010) The computerized self test (CST): An interactive, internet accessible cognitive screening test for dementia. J Alzheimers Dis 20, 185–195.

34.

Wong

, Fong

, Mok

, Leung

, Tong

(2017) Cognitive Screen (CoCoSc): A self-administered computerized test for screening for cognitive impairment in community social centers. J Alzheimers Dis 59, 1299–1306.

35.

Huang

, Singh

, Chen

, Sun

, Huang

, Liu

(2019) Validity of a novel touch screen tablet based assessment for mild cognitive impairment and probable AD in older adults. Assessment 26, 1540–1553.

36.

American Psychiatric Association (2013) Neurocognitive disorder. In Diagnostic and Statistical Manual of Mental Disorder: Fifth Edition (DSM-5), American Psychiatric Publishing, Washington, DC, pp. 591–644.

37.

Guo

, Liu

, Wong

, Liao

, Yan

, Lin

, Chang

, Hsu

(1988) Chinese version and norms of the Mini-Mental State Examination. Taiwan J Phys Med Rehabil 16, 52–59.

38.

Lin

, Wang

, Liu

, Chen

, Lee

, Liu

(2002) Cutoff scores of the cognitive abilities screening instrument, Chinese version in screening of dementia. Dement Geriatr Cogn Disord 14, 176–182.

39.

Mahoney

, Cloutterbuck

, Neary

, Zhan

(2005) African American, Chinese, and Latino family caregivers’ impressions of the onset and diagnosis of dementia: Cross-cultural similarities and differences. Gerontologist 45, 783–792.

40.

Helvik

, Engedal

, Šaltytė Benth

, Selbæk

(2018) Time from symptom debut to dementia assessmentby the specialist healthcare service in Norway. Dement Geriatr Cogn Dis Extra 8, 117–127.

41.

Creavin

, Wisniewski

, Noel-Storr

, Trevelyan

, Hampton

, Rayment

, Thom

, Nash

KJE

, Elhamoui

, Milligan

, Patel

, Tsivos

, Wing

, Phillips

, Kellman

, Shackleton

, Singleton

, Neale

, Watton

, Cullum

(2016) Mini-Mental State Examination (MMSE) for the detection of dementia in clinically unevaluated people aged 65 and over in community and primary care populations. Cochrane Database Syst Rev, CD011145.

42.

Teng

, Hasegawa

, Homma

, Imai

, Larson

, Graves

, Sugimoto

, Yamaguchi

, Sasaki

, Chiu

, et al. (1994) The Cognitive Abilities Screening Instrument (CASI): A practical test for cross-cultural epidemiological studies of dementia. Int Psychogeriatr 6, 45–58.

43.

Chiu

, Yip

, Woo

, Lin

(2019) Test-retest reliability and minimal detectable change of the Cognitive Abilities Screening Instrument in patients with dementia. PLos One 14, e0216450.

44.

Tong

, Chignell

, Tierney

, Lee

(2016) A serious game for clinical assessment of cognitive status: Validation study. JMIR Serious Games 4, e7.

45.

McCallum

(2012) Gamification and serious games for personalized health. Stud Health Technol Inform 177, 85–96.

46.

Tsai

, Lin

, Wang

, Liu

(2007) Evaluating the uses of the total score and the domain scores in the Cognitive Abilities Screening Instrument, Chinese Version (CASI C-2.0): Results of confirmatory factor analysis. Int Psychogeriatr 19, 1051–1063.

47.

Kipps

, Hodges

(2005) Cognitive assessment for clinicians. J Neurol Neurosurg Psychiatry Suppl 1 (Suppl I), i22–30.

48.

Salwa

, Rodrigo

, Robert

, Amrish

, Richard

JSW

(2018) Speech registration in symptomatic memory impairment. Front Aging Neurosci 10, 201.

49.

Bart

(2012) Assessment scales in dementia. Ther Adv Neurol Disord 5, 349–358.

50.

Cassady

(2004) The impact of cognitive test anxiety on text comprehension and recall in the absence of external evaluative pressure. Appl Cogn Psychol 18, 311–325.

51.

Potvin

, Bergua

, Meillon

, Le Goff

, Bouisson

, Dartigues

, Amieva

(2013) State anxiety and cognitive functioning in older adults. Am J Geriatr Psychiatry 21, 915–924.

52.

Koo

, Vizer

(2019) Mobile technology for cognitive assessment of older adults: A scoping review. Innov Aging 3, igy038.

53.

de Oliveira

, Brucki

SMD

(2014) Computerized Neurocognitive Test (CNT) in mild cognitive impairment and Alzheimer’s disease. Dement Neuropsychol 8, 112–116.

54.

Eve

, de Jager

(2014) Piloting and validation of a novel self-administered online cognitive screening tool in normal older persons: The Cognitive Function Test. Int J Geriatr Psychiatry 29, 198–206.