Validity of the MemTrax Memory Test Compared to the Montreal Cognitive Assessment in the Detection of Mild Cognitive Impairment and Dementia due to Alzheimer’s Disease in a Chinese Cohort

Abstract

Background:

A valid, reliable, accessible, engaging, and affordable digital cognitive screen instrument for clinical use is in urgent demand.

Objective:

To assess the clinical utility of the MemTrax memory test for early detection of cognitive impairment in a Chinese cohort.

Methods:

The 2.5-minute MemTrax and the Montreal Cognitive Assessment (MoCA) were performed by 50 clinically diagnosed cognitively normal (CON), 50 mild cognitive impairment due to AD (MCI-AD), and 50 Alzheimer’s disease (AD) volunteer participants. The percentage of correct responses (MTx-% C), the mean response time (MTx-RT), and the composite scores (MTx-Cp) of MemTrax and the MoCA scores were comparatively analyzed and receiver operating characteristic (ROC) curves generated.

Results:

Multivariate linear regression analyses indicated MTx-% C, MTx-Cp, and the MoCA score were significantly lower in MCI-AD versus CON and in AD versus MCI-AD groups (all with p≤0.001). For the differentiation of MCI-AD from CON, an optimized MTx-% C cutoff of 81% had 72% sensitivity and 84% specificity with an area under the curve (AUC) of 0.839, whereas the MoCA score of 23 had 54% sensitivity and 86% specificity with an AUC of 0.740. For the differentiation of AD from MCI-AD, MTx-Cp of 43.0 had 70% sensitivity and 82% specificity with an AUC of 0.799, whereas the MoCA score of 20 had 84% sensitivity and 62% specificity with an AUC of 0.767.

Conclusion:

MemTrax can effectively detect both clinically diagnosed MCI and AD with better accuracy as compared to the MoCA based on AUCs in a Chinese cohort.

Keywords

Alzheimer’s disease cognitive assessment instrument continuous recognition task paradigm mild cognitive impairment

INTRODUCTION

Alzheimer’s disease (AD) is a major disease that has no current curative therapy or treatment that can reverse, arrest, or even slow down disease progression [1]. Nonetheless, early detection of cognitive decline has been recognized as a critical first step in the efforts to stop dementia progression and find a cure for AD and other types of dementia [1]. Early detection of mild cognitive impairment (MCI) as a clinical priority is supported by the recent consensus of an international working group which consisted of MCI and AD experts (the Global Advisory Group on Future MCI Care Pathways) [2 –4]. This group “ultimately supports the idea that detection of MCI is an important component of whole person care” [2], “acknowledges that cognitive screening by default is not recommended”, and proposes large-scale “evaluation of individuals with a concern or interest in their cognitive performance” [3]. Early detection is also advocated by recent calls for proposals for low-cost detection of cognitive decline, especially via digital biomarkers, sets of objective, quantifiable physiological or behavioral data that are generated through using digital devices, from both the public [5] and private [6] sectors in the United States. The goal is to develop and validate reliable, affordable, and accessible tools that can identify and monitor subtle, yet pertinent, changes caused by early AD pathologies. Given the current limitation of available resources, the international working group “agreed that the most feasible strategy to optimize early detection of MCI in the near-term will be to boost primary care capacity for detection by providing infrastructure and equipment that improve the accuracy and efficiency of tools for cognitive assessment without substantially increasing workload for primary care clinicians [2].” The MemTrax memory test was cited as an example of currently available digital cognitive assessment tools for such a purpose [4].

There have been significant developments in recent decades in cognitive assessment sensitive enough to detect changes associated with prodromal or preclinical AD [7 –11]. For example, with the Loewenstein-Acevedo Scale of Semantic Interference and Learning (LASSI-L) [10], a cognitive stress test which has been generalized to be used in both English and Spanish speaking people [11], performance deficits on various LASSI-L indices differentiated persons with normal cognition from those with amnestic MCI (aMCI) and most importantly associated with cortical thinning in AD–prone regions [11].

Changes in episodic memory tests alone for predicting the development of AD has been demonstrated previously, where 1.5 standard deviations (SD) below the mean in both Logic memory story A (LM-II) and Rey auditory verbal learning test (AVLT) delay recall scores predicted MCI to AD conversion [12]. This prediction was made two years before the clinical diagnosis of AD with 79% predicted AD with 76% accuracy. Most available memory and cognition assessment tools, such as the LM-II, AVLT, and the Montreal Cognitive Assessment (MoCA) have alternate forms to avoid learning effects, but oftentimes the alternate forms may not be available or available only in a limited number. Accordingly, this restricted capacity compromises monitoring changes over time with a high level of precision and confidence due to the learning effect with frequently repeated administrations. Online instruments with effectively unlimited test variations (not identical, but comparable) which are not or minimally influenced by language or culture and at the same time are fun and engaging would offer the best opportunity to practically address this limitation and meet the need for effective large-scale screening and early detection of MCI.

To this end, the MemTrax memory test, based on the continuous recognition task (CRT) with a variable N-back paradigm, which is widely used in academic memory research [13, 14], was developed and adapted for easy and simple online administration [15]. Whereas the CRT with variable N-back paradigm requires attention and working memory, MemTrax consists of stimuli exceeding the capacity of working memory and operates as an episodic memory test through the repeated encoding and recognition of newly learned or experienced pictorial scenes (images, pictures). To facilitate the encoding of new information into long-term, episodic memory, there are built-in iterations (5 items) to enhance consolidation. Accordingly, MemTrax is an episodic memory test suitable for detecting cognitive impairment. With the online adaptation, MemTrax uses recognition to assess episodic memory and also attention and processing speed, as well as motor function through response time measurement. Accordingly, MemTrax is a test suitable for assessing episodic memory and other elements of cognitive function.

Pictures were selected as stimuli in lieu of verbal and other forms of stimuli commonly used in cognitive screen instruments such as the MoCA [16]. These were chosen to accommodate a computer setting and to utilize stimuli suitably more complex than simple verbal or visual cues which, moreover, could only be implemented with the advent and development of complex picture display capability on computer monitors. Additional important advantages in using visual stimuli include the capacity to: 1) minimize the influence of language and culture, 2) ensure a format that could be understood and followed by people representing a wide range of cognitive abilities from normal to subtle impairment to moderately impaired, and 3) thus provide the potential for a harmonized and universal cognitive screen instrument for global early detection of cognitive impairment. The MemTrax test has since been successfully implemented and utilized online in countries around the world and on various platforms in several languages, including English, French, Dutch, and Chinese. Applications have included the social media platform WeChat and web portal in China (SJN Biomed, LTD: mini program version of MemTrax and http://www.memtrax.com.cn) ([17] and Zhou et al., manuscript in preparation) and web portals in France (HAPPYneuron, Inc.) [15, 18], the United States (MemTrax, LLC: http://www.memtrax.com, http://www.memoryhealthregistry.com, the Brain Health Registry: http://www.brainhealthregistry.org) [19, 20], and a site provided by the Alzheimer’s Founda-tion of America: https://afamemorytest.com/. Moreover, a prior study carried out in the Netherlands, Wageningen University, showed a favorable comparison of MemTrax to the MoCA in community dwelling elderly [21].

Here we report on the clinical utility of MemTrax in a Chinese cohort as a digital cognitive assessment instrument to detect cognitive impairment associated with MCI and AD in a clinical setting. We focused on the cross-validation of MemTrax with one of the most widely used cognitive screen instruments for detecting cognitive impairment—the MoCA. Notably, there is no current generally recognized gold-standard brief cognitive screen instrument to be used in the clinic [7]. If comparatively validated, MemTrax could be appropriately promoted and effectively utilized as a preferred initial cognitive health screen option in clinical practice.

MATERIALS AND METHODS

Study population

Participants were enrolled continuously between August 2018 and April 2020. All participants were recruited through in-person discussion with patients who volunteered to participate in the study at the out-patient neurology and memory clinics of the first Affiliated Hospital of Kunming Medical University. Investigators who implemented the tests were blinded to whether participants were cognitively impaired patients or cognitively healthy controls as were the participants themselves prior to the tests. After detailed history-taking and neurological examination, comprehensive blood tests and neuroimages including magnetic resonance imaging (MRI) and/or fluorodeoxyglucose positron emission tomography (FDG-PET) were carried out for all participants. Possible vascular, traumatic, other neurodegenerative, or medical causes of cognitive decline were ruled out, and the clinical diagnoses of MCI due to AD and AD were based on the guideline in the Diagnostic and Statistical Manual of Mental Disorders: DSM-5. This study was performed according to the Helsinki declaration of 1975 and was approved by the ethical committee of the first Affiliated Hospital of Kunming Medical University in Kunming, Yunnan, China. All participants voluntarily signed an informed consent form.

Data collection

A general questionnaire was administered to collect demographic information including age, sex, years of education, occupation, living situation (alone or with family), and medical history. Following completion of the questionnaire, the MoCA and MemTrax tests were administered in sequential order with MoCA administered first (without randomization for logistical reasons, including that the MemTrax test administration is relatively free of administrator bias). The two tests were administered within 20 minutes of each other. The evaluators were blinded to the diagnosis of all the subjects. MemTrax scores were recorded on paper for each participant tested, and the completed questionnaire and the results of MoCA were uploaded into Excel spreadsheets by the researcher who administered the tests. Entries were verified by a colleague before the Excel files were saved for analyses.

MemTrax test procedure

Detailed description of the theory and design of MemTrax has been published previously [15]. Briefly, with each MemTrax test, a series of 50 images were shown—25 new images and 25 repeated images (one image in each of the five categories, was repeated twice). Each image was shown for three seconds or until a behavioral response was recorded. The users were instructed to respond and touch the screen as quickly as possible only when repeated images were shown. At the end of the test, the program calculated and showed the percentage of correct responses (%, MTx-% C) and the mean response time (in seconds, MTx-RT) of all responses. MTx-RT was computed using the device’s internal clock which was corrected for a long press (hold-on to the screen) automatically using the default touch screen setting. The MemTrax composite score (MTx-Cp) was derived by multiplying the numbers in MTx-% C and the reciprocal of MTx-RT.

The MemTrax test was explained in detail to each participant and a practice test was provided. Participants who could not understand the instructions or were not able to perform the practice run were excluded from this study. To avoid repetition of images during the actual test being reported here, only images not included in the registered user database were used for the practice test. The MemTrax test typically takes less than 2 minutes to complete, and it finishes on its own in 2.5 minutes if no responses are made. The participants’ vision was also generally assessed by their ability to read the 4 Chinese characters on the first item of the MoCA which are equivalent to English letters A, B, C, and D. Participants who could not read these 4 characters were excluded from this study.

Montreal cognitive assessment and scoring

The Beijing version [22] of the MoCA (MoCA-BJ), which is one of the five Chinese translated versions of the English MoCA (http://www.mocatest.org) [16], was used for our study. The tests were administered and scored by trained researchers according to the official test instructions. The MoCA score was adjusted to account for education influences where a numerical value of 1 was added to the score of participants whose education was 12 years or less, unless the MoCA score was 30 already, in which case the score was not increased (score range for the MoCA is 0–30). Administration of a single MoCA test took about 10 to 30 minutes.

Statistical analyses

Data were collected and entered in Microsoft^® Excel 16.16 and verified by an independent re-searcher. Student’s t-test, paired t-test, Wilcoxon Mann-Whitney test, or χ² test were performed as appropriate. Receiver operating characteristic (ROC) curves were generated to assess the diagnostic accuracy of parameters. Pairwise comparison of ROC curves tested the equality of two or more ROC areas obtained from applying two or more test modalities to the same subject. The optimized cut-off was computed using Yoden’s index. One-way analysis of variance (one-way ANOVA) was conducted to compare the baseline variables when such variables were continuous with normal distribution. As this was not a randomized controlled trial, those baseline variables with statistically significant differences were adjusted by multivariable linear regression models. We used an alpha level of 0.05 for all statistical tests. All statistical analyses were performed using STATA/SE 13 (serial number: 401306302851).

RESULTS

Participant characteristics

A total of 150 participants were enrolled in this study. Analyses were performed on these three groups: 1) Cognitively normal control group (CON) (n = 50); 2) Mild cognitive impairment of AD type (MCI-AD) (n = 50); 3) AD (n = 50). Demographic and clinical characteristics of all the participants including age, education, typical sleep length, sex, physical or mental job, live alone or with someone, smoke (tobacco use), regular alcohol consumption, right- or left-handed, hypertension, diabetes mellitus, hypercholesteremia, brain trauma, and stroke are shown in Table 1. Briefly, the mean education years was 9 years in the CON group and the MCI-AD group and 6 years in the AD group, respectively (p = 0.082). Five subjects in the MCI-AD group reported living alone, whereas all subjects in CON and AD group were living with family members (p = 0.003). As for baseline health condition, the three groups were statistically differentiated by hypertension (p = 0.002), diabetes mellitus (p = 0.050), and hypercholesteremia (p = 0.002), but stroke did not reach statically significance (p = 0.081). There were no statistical differences in the other factors assessed among the three groups. All the factors with p-value below 0.1 were included in the multivariate linear regression analyses.

Table 1

Demographic features of the participants^a

	CON (n = 50)	MCI-AD (n = 50)	AD (n = 50)	p
Age (y)	68±8.19	67.7±10.79	70.5±10.57	0.316^b
Education (y)	9 (4, 12)	9 (6, 12)	6 (6, 9)	0.082^c
Sleep length (h)	7 (5,6)	7 (6, 8)	7 (5, 9)	0.448^c
Sex (Male) (%)	30 (60%)	24 (48%)	26 (52%)	0.472^d
Physical job (%)	27 (54%)	17 (34%)	23 (46%)	0.129^d
Live alone (%)	0 (0%)	5 (10%)	0 (0%)	0.003 ^d
Smoke (%)	18 (36%)	11 (22%)	19 (38%)	0.174^d
Alcohol (%)	14 (28%)	18 (36%)	19 (38%)	0.536^d
Right-handed (%)	49 (98%)	48 (96%)	49 (98%)	0.245^d
Hypertension (%)	10 (20%)	27 (54%)	19 (38%)	0.002 ^d
Diabetes Mellitus (%)	17 (34%)	7 (14%)	10 (20%)	0.050^d
Hypercholesteremia (%)	14 (28%)	5 (10%)	2 (4%)	0.002 ^d
Brain Trauma (%)	3 (6%)	8 (16%)	8 (16%)	0.222^d
Stroke (%)	7 (14%)	4 (8%)	12 (24%)	0.081^d

CON, normal control subjects; MCI-AD, subjects with mild cognitive impairment due to Alzheimer’s disease; AD, Alzheimer’s disease; ^aValues are number (percentage) or mean±standard deviation, median (interquartile range [IQR]); bold values are statistically significant (p < 0.05); ^bVariance analysis; ^cKruskal Wallis test; ^dChi-square test.

Differentiating clinically diagnosed cognitive normal, MCI-AD and AD

As shown in Table 2 and Fig. 1, MTx-% C was significantly lower in the MCI-AD group as compared to the CON group (p < 0.001) and in the AD group as compared to the MCI-AD group (p < 0.001). MTx-RT was not significantly different between CON and MCI-AD groups (p = 0.67); but it was significantly longer in the AD group as compared to the MCI-AD group (p < 0.001). The composite score MTx-Cp was significantly lower in the MCI-AD group as compared to the CON group (p < 0.01) and in the AD group as compared to the MCI-AD group (p < 0.001). The MoCA score was different between CON and MCI-AD groups (p < 0.001), as well as between MCI-AD and AD groups (p < 0.001), where each of the former groups had higher scores than the latter groups, respectively.

Table 2

MoCA and MemTrax results of CON, MCI-AD, and AD groups^a

	CON (n = 50)	MCI-AD (n = 50)	AD (n = 50)	p (CON versus MCI-AD)	p (MCI-AD versus AD)
MoCA score	25 (23, 26)	22 (18, 25)	17 (11, 20)	0.000 ^b	0.000 ^b
MTx-% C	86 (82, 92)	74 (64, 83)	62 (54, 68)	0.000 ^b	0.000 ^b
MTx-RT	1.311±0.208	1.331±0.259	1.543±0.244	0.671^c	0.000 ^c
MTx-Cp	68.34±14.44	57.64±17.76	41.66±10.59	0.001 ^c	0.001 ^c

CON, normal controls; MCI-AD, mild cognitive impairment due to Alzheimer’s disease; AD, Alzheimer’s disease; MoCA, Montreal Cognitive Assessment; MTx-% C, MemTrax percent correct; MTx-RT, MemTrax mean response time; MTx-Cp, MemTrax composite score; ^aValues are median (interquartile range [IQR]) or mean±standard deviation; bold values are statistically significant (p < 0.05); ^bMann-Whitney test; ^cStudent t-test.

Fig. 1

The MoCA (a), MTx-% C (b), MTx-RT (c), and MTx-Cp (d) scores were plotted for the clinical diagnosed cognitive normal (CON) (n = 50), MCI-AD (n = 50), and AD (n = 50). Excepted for MTx-RT did not distinguish CON from MCI-AD, the MoCA, MTx-% C, MTx-RT, and MTx-Cp scores were statistically different in all paired comparisons: CON versus MCI-AD and MCI-AD versus AD.

Results from the multivariate linear regression analyses after adjustment for education, living alone, hypertension, diabetes mellitus, hypercholesteremia, and stroke are shown in Table 3. MTx-% C was significantly lower in the MCI-AD group as compared to the CON group (p < 0.001, 95% CI [–16.915∼–8.591]) and in the AD group as compared to the MCI-AD group (p < 0.001, 95% CI [–28.328∼–20.405]). MTx-RT was not significantly different between CON and MCI-AD groups (p = 0.483, 95% CI [–0.069∼0.144]), but it was significantly longer in the AD group as compared to the MCI-AD group (p < 0.001, 95% CI [0.125∼0.328]) after the multivariate linear regression analysis. The composite score MTx-Cp was significantly lower in the MCI-AD group as compared to the CON group (p = 0.001, 95% CI [–17.489∼4.86]) and in the AD group as compared to the MCI-AD group (p < 0.001, 95% CI [–32.783∼20.76]). The MoCA score was different between CON and MCI-AD groups (p < 0.001, 95% CI [–5.298∼1.810]), as well as between MCI-AD and AD groups (p < 0.001, 95% CI [–9.694∼6.374]).

Table 3

Multivariate linear regression analyses of confounding factors for cognitive function

	Coef	p	95% CI
MoCA score
Education (y)	0.441	0.000	[0.293∼0.588]
Live alone	–2.032	0.274	[–5.693∼1.629]
Hypertension	–0.797	0.267	[–2.212∼0.617]
Diabetes Mellitus	–1.493	0.069	[–3.102∼0.116]
Hypercholesteremia	–0.331	0.733	[–2.249∼1.586]
Stroke	1.124	0.216	[–0.665∼2.912]
CON	–
MCI	–3.554	0.000	[–5.298∼1.810]
AD	–8.034	0.000	[–9.694∼–6.374]
MTx-% C
Education (y)	0.426	0.018	[0.073∼0.778]
Live alone	–4.558	0.304	[–13.295∼4.179]
Hypertension	–2.794	0.104	[–6.169∼0.581]
Diabetes Mellitus	–0.936	0.630	[–4.776∼2.903]
Hypercholesteremia	–2.354	0.311	[–6.930∼2.221]
Stroke	3.148	0.147	[–1.120∼7.416]
CON	–
MCI	–12.753	0.000	[–16.915∼–8.591]
AD	–24.366	0.000	[–28.328∼–20.405]
MTx-RT
Education (y)	–0.010	0.024	[–0.020∼–0.001]
Live alone	–0.036	0.749	[–0.260∼0.187]
Hypertension	0.004	0.927	[–0.082∼0.090]
Diabetes Mellitus	0.038	0.448	[–0.060∼0.136]
Hypercholesteremia	0.033	0.581	[–0.084∼0.150]
Stroke	0.028	0.610	[–0.081∼0.137]
CON	–
MCI	0.038	0.483	[–0.069∼0.144]
AD	0.227	0.000	[0.125∼0.328]
MTx-Cp
Education (y)	0.858	0.002	[0.323∼1.393]
Live alone	–2.805	0.676	[–16.056∼10.447]
Hypertension	–2.476	0.341	[–7.596∼2.643]
Diabetes Mellitus	–3.026	0.306	[–8.850∼2.798]
Hypercholesteremia	–5.510	0.119	[–12.450∼1.431]
Stroke	0.514	0.875	[–5.959∼6.988]
CON	–
MCI	–11.176	0.001	[–17.489∼–4.863]
AD	–26.774	0.000	[–32.783∼–20.76]

ROC curve analyses

To determine the diagnostic accuracy of the MemTrax and the MoCA, ROC curve analyses were conducted (Fig. 2). When the CON and MCI-AD groups were compared, the AUCs were 0.740 (95% CI [0.643∼0.836]) for the MoCA score, 0.839 (95% CI [0.761∼0.917]) for MTx-% C, 0.527 (95% CI [0.412∼0.641]) for MTx-RT and 0.708 (95% CI [0.606∼0.811]) for MTx-Cp. When the MCI-AD and AD groups were compared, the AUCs were 0.767 (95% CI [0.673∼0.860]) for the MoCA score, 0.760 (95% CI [0.665∼0.854]) for MTx-% C, 0.716 (95% CI [0.616∼0.816]) for MTx-RT and 0.799 (95% CI [0.712∼0.886]) for MTx-Cp to predict AD from MCI-AD. Based on the AUC results, the best metric of MemTrax to differentiate the CON and MCI-AD was MTx-% C, whereas MTx-Cp was the best metric to differentiate MCI-AD and AD groups.

Fig. 2

ROC analyses with AUCs were carried out using the MoCA, MTx-% C, MTx-RT, and MTx-Cp scores from clinically diagnose cognitive normal (CON) (n = 50), MCI-AD (n = 50), and AD (n = 50) for the prediction of MCI from CON and AD from MCI as indicated in a-h.

Pairwise comparison was performed to contrast the ROC curves of MTx-% C with the MoCA score between the CON and MCI-AD groups. The ROC curve of MTx-Cp was also compared to that of the MoCA between the MCI-AD and AD groups. The AUC of MTx-% C between the CON and MCI-AD groups was higher than that of the MoCA score though the difference was not statistically significant (χ²: 2.95, p-value: 0.086). The AUC of MTX-Cp between MCI-AD and AD groups was also higher than that of the MoCA, again without statistical significance (χ²: 0.41, p-value: 0.522).

Sensitivity and specificity analyses for different optimized cut-off values of all the MemTrax metrics and the MoCA score are shown in Table 4. When using the MTx-% C to compare the CON and MCI-AD groups, the optimized cut-off value which maximized true positives while minimizing false positives was 81%, with the sensitivity and specificity of diagnosing MCI-AD 72% and 84%, respectively. Using 23 as an optimized cut-off value for the MoCA score, the sensitivity and specificity of diagnosing MCI-AD were 54% and 86%, respectively.

Table 4

Sensitivity and specificity analyses for different cut-off values of MemTrax metrics and the MoCA score^a,b

CON versus MCI-AD			MCI-AD versus AD
Cut-off	Sensitivity (%)	Specificity (%)	Cut-off	Sensitivity (%)	Specificity (%)
MTx-% C
< 79	66	90	< 65	68	74
< 81	72	84	< 67	74	70
< 83	76	68	< 69	78	62
MTx-RT
> 1.384	42	64	> 1.413	76	60
> 1.400	42	66	> 1.423	76	62
> 1.413	36	68	> 1.430	74	62
MoCA
< 22	48	88	< 19	70	66
< 23	54	86	< 20	84	62
< 24	64	68	< 21	90	52

CON, normal control; MCI-AD, mild cognitive impairment due to Alzheimer’s disease; AD, Alzheimer’s disease; MoCA, Montreal Cognitive Assessment; MTx-% C, MemTrax percent correct; MTx-RT, MemTrax mean response time; MTx-Cp, MemTrax composite score; ^aWhen sum of sensitivity and specificity reached maximum, the cut-off value was considered to be the best one; ^bThe gray lines are the recommended optimized cut-off values.

When using the MTx-Cp to compare the MCI-AD and AD groups, the optimized cut-off value was 43.0, with the sensitivity and specificity of diagnosing AD 70% and 82%, respectively. Using 20 as a cut-off value for the MoCA score, the sensitivity and specificity of diagnosing AD from MCI-AD were 84% and 62%, respectively.

DISCUSSION

Our comparison of the MemTrax memory test to the MoCA for estimating cognitive impairment in a Chinese cohort in a hospital setting in these selected out-patient clinics extended earlier research in a community dwelling of elderly adults in the Netherlands [12] in both the cognitive impairment severity and a substantially different Chinese culture. Our findings support MemTrax’s clinical utility in detecting cognitive impairment associated with MCI-AD and AD which is comparable or better than the widely used MoCA. In fact, from a clinically practical perspective for large-scale cognitive screening in the current global healthcare environment with the limitation of physician’s time and other infrastructures [2], it is arguable that the online digital MemTrax test is a superior instrument compared to the MoCA for widespread cognitive screening. This is especially noted given that the MemTrax test only takes 1.5 to 2.5 minutes without the need of trained personnel and that the scoring is automatically carried out and immediately provided to the user. In comparison, a trained professional is needed to administer the MoCA and it takes 10–30 minutes to carry it out and score the test.

In real-world practice, patients in memory clinics with MCI and dementia are mixed. Therefore, the Chinese cohort in this study included patients with MCI and dementia which enabled us to readily explore whether the MemTrax test can quickly detect patients with MCI and distinctively identify others with dementia. Moreover, as shown in the multivariate linear regression analyses, education was an independent risk factor for the performance in cognitive tests. For every additional year of academic education, the MoCA score increased by 0.441 points (95% CI 0.293∼0.588), the MTx-% C increased by 0.426% (95% CI 0.073% ∼0.778%), the MTx-RT decreased by 0.01s (95% CI –0.020 s∼–0.001s), and the MTx-Cp increased by 0.858 points (95% CI 0.323∼1.393).To validate the utility and efficacy of a test in a new environment, it is ideal to compare the test results in the applicable context to those of a gold standard preferably using similar constructs and test format for the same (or similar) purpose. However, given there is no gold standard cognitive screening instrument, the next logical selection for comparison and cross validation purpose is to use a widely used and accepted standard. There is currently no other cognitive screen instrument using pictorial stimuli that is as widely used or as accepted as the MoCA to compare with MemTrax. Thus, the MoCA, was selected for comparison with full knowledge of its limitations, especially regarding the potential influence of language and culture on MoCA performance and accuracy [22, 23]. Though widely used in academic environments, the MoCA is time consuming (10–30 minutes) for brief cognitive screening, and the score is dependent on the training of the professional and the accuracy of the hand tabulation. Though the MoCA has several sub-scores purportedly reflecting specific cognitive domains, these values are not used in clinical situations, where only the summed score, irregularly corrected for education, is used. The contents and constructs of the two tests are differentiated by the pictorial only stimuli for MemTrax versus more diverse pictorial, language, and numbers stimuli for the MoCA. Notably, the pictorial nature of the items used as stimuli in MemTrax is simple in concept, but it contains rich content and context with complex and diverse pictures that require cognitive integrity to encode and recognize. These features make the utilization of the MoCA to cross-validate the MemTrax more meaningful, given both tests require functions of multiple cognitive domains.

It is interesting to note that our studies identified a score of 23 as an optimal cut-off in screening for MCI as recently reported [21] instead of the original reported score of 26 [16, 22], whereas the optimal MoCA cut-off of 20 for dementia was also comparable to the results of recent studies [24, 25]. Moreover, in our recent clinical study of aMCI, the best cut-off values are also 81% for MTx-% C and 23 for MoCA for the prediction of aMCI [26]. In our current study, MemTrax had AUCs of 0.839 (MTx-% C) and 0.799 (MTx-Cp) for predicting MCI and AD, respectively, which are better than MoCA’s AUCs of 0.740 and 0.760, respectively, in the same Chinese cohort. The three statistical methods, Student t-test, multivariate linear regression analysis, as well as the ROC analysis, confirmed that under the conditions of this study, MTx-RT could not identify patients who were suffering from mild cognitive impairment from normal individuals, whereas MTx-RT could partially distinguish patients with dementia from patients with MCI.

Assessment of an individual’s precise degree of impairment along the continuum of cognitive health would require a more comprehensive array of indicators beyond the MemTrax results. To this point, Bergeron et al. [17] demonstrated a more inclusive machine learning modeling approach using MemTrax in predicting MCI and differentiating dementia severity. The utility of MemTrax per se lies in cognitive screening at home or in a clinical setting as the initial screen, analogous to how other brief cognitive screen instruments are used.

Dementia and especially AD progression is increasingly accepted as a continuum with gradual decline of cognitive ability [7]. Despite the clear cutoff scores shown here for MCI detection and dementia separation, a cutoff score may not be the optimal approach to informatively detect and measure early and progressive memory and cognitive changes in AD at an individual level. Nonetheless, the large repertoire of unique tests available for MemTrax (the Chinese version consists of pictures that could produce over 600 unique tests, thus minimizing a learning effect) along with pictorial stimuli to minimize language and culture influences supports the potential for MemTrax to be effective in a longitudinal assessment and tracking scheme for practical clinical utility in differentiating progressive changes in evolving dementia severity through the early continuum of dementia which is a direction for future studies.

Increasingly, evidence suggests that with AD the benefits of early detection outweigh purported risks, including improved patient disease management and emotional, social, and economic value [13]. The recently published seminal work on individualized approach for the clinical management of patients at risk for AD showed not only a preventative and treatment effect for patients without clinical symptoms and early symptoms, respectively, but most excitingly an improvement in cognition in both groups after interventions [27]. Future studies can use MemTrax to track the efficacy of this individualized approach for the clinical management of AD, assess the potential to use MemTrax metrics as clinically significant endpoints for experimental treatments or clinical trials for dementia, and for the timely detection of MCI in longitudinal studies. A case study already demonstrated MemTrax can detect a therapeutic effect from an integrated systems approach to treat AD [28].

Limitations

Previous studies have shown that age and education have significant effects on the MoCA score [29, 30]. A previous analysis of MemTrax showed the effect of age on MemTrax metrics, which are small, and essentially no effect of sex [15]. Due to the limitations of sample size, the norms for the optimal cut-off scores for MCI and dementia for the different age and educational level were not reported but need to be the elucidated in future studies. Further, demographic and clinical characteristics including hypertension, diabetes mellitus, and hypercholesteremia are statistically different among the groups. Although statistical analyses indicated that these factors did not affect the main conclusion, more balanced groups would strengthen the support for MemTrax’s clinical utilities.

CONCLUSIONS

Based on the results reported here, we conclude that the 1.5 to 2.5-minute MemTrax test can be utilized as an effective brief cognitive assessment instrument to detect cognitive impairments associated with MCI-AD and AD in a Chinese cohort in the clinic with better or similar accuracy, sensitivity, and specificity as the widely used MoCA. MemTrax has the potential to serve as a reliable and accessible digital cognitive screen instrument for mass screening, clinic use, and research in detecting early cognitive impairment and its most common cause—AD—that can be adopted by the clinicians and other parties of interest worldwide. This utility and efficacy of MemTrax would be directly responsive to the recent call by the Global Advisory Group on Future MCI Care Pathways in their 2019 consensus [2 –4].

Footnotes

ACKNOWLEDGMENTS

We would like to thank Dr. J. Wesson Ashford and Mr. Curtis Ashford for developing the MemTrax test, thank Mr. Curtis Ashford the owner of MemTrax, LLC for licensing MemTrax to SJN Biomed and his MemTrax LLC colleagues including Joanne Ashford and Diego de Oliveira to make the Chinese version of MemTrax possible, thank all the colleagues and friends of SJN Biomed and the patients to participate in this study, and thank Ms. Melissa Zhou for her critical reading of the manuscript. This study was supported in part by the MD Scientific Research Program of Kunming Medical University (Grant no. 2017BS028 to X.L.) and the Research Program of Yunnan Science and Technology Department (Grant no. 2019FE001(–222) to X.L).

Authors’ disclosures available online ().

References

Zhou

, Ashford

(2019) Advances in screening instruments for Alzheimer’s disease. Aging Med 2, 88–93.

Sabbagh

, Boada

, Borson

, Chilukuri

, Doraiswamy

, Dubois

, Ingram

, Iwata

, Porsteinsson

, Possin

, Rabinovici

, Vellas

, Chao

, Vergallo

, Hampel

(2020) Rationale for early diagnosis of mild cognitive impairment (MCI) supported by emerging digital technologies. J Prev Alzheimers Dis 7, 158–164.

Sabbagh

, Boada

, Borson

, Chilukuri

, Dubois

, Ingram

, Iwata

, Porsteinsson

, Possin

, Rabinovici

, Vellas

, Chao

, Vergallo

, Hampel

(2020) Early detection of mild cognitive impairment (MCI) in primary care. J Prev Alzheimers Dis 7, 165–170.

Sabbagh

, Boada

, Borson

, Doraiswamy

, Dubois

, Ingram

, Iwata

, Porsteinsson

, Possin

, Rabinovici

, Vellas

, Chao

, Vergallo

, Hampel

(2020) Early detection of mild cognitive impairment (MCI) in an at-home setting. J Prev Alzheimers Dis 7, 171–178.

Department of Health and Human Services, Low Cost Detection of Cognitive Decline in Clinical Settings. https://grants.nih.gov/grants/guide/rfa-files/RFA-AG-20-050.html. Accessed November 20, 2019.

Alzheimer’s Disease Drug Discovery Foundation, Dia-gnostic Accelerator: Digital Biomarkers-Advancing Digi-tal Biomarkers for Alzheimer’s and Related Dementias. https://www.alzdiscovery.org/research-and-grants/funding-opportunities/digital-biomarkers. Last Accessed November 20, 2019.

Ashford

(2008) Screening for memory disorders, dementia, and Alzheimer’s disease. Aging Health 4, 399–432.

Jack

Jr , Knopman

, Jagust

, Petersen

, Weiner

, Aisen

, Shaw

, Vemuri

, Wiste

, Weigand

, Lesnick

, Pankratz

, Donohue

, Trojanowski

(2013) Tracking pathophysiological processes in Alzheimer’s disease: An updated hypothetical model of dynamic biomarkers. Lancet Neurol 12, 207–216.

Loewenstein

, Curiel

, Duara

, Buschke

. (2018) Novel cognitive paradigms for the detection of memory impairment in preclinical Alzheimer’s disease. Assessment 25, 348–359.

10.

Loewenstein

, Curiel

, Greig

, Bauer

, Rosado

, Bowers

, Wicklund

, Crocco

, Pontecorvo

, Joshi

, Rodriguez

, Barker

, Hidalgo

, Duara

(2016) A novel cognitive stress test for the detection of preclinical Alzheimer disease: Discriminative properties and relation to amyloid load. Am J Geriatr Psychiatry 24, 804–813.

11.

Curiel Cid

, Loewenstein

, Rosselli

, Matias-Guiu

, Piña

, Adjouadi

, Cabrerizo

, Bauer

, Chan

, DeKosky

, Golde

, Greig-Custo

, Lizarraga

, Peñate

, Duara

(2019) A cognitive stress test for prodromal Alzheimer’s disease: Multiethnic generalizability. Alzheimers Dement 11, 550–555.

12.

Callahan

, Ramirez

, Berezuk

, Duchesne

, Black

, Alzheimer’s Disease Neuroimaging Initiative (2015) Predicting Alzheimer’s disease development: A comparison of cognitive criteria and associated neuroimaging biomarkers. Alzheimers Res Ther 7, 68.

13.

Shepard

, Teghtsoonian

(1961) Retention of information under conditions approaching a steady state. J Exp Psychol 62, 302–309.

14.

Wixted

, Goldinger

, Squire

, Kuhn

, Papesh

, Smith

, Treiman

, Steinmetz

(2018) Coding of episodic memory in the human hippocampus. Proc Natl Acad Sci U S A 115, 1093–1098.

15.

Ashford

, Bernard-Tarpin

, Ashford

(2019) A computerized continuous-recognition task for measurement of episodic memory. J Alzheimers Dis 69, 385–399.

16.

Nasreddine

, Phillips

, Bédirian

, Charbonneau

, Whitehead

, Collin

, Cummings

, Chertkow

(2005) The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. J Am Geriatr Soc 53, 695–699.

17.

Bergeron

, Landset

, Zhou

, Ding

, Khoshgoftaar

, Luo

, Zhao

, Du

, Chen

, Wang

, Zhong

, Liu

, Ashford

(2020) Utility of MemTrax and machine learning modeling in classification of Mild Cognitive Impairment. J Alzheimers Dis 77, 1545–1558.

18.

Bergeron

, Landset

, Tarpin-Bernard

, Ashford

, Khoshgoftaar

, Ashford

(2019) Episodic-memory performance in machine learning modeling for predicting cognitive health status classification. J Alzheimers Dis 70, 277–286.

19.

Weiner

, Nosheny

, Camacho

, Truran-Sacrey

, Mackin

, Flenniken

, Ulbricht

, Insel

, Finley

, Fockler

, Veitch

(2018) The Brain Health Registry: An internet-based platform for recruitment, assessment, and longitudinal monitoring of participants for neuroscience studies. Alzheimers Dement 14, 1063–1076.

20.

Cholerton

, Weiner

, Nosheny

, Poston

, Mackin

, Tian

, Ashford

, Montine

(2019) Cognitive performance in Parkinson’s disease in the Brain Health Registry. J Alzheimers Dis 68, 1029–1038.

21.

van der Hoek

, Nieuwenhuizen

, Keijer

, Ashford

(2019) The MemTrax Test compared to the Montreal Cognitive Assessment estimation of mild cognitive impairment. J Alzheimers Dis 67, 1045–1054.

22.

, Li

, Huang

(2012) The Beijing version of the Montreal cognitive assessment as a brief screening tool for mild cognitive impairment: A community-based study. BMC Psychiatry 25, 156.

23.

Carson

, Leach

, Murphy

(2018) A re-examination of Montreal Cognitive Assessment (MoCA) cutoff scores. Int J Geriatr Psychiatry 33, 379–388.

24.

Tan

, Li

, Gao

, Wang

, Zhao

, Yu

, Du

, Zhan

, Cui

, Wang

, Li

, Yang

, Yu

, Xia

, Zhou

(2015) Optimal cutoff scores for dementia and mild cognitive impairment of the Montreal Cognitive Assessment among elderly and oldest-old Chinese population. J Alzheimers Dis 43, 1403–1412.

25.

Pinto

TCC

, Santos

MSP

, Machado

, Bulgacov

, Rodrigues-Junior

, Silva

, Costa

MLG

, Ximenes

, Sougey

(2019) Optimal cutoff scores for dementia and mild cognitive impairment in the Brazilian version of the Montreal cognitive assessment among the elderly. Dement Geriatr Cogn Disord Extra 9, 44–52.

26.

Chen

, Zhao

, Shang

, Dai

, Xu

, Ai

, Zhang

, Liu

(2021) Validity of internet-based continuous visual recognition tasks as a screening test for amnestic mild cognitive impairment in Chinese population. Chin J Neurol, in press.

27.

Isaacson

, Hristov

, Saif

, Hackett

, Hendrix

, Melendez

, Safdieh

, Fink

, Thambisetty

, Sadek

, Bellara

, Lee

, Berkowitz

, Rahman

, Meléndez-Cabrero

, Caesar

, Cohen

, Lu

P-L

, Dickson

, Hwang

, Scheyer

, Mureb

, Schelke

, Niotis

, Greer

, Attia

, Mosconi

, Krikorian

(2019) Individualized clinical management of patients at risk for Alzheimer’s dementia. Alzheimers Dement 15, 1588–1602.

28.

Bredesen

(2014) Reversal of cognitive decline: A novel therapeutic program. Aging (Albany NY) 6, 707–717.

29.

Zheng

, Teng

, Varma

, Mack

, Mungas

, Lu

, Chui

(2012) Chinese-language Montreal Cognitive Assessment for Cantonese or Mandarin speakers: Age, education, and gender effects. Int J Alzheimers Dis 2012, 204623.

30.

Rossetti

, Lacritz

, Cullum

, Weiner

(2011) Normative data for the Montreal Cognitive Assessment (MoCA) in a population-based sample. Neurology 77, 272–1275.