Abstract
Background:
Rapid technological advances offer a possibility to develop cost-effective digital cognitive assessment tools. However, it is unclear whether these measures are suitable for application in populations from Low and middle-income countries (LMIC).
Objective:
To examine the accuracy and validity of the Brain Health Assessment (BHA) in detecting cognitive impairment in a Cuban population.
Methods:
In this cross-sectional study, 146 participants (cognitively healthy = 53, mild cognitive impairment (MCI) = 46, dementia = 47) were recruited at primary care and tertiary clinics. The main outcomes included: accuracy of the BHA and the Montreal Cognitive Assessment (MoCA) in discriminating between controls and cognitively impaired groups (MCI and dementia) and correlations between the BHA subtests of memory, executive functions, and visuospatial skills and criterion-standard paper-and-pencil tests in the same domains.
Results:
The BHA had an AUC of 0.95 (95% CI: 0.91–0.98) in discriminating between controls and cognitively impaired groups (MCI and dementia, combined) with 0.91 sensitivity at 0.85 specificity. In discriminating between control and MCI groups only, the BHA tests had an AUC of 0.94 (95% CI: 0.90–0.99) with 0.71 sensitivity at 0.85 specificity. Performance was superior to the MoCA across all diagnostic groups. Concurrent and discriminant validity analyses showed moderate to strong correlations between the BHA tests and standard paper-and-pencil measures in the same domain and weak correlations with standard measures in unrelated domains.
Conclusion:
The BHA has excellent performance characteristics in detecting cognitive impairment including dementia and MCI in a Hispanic population in Cuba and outperformed the MoCA. These results support potential application of digital cognitive assessment for older adults in LMIC.
INTRODUCTION
Population aging will lead to a dramatic increase in global dementia prevalence and incidence [1]. Much of the increase is projected to occur in low and middle-income countries (LMICs). In 2015, 58% of people with dementia were living in LMICs and this proportion is estimated to increase to 63% by 2030 and 68% by 2050 [2]. At the same time, there is evidence suggesting that awareness of dementia and other cognitive disorders affecting older adults in LMIC remains low, which may further elevate the burden of the disease in these communities [3]. This issue is further complicated by the fact that currently available clinical tools in LMIC often fail to sufficiently account for the richness of linguistic, ethnic, cultural, and socioeconomic diversity represented across these societies. Among diagnostic procedures, cognitive assessment plays a key role in the diagnosis and characterization of neurodegenerative diseases. Thus, the scarcity of culturally appropriate cognitive assessment tools in LMIC communities with limited access to other diagnostic procedures, such as neuroimaging or biomarker studies, poses a threat for inaccurate, delayed, or missed diagnosis, which in turn creates a barrier to appropriate treatment and early planning [4, 5].
Timely detection of cognitive decline not only increases treatment and intervention opportunities but also allows patients and caregivers to plan for the future and address practical issues such as financial and legal representation, transition to assisted living, and participation in medical decision making [6–8]. In light of this growing need, the role of primary care in detection of cognitive impairment and dementia is widely recognized as a critical frontline for these efforts [9]. However, estimates from the U.S. indicate that only a third of seniors are routinely assessed for cognitive impairment in primary care settings [9, 10]. Furthermore, approximately 50% of dementia diagnoses are missed in primary care settings, delaying detection until later in the disease course [11–14]. Among the most significant barriers for detection of cognitive impairment in primary care settings are relatively short consultation times and the need for more guidance on cognitive assessment [15–17]. Overcoming these challenges requires strategies that allow for cost- and time-efficient case identification, which can be made possible by using cognitive assessment tools that are brief, have high sensitivity and specificity, and feature a built-in capability to guide providers with automated reports on individual patient’s performance.
Rapid technological advances and a growing body of knowledge regarding the earliest cognitive changes in neurodegenerative diseases suggests a potential solution to this challenge through development of easily accessible, well validated, and low-cost cognitive assessment tools [11, 18]. One such tool is the UCSF Brain Health Assessment (BHA) battery, which has shown excellent sensitivity and specificity in detecting mild cognitive impairment (MCI) and dementia in English speaking and Spanish speaking older adults in the U.S. [19, 20]. The BHA is comprised of subtests measuring key cognitive domains that are affected in cognitive disorders and are recommended to be assessed in published diagnostic criteria [21, 22]. In addition, the BHA features a 10 min administration time and automated scoring that includes a user-friendly interpretive report (memory.ucsf.edu/tabcat), making it feasible for integration into busy primary care and specialty practices. However, little is known about the performance of this battery in diverse populations, particularly Hispanic individuals in LMICs. A Cuban version of BHA was recently developed by means of translation of the original tests into Spanish and cultural adaptation of verbal stimuli for word frequency and complexity to match linguistic characteristics of the Cuban population. This Cuban version of the BHA was designed to address the need for development of culturally sensitive cognitive tests that would appropriately reflect cultural and linguistic differences among the vast Spanish-speaking population. However, there have been no validation studies of this measure in Cuban cohorts yet.
Cuba is a middle-income country with the highest proportion of older adults in Latin America (25% aged 60 years and over) [23]. Due to this rise in the aging population, dementia has become the main cause of disability, dependency, and financial burden among older adults in Cuba [24]. The Cuban government has embraced dementia as a national public health priority [25]. Thus, Cuba is facing a critical need for a brief well-validated cognitive measure that can be easily implemented into clinical practice to streamline diagnostic and care pathways, and the existence of universal healthcare access based on prevention and primary care creates an ideal environment for potential implementation of the assessment for detection of cognitive impairment in non-specialty settings. In this study, we aimed to evaluate the feasibility and validity of the BHA and its accuracy in detecting MCI and dementia in a Hispanic population in Cuba. We also compared the BHA to the Montreal Cognitive Assessment (MoCA), a widely used brief cognitive assessment in Cuba [26] to evaluate how these two measures perform in detection of cognitive impairment in older Cuban adults. The MoCA was selected for comparison with the BHA because it is a standard brief cognitive assessment in Cuba and has shown greater accuracy in detection of MCI compared to other brief cognitive measures [10, 27].
METHODS
Setting and study participants
The present study received institutional review board (IRB) approval from the Alzheimer Research Center (CEA) and Havana Medical University. Written informed consent was obtained from all participants and their study partners.
Sixty presumably cognitively healthy participants were recruited from the community by their primary care physician. After recruitment and enrollment at the community level, participants were invited for an extensive, multidisciplinary in-person diagnostic assessment at the Instituto Nacional de Neurologia (INN) to validate their diagnostic status as a control (cognitively healthy), MCI, or dementia (details about diagnostic procedures are provided below). Of those recruited at the community, 6 individuals were ultimately diagnosed with MCI, and 53 were confirmed as cognitively healthy. One of the participants did not complete the diagnostic visit and was excluded from the analysis. Additionally, 87 participants with a recent (within 6 months) diagnosis of MCI (n = 40) or dementia (n = 47) were recruited from ongoing research studies following the same diagnostic procedures at the Alzheimer’s Research Center (ARC) and the INN.
Diagnostic procedure
Diagnoses of all participants were made in a multidisciplinary consensus conference by the assessing neurologist and neuropsychologist based on published criteria based on data from the following clinical assessments and structural neuroimaging, when available [28–31]. Clinical assessment of all participants, including controls, was comprised of a clinical history, physical and neurological examination, a functional interview with an informant including the Clinical Dementia Rating Scale (CDR) [32], and a neuropsychological assessment. The neuropsychological battery consisted of the tasks measuring the following cognitive domains:
Episodic memory
Verbal learning and memory were assessed via the adapted version of the Consortium to Establish a Registry for Alzheimer’s Disease (CERAD) Word List Memory Task following the protocol of the 10/66 study [33–36]. In this task, participants complete four immediate recall trials of a 9-item word list followed by a 30 s short-delay free recall and a 10 min long-delay free and cued recalls. Visual memory was assessed by a 10 min free recall of the Benson Complex Figure [37].
Processing speed, attention, and executive function
Attention and working memory were assessed by the forward and backward digit span task [38]. Inhibition was assessed via the Stroop Interference Task [39]. Verbal and nonverbal initiation were measured by the lexical fluency task (letter “P”) and the Delis-Kaplan Executive Function System Design Fluency (Filled Dots Condition) subtest [40]. Speed and set-shifting was assessed via the Modified Trails test [41].
Visuospatial skills
Visuospatial and visuoconstructional skills were assessed via the Visual Object and Space Perception Battery (VOSP) Number Location subtest [41] and the copy of the Benson Complex Figure [22].
Language
Confrontation naming was assessed by the 15-item version of the Multilingual Naming Test (MINT) [42, 43].
Measures
The BHA and the MoCA were administered to all participants by a neuropsychologist independent of diagnostic evaluation. Both measures were administered on the same day within 2 weeks of the multi-disciplinary in-person clinical assessments.
MoCA
The MoCA [26, 45] is a widely used brief cognitive test assessing visuoconstruction and executive functions, attention, language, abstraction, memory, and orientation [26]. Time of administration is 10 min. We used an adapted version of the MoCA, MoCA-S (Version 7.1), and the total score was adjusted for educational attainment as indicated by Delgado et al. [46].
BHA
The BHA [18] is a brief tablet-based cognitive battery comprised of 4 tests: Favorites (associative memory), Match (processing speed and executive function), Line Orientation (visuospatial skills), and Animal Fluency (language). Total time of administration is 10 min. In addition to cognitive tests, the BHA includes an optional 4-min informant-administered functional survey, the Brain Health Survey (BHS), which elicits symptoms observed in Alzheimer’s type and atypical dementia syndromes [18]. Details about the BHA development and validation in English speakers, and for Mexican- and Central American-born Spanish speakers living in the U.S., are published elsewhere [19]. For this study, the BHA was translated and adapted into Cuban-Spanish by a multidisciplinary team of four language experts: one geriatrician, one psychiatrist, one neurologist, and three neuropsychologists. The adaptation was performed in accordance with the current guidelines for cross-cultural test development [47]. In particular, verbal stimuli on the Favorites task were adapted to have similar frequency and complexity as the English version, and to represent culturally appropriate concepts in the Cuban context. Cultural appropriateness of the final version was established in a preliminary pilot study with 15 Cuban healthy older adult volunteers (age: mean [SD] = 72.6[5. 6]) with variable degrees of educational levels (mean [SD] = 13.9 [4.9], female = 8 [53.3%]). Details of the pilot study are described in the Supplementary Material.
Statistical analyses
Normative corrections
Raw scores on the BHA subtests were converted to z-scores using a regression-based approach adjusting for age, education, and sex based on the sample controls. Regression-based norms have been reported to have greater accuracy in prediction when compared to traditional norming approaches, particularly for small samples [48, 49]. Additionally, this approach takes into account all regressed demographic indicators at once, thus making use more information from the data than cut-off scores do, increasing prediction accuracy [48, 49]. To estimate regression-based norms for our sample, we first conducted multiple linear regression analyses for each of the BHA tests. The models included the following demographic predictors: age (years), age2 (to assess for non-linear effects of age), education (years), and sex. Additionally, we supplemented these analyses by testing each model for all potential interactions between predictors (e.g., age x education, sex x education, etc.). These models were then reduced in a stepwise fashion by excluding non-significant predictors (significant level was set at 0.10 to minimize Type I error). Finally, the demographically-adjusted z-scores were estimated based on the following formula: (Y –Y’)/RSE, where Y is the observed raw score on a given BHA test, Y’ is the predicted score derived from the multiple regression model for this test, and RSE is the residual standard error of the multiple regression equation.
Discriminant analyses
We conducted separate discriminant analyses be-tween control and cognitively impaired groups (MCI and dementia combined and separately) and calculated receiver operating characteristic (ROC) curves for the discriminant scores for both the BHA tests and the MoCA total score. Similar to Possin et al. [19], we evaluated accuracy metrics at the best threshold and at a set specificity level of 0.85 to limit false positives. Additionally, due to the fact that informant survey was not completed by all participants, we performed discriminant function analyses with and without the BHS data.
Concurrent and discriminant validity analyses
We performed Pearson’s correlation analyses be-tween the BHA tests and paper-and-pencil measures administered as a part of the diagnostic neuropsychological test battery. Specifically, we correlated scores on each of the BHA measures with scores on analogous traditional tests of verbal memory (CERAD Word List Memory Task 10-minute Recall) [35, 36], visual memory (Benson Complex Figure Delayed Recall) [37], executive function and speed (Stroop Test and Modified Trails) [41], and visuospatial skills (VOSP and Benson Complex Figure Copy) [41]. To assess discriminant validity, we performed correlations between scores on the BHA tests and on conventional measures in unrelated domains: visuospatial skills for Favorites and Match (due to past evidence of high percent of variance shared by executive and episodic memory tasks [50]) and verbal and visual memory tests for Line Orientation. We hypothesized that each of the BHA measures will be most strongly correlated with a traditional test in the same domain and will be least strongly corrected with a traditional test in an unrelated domain.
All analyses were performed in R (v3.6.0, R Project for Statistical Computing) with 2-tailed significance level set at 0.05.
RESULTS
Sociodemographic characteristics
Three hundred participants were invited for a full diagnostic assessment at the INN. Of these, 146 participants completed the full diagnostic visit and study measures. Fifty-nine participants (32 controls, 9 MCI, and 18 dementia) also completed the informant survey (BHS).
The final sample included 53 controls, 46 individuals with MCI (amnestic = 21, non-amnestic = 25), and 47 individuals with dementia (Alzheimer’s type dementia = 26, vascular dementia = 11, behavioral variant frontotemporal dementia = 5, primary progressive aphasia = 3, other = 2). The cognitively impaired group was older (mean age = 73.4, SD = 6.7, p = 0.006) and had fewer years of education (mean age = 13.6, SD = 4.7, p < 0.001) compared to controls (Table 1).
Sample characteristics and group differences
*Based on differences between control and cognitively impaired groups and estimated using independent sample t-tests for continuous variables or Pearson’s χ2 tests for categorical variables. †Greater scores represent poorer performance.
Normative corrections
Results of the multiple regression models did not show significant non-linear effects of age on any of the BHA tests. We also did not find significant interactions among the demographic predictors which would suggest a non-linear effect on performance. Based on the results of multiple regressions, the scores on Favorites were corrected for female sex (b = 1.782, SE = 0.967, p = 0.072). Match scores were adjusted for age (b = –0.497, SE = 0.164, p = 0.004) and education (b = 0.610, SE = 0.236, p = 0.013). Line Orientation performance was corrected for education (b = –0.630, SE = 0.180, p = 0.001). No significant effects of age, education, or sex were found on Animal Fluency, thus, the z-scores for this test were calculated using the mean (18.2) and standard deviation (4.5) of the control group.
Sensitivity and specificity
The BHA tests had an AUC of 0.95 (95% CI: 0.91–0.98) discriminating between controls and the cognitively impaired group (MCI and dementia, Table 2 and Fig. 1A). At 0.85 specificity, the BHA tests had a sensitivity of 0.91. When the BHS informant survey data were incorporated into the discriminant analyses, the full BHA had an AUC of 0.99 (95% CI: 0.98–1.00) with a sensitivity of 0.96 at 0.85 specificity. The MoCA had an overall AUC of 0.83 (95% CI: 0.76–0.90) in discriminating between controls and the cognitively impaired group. When specificity was set at 0.85, the MoCA’s sensitivity was 0.60.
Results of the discriminant analyses with sensitivity and specificity values at different thresholds on the BHA and the MoCA
Receiver operating characteristic curves showing discrimination between A) controls and individuals with dementia and B) controls and individuals with MCI based on the BHA tests and the MoCA.
Associations between individual BHA tests and traditional neuropsychological tests. Controls are shows as blue triangles and cognitively impaired individuals (MCI and dementia) are shown as orange circles.
In discriminating between control and dementia groups, the BHA had an AUC of 0.98 (95% CI: 0.93–1.00) with a sensitivity of 0.96 at 0.85 specificity. The results on the MoCA revealed an AUC of 0.97 (95% CI: 0.94–1.00) with sensitivity of 0.92 at 0.85 specificity. Analyses with the BHS data were not performed for discriminating control and dementia groups due to a low number of individuals with dementia with the informant survey data (n = 18).
In discriminating between control and MCI groups only, the BHA tests had an AUC of 0.94 (95% CI: 0.90–0.99) with a sensitivity of 0.87 at 0.85 specificity. The MoCA had an AUC of 0.73 (95% CI: 0.63–0.84) with sensitivity of 0.36 at 0.85 specificity (Fig. 1B). We did not perform the analyses using the BHS data for discriminating controls and MCI due to a low number of individuals with MCI who completed the informant survey (n = 9).
Concurrent and discriminant validity with traditional measures
As hypothesized, each BHA sub-tests was significantly and most strongly correlated with paper-and-pencil measures in the same domain (Table 3). Specifically, Favorites was associated with both verbal (CERAD Word List Memory Task 10-minute Recall, r = 0.85, p < 0.001) and visual (Benson Complex Figure Delayed Recall, r = 0.72, p < 0.001) memory. Match was most strongly correlated with speed and set-shifting (Modified Trails Time, r =–0.68, p < 0.001), and was also associated with speed and inhibition (Stroop Total Correct, r = 0.65, p < 0.001). Lower but statistically significant correlations were observed between Line Orientation and tasks of visuoconstruction (Benson Figure Copy, r = –0.25, p < 0.01) and visual discrimination (VOSP, r = –0.35, p < 0.001). Consistent with our hypotheses, we also found evidence for discriminant validity for all three tasks presented in Table 3.
Correlation coefficients between the BHA tests and traditional neuropsychological tests
*p < 0.05, **p < 0.01, ***p < 0.001.
DISCUSSION
Our study found that the BHA has excellent performance characteristics in detecting cognitive impairment in a Hispanic population from Cuba. These results demonstrate a successful validation of a low-cost, brief tablet-based cognitive assessment for detection of MCI and dementia in the Cuban population and support potential application of digital cognitive measures for older adults in LMIC. Such applications may be particularly important in rural areas where access to specialist services is often limited and the responsibility and diagnostic decision-making fall on primary care physicians.
Our findings are consistent with previous study by Possin et al. [19] in a U.S.-based English-speaking sample and expand on these past results by validating the BHA in an ethnically and culturally different sample from the prior study. In both studies, the AUC for classifying normal from cognitively impaired participants was 0.95. Similarly, the AUC in a smaller sample of Spanish speakers with low to moderate levels of education in the San Francisco area was 0.87 [20]. Taken together, our findings support the notion and potential for development of valid, psychometrically robust cognitive measures that exhibit cultural fairness and are minimally subjected to cultural bias.
Our study also found support for the concurrent and discriminant validity of the individual components of the BHA. Each of the BHA subtests correlated with corresponding conventional measures in domains of episodic memory, executive functions, and visuospatial skills. These findings highlight the potential of novel brief computerized cognitive tests to efficiently capture domain-specific cognitive performance. The strongest correlations were observed in the domain of episodic memory followed by executive functions. Lower correlations were observed between the BHA Line Orientation test and standard measures of visuospatial skills [41]. This was likely related to a restricted range of scores on both the Benson Figure Copy and the VOSP Number Location measures, both of which have ceiling effects, particularly in cognitively normal controls. Thus, it is possible that the more psychometrically robust Line Orientation, which is associated with neuroanatomical structures underlying the dorsal visual stream [18], captured more nuanced aspects of visuospatial performance.
Our study is not without limitations. First, our sample had moderate to high levels of education which limits the generalizability of results to other populations across Latin American countries. Also, we did not directly assess the potential effects of familiarity with technology on our findings, although prior studies of computerized cognitive measures have suggested minimal impact of these variables on overall performance.[51, 52] Nevertheless, future studies should explore the validity of the BHA in diverse LMIC populations with low education and examine potential differences in performance by access/experience in using technology. Cross-validation of our findings in other cohorts are also needed. Second, the BHA administration was carried out by a trained neuropsychologist rather than by a primary care physician or other non-specialist healthcare professional. Therefore, future studies should examine the feasibility and cost-effectiveness of implementing the use of the BHA by physicians and other healthcare providers in primary care settings. Future directions also include non-specialists’ perspectives on practical and logistical issues related to implementation of the BHA in real-life clinical settings in LMIC. Finally, future studies on test-retest reliability and longitudinal stability of the BHA in LMIC populations are needed.
Our study has several strengths including the use of a well-characterized community-based cohort and a thorough diagnostic assessment based on widely accepted diagnostic criteria and standards. A particular strength is that the cognitively normal older adults used as controls underwent a full diagnostic assessment to ensure their neurologically healthy status. This study contributes to a paucity of prior data on brief computerized cognitive testing in LMIC. With regards to clinical implications, the current findings support the evidence that the BHA reliably detects cognitive impairment in the early symptomatic stages of neurodegenerative disease. Furthermore, the BHA’s brevity, automated scoring, and minimal training requirements are important attributes that may facilitate its widespread implementation in primary care settings. Profiles of performance on the BHA tasks could be used to generate reports for improving diagnostic accuracy and enhance care pathways, as well as increase access to new therapies or future clinical trials [53].
Conclusions
The BHA exhibited excellent performance characteristics in detecting MCI and dementia in older Cuban adults, similar to previous findings in U.S.-based English-speaking and Spanish-speaking cohorts, which supports robust cross-cultural validity of the measure. Although the BHA has a strong potential for use in primary care settings, the feasibility and cost-effectiveness of this potential implementation needs to be established in the context of routine primary care practice.
Footnotes
ACKNOWLEDGMENTS
We are grateful to our participants and their families and to the 10/66 research group in Cuba for their contributions to this study.
This study was supported by a grant from a partnership among the Global Brain Health Institute, the U.S. Alzheimer’s Association, and the Alzheimer’s Society, UK (GBHI ALZ UK-19-585014).
