An Initial Exploratory Examination of the Sensitivity and Specificity of MyCog Mobile Using the Mini-Cog as a Proxy for Suspected Cognitive Impairment

Introduction

Primary care clinics are an ideal setting to identify cognitive impairment in the early stages, particularly since Medicare has covered cognitive screening as part of an Annual Wellness Visit since 2011. ¹ However, primary care clinics face significant time and resource constraints, and thus cognitive screening measures must be fast, accurate, and easy to administer. ² The Mini-Cog is one of the most commonly used screeners in primary care settings, as it is relatively easy to administer, and it takes about 3 min to complete. ³

The Mini-Cog has demonstrated utility in community primary care settings. The brief screener is is relatively robust to education, gender, and ethnicity influences, ⁴ and demonstrates an estimated 73% sensitivity and 84% specificity in detecting either MCI, dementia, or CI. ⁵ The Mini-Cog consists of 2 parts: a 3-word list recall (scored 1 point per word), and a clock drawing test (scored 0 or 2). Standard practice guidelines suggest participants who score 2 points or less on the Mini-Cog should be flagged for impairment, and further evaluation is recommended. ⁵

Despite the widespread use of the Mini-Cog, there are some disadvantages to using this tool in primary care. The Mini-Cog requires a trained examiner to administer following standardized instructions, which takes up valuable staff resources and time during ultra brief in-person visits. Moreover, although relatively simple to administer, human errors are common; 1 study found 61% of nurses made an error in the administration or scoring of the Mini-Cog. ⁶ The Mini-Cog is also recorded and scored by hand, and results must be entered into an Electronic Health Record (EHR), creating further steps in the workflow and opportunities for error.

MyCog Mobile is an app-based screener that may help overcome some of the limitations of the Mini-Cog. MyCog Mobile is self-administered on a patient’s personal smartphone remotely prior to a primary care visit, and results are automatically sent into patient’s EHR triggering appropriate best practice alerts. ⁷ The MyCog Mobile model saves valuable clinic time and staff resources and eliminates human errors in administration and scoring. While multiple digital screeners designed for at-home use are emerging on the market, few have been psychometrically validated, which limits their utility in clinical settings. ⁸ MyCog Mobile is unique in that it has demonstrated evidence of usability, feasibility, reliability, and validity in several previous studies.^7,9,10 However, it remains unclear how MyCog Mobile relates to well-established primary care screeners like the Mini-Cog. In this paper, we evaluated the sensitivity, specificity, and area under the curve of the MyCog Mobile tests using a Mini-Cog score 2 or less as a proxy for those who would be suspected of impairment and recommended for further evaluation under usual care standards for primary care .

Methods

A sample of 200 older adults was recruited by a market research agency through their large, nationally representative database of research volunteers. All participants were compensated and provided informed consent prior to participation. Inclusion criteria required participants to be age 65 years or older, speak English and provide informed consent prior to participation. Testing was conducted in 5 study facilities in Georgia, Texas, New Jersey, Florida, and Arizona. During the in-person visit, Mini-Cogs were administered first by a trained examiner to build rapport with participants before they started a completely self-administered smartphone task. Participants then self-administered MyCog Mobile on iPhones provided at the study facility. The MyCog Mobile battery takes approximately 20 min to complete. Given the brevity of both tests and the distinct task paradigms, we did not have significant concerns about the effects of counterbalancing. Following administration, Mini-Cogs were double-scored by 2 independent study staff, and any discrepancies were resolved with the study team to ensure consensus.

Results

A sample of 200 older adults ages 65 to 87 years (mean age = 72.56 years; SD = 5.11 years) were enrolled in this study (Table 1). A total of 20 participants earned a Mini-Cog score of 2 or less.

OPEN IN VIEWER

Table 1.

Sample Demographics.

	n (%)
Total N	200 (100)
Gender
Female	109 (45.5)
Male	91 (54.5)
Racial identity
Asian	1 (0.5)
Black or African American	35 (17.5)
Middle Eastern or North African	1 (0.5)
Native Hawaiian or Pacific Islander	1 (0.5)
White or European American	157 (78.5)
Other identity	8 (4)
Prefer not to answer	1 (0.5)
Ethnic identity
Hispanic	40 (20)
Not Hispanic	160 (80)
Education
Less than high school	1 (0.5)
High School Diploma or GED	38 (19)
Some college or 2-year degree	65 (32.5)
4-year college degree	61 (30.5)
Graduate degree	35 (17.5)
Mini-cog score
0	1 (0.5)
1	5 (2.5)
2	14 (7)
3	31(15.5)
4	49 (24.5)
5	100 (50)

The AUC of MyCog Mobile was 0.83 (bootstrapped 95% CI: [0.75, 0.95]), with a sensitivity of 0.76 (bootstrapped 95% CI: [0.63, 0.97]) and specificity of 0.88 (bootstrapped 95% CI: [0.63, 0.10]). The pseduo R² value of .25 indicated an excellent logistic regression model fit to the data. MyFaces Name Matching and MySorting were the only significant predictors of failed Mini-Cogs (see Table 2).

OPEN IN VIEWER

Table 2.

Logistic Regression Predictors of Suspected Cognitive Impairment Indicated by Mini-Cog.

	β (SE)	P value
MyPictures
Trial 1	0.24 (0.44)	.59
Trial 2	0.47 (0.39)	.23
MyFaces
First letter	−0.44 (0.40)	.26
Recognition	0.24 (0.31)	.44
Name matching	0.78 (0.35)	.02*
MySequences	0.22 (0.29)	.44
MySorting	0.79 (0.32)	.01*

*

P < .05.

The model initially achieved an AUC of 0.83 when fitted to the original dataset. After applying optimism-correction, the ROC AUC decreased to 0.77. This reduction indicates that while the initial AUC value may have been slightly overestimated, it fell within the bootstrapped confidence interval and above acceptable a priori thresholds. The distribution of AUC values for the bootstrap samples evaluated on the original dataset, and the estimated apparent and adjusted test performance are illustrated in Figure 1.

OPEN IN VIEWER

Figure 1.

The optimism-adjusted AUC (0.77) is lower than the unadjusted AUC (0.83) but above acceptable a priori threshold of 0.70.

Discussion and Limitations

Findings from this exploratory study suggest that MyCog Mobile demonstrates adequate sensitivity and specificity to identify participants who fail a Mini-Cog. MyFaces Name Matching, a memory subtest, and MySorting, an executive functioning test, were the best predictors of a failed Mini-Cog in our model. These results should be considered preliminary for several reasons. It is important to stress we used a general population sample and did not have information about the participants clinical diagnostic status. Although the Mini-Cog is commonly used to flag individuals for further evaluation, it is not a diagnostic tool on its own nor does it have perfect accuracy to detect impairment. Thus, it is unclear if any participants in our study would be diagnosed with a cognitive impairment in a clinical setting. An estimated 17% of community-dwelling older adults live with MCI, ¹⁹ and the ratio of suspected cases to controls in our sample (20:180) was small for analysis purposes, which increases the risk of bias in our findings. We mitigated this risk by applying an optimism correction to our bootstrap resampled estimates, which suggested that the AUC in a sample of independent data would likely be acceptable, yet this will need to be confirmed in future work.

The generalizability of these results is further limited by lack of representation of racial identities other than Black/African American or White/European American, and few participants with less than a high school education. Moreover, participants self-administered MyCog Mobile in a research setting, not in their own homes prior to a primary care clinic visit. It will be important to examine the feasibility of completing MyCog Mobile in a home setting, taking into consideration the technology experience, effort, and time required of the patients to complete MyCog Mobile, especially among vulnerable subgroups with worse physical and mental health. Findings support clinical validation of MyCog Mobile in a sample that includes healthy older adults as well as those with confirmed diagnoses of cognitive impairment in primary care settings.

Conclusions

MyCog Mobile shows promise as a potential screening tool for cognitive impairment in older adults. While preliminary, these findings suggest MyCog Mobile demonstrates adequate sensitivity and specificity to detect a failed Mini-Cog, with the MyFaces and MySorting subtests showing the most potential for detection. Further validation in a clinical setting with a larger sample, including individuals with confirmed cognitive diagnoses, is necessary to establish the clinical utility of MyCog Mobile and determine which subtests should be prioritized in the battery.