A Validation Study of the Remotely Administered Montreal Cognitive Assessment Tool in the Elderly Japanese Population

Abstract

Background:

In an aging society, neuropsychological testing using video teleconferencing (VTC) is increasingly important. Despite the potential benefit of a VTC-administered Montreal Cognitive Assessment Tool (MoCA) to detect cognitive decline, only a limited number of studies have investigated this tool's reliability. Therefore, we aimed to evaluate the reliability of VTC-administered MoCA compared with face-to-face (FTF)-administered MoCA among elderly Japanese participants. Moreover, we examined participants' satisfaction with VTC-administered MoCA.

Methods:

Participants ≥60 years of age with and without cognitive impairment (i.e., those with mild cognitive impairment [MCI], those with dementia, and healthy controls [HC]) were assessed with VTC- and FTF-administered MoCA at an interval of >2 weeks and <3 months. The order effect (VTC first vs. FTF first) and time effect (first vs. second testing session), as well as several covariates such as age and years of education were controlled. Intraclass correlation coefficients (ICCs) were calculated using a mixed-effects model to assess the agreement between the two (VTC- vs. FTF-administered) groups. Participants' satisfaction with VTC-administered MoCA was examined using a Likert scale asking seven questions.

Results:

We included 73 participants in the study (36 men; age, 76.3 ± 7.5 years). The ICC for the MoCA total score was high in the entire sample (0.85), whereas ICCs were moderate to high for the subgroups (MCI: 0.82, dementia: 0.82, and HC: 0.53). Furthermore, we found good overall participant satisfaction with VTC-administered MoCA.

Discussion:

VTC-administered MoCA appears viable as an alternative to FTF-administered MoCA, although further replication studies with larger sample sizes are needed.

Introduction

Neuropsychological assessments are valuable in measuring cognitive capabilities, such as attention, memory, language, visuospatial abilities, and executive functions in elderly people. The assessment procedure includes conversation, writing, and drawing sections. In an aging society with an increasing prevalence of dementia (i.e., the estimated number of people living with dementia around the world was 46.8 million in 2015, and this number is predicted to increase to 131.5 million by 2050),¹ neuropsychological tests are heavily used as a screening tool for dementia. In addition, early diagnosis and intervention in mild cognitive impairment (MCI),² which have an estimated prevalence of 10–20% in individuals ≥65 years of age² (and of that population an annual average of 10–15% may develop into dementia^3,4), are important.

This is because nonpharmacological interventions such as cognitive and behavioral interventions in MCI have demonstrated positive effects on several functional aspects (e.g., cognition, daily activities, mood, and quality of life),^5
–7 and control of vascular risk factors, including blood pressure control, smoking cessation, and statin therapy, may reduce the risk of progression from MCI to dementia²; however, no pharmacological or nonpharmacological treatments have proven effective in preventing MCI from developing into dementia.² Therefore, it is crucial to detect MCI using clinical assessments, such as neuropsychological tests, as early as possible.

Given the increasing number of elderly people (i.e., the global population of people ≥60 years of age was over 962 million in 2017 and is expected to double by 2050),⁸ there is an expanding imbalance between people requiring neuropsychological tests to assess cognitive function, and trained experts to administer the tests. In addition, people who live in rural and/or remote areas often face barriers to health care access,^9,10 which results in delayed medical intervention. Telemedicine can be an effective way to address such issues. Neuropsychological testing using video teleconferencing (VTC) can be done remotely with elderly who cannot attend on-site because of physical limitations and/or travel difficulties, providing secure benefits to them.¹⁰ So far, neuropsychological tests using VTC in elderly populations have yielded promising results in terms of their feasibility and reliability.^11

–15

Among neuropsychological tests, the Montreal Cognitive Assessment Tool (MoCA), which was developed by Nasreddine et al.¹⁶ and validated,¹⁷ is a simple and effective test used commonly around the world to assess cognitive functions in elderly populations and detect MCI. It is a paper-based assessment tool that is administrated by physicians or psychologists and takes ∼15 min to complete. It consists of tests for visual/executive function (assessed through Trail B test and a task that requires participants to draw a copy of a cube shape and clock), naming (assessed by asking participants to name three pictures of animals), attention (assessed by asking participants to repeat a series of numbers forward and then a different series backwards), language (assessed by asking participants to repeat two sentences correctly and list all of the words that can be recalled that begin with the letter “F” within 1 min: the Japanese version of MoCA [MoCA-J] applies the Japanese kana character “ka” instead of the letter “F”), abstraction (assessed by asking participants to explain how two items are alike, such as a train and a bicycle), delayed recall (assessed by asking participants to repeat five words that are read aloud, and after completing other tasks, the participants are asked to repeat each of five words again), and orientation (assessed by asking participants to state the date, month, year, day, place, and city).

A normal score range on this 30-point test is considered to be ≥26.¹⁸ Given the characteristics of the MoCA, the VTC-administered MoCA is considered useful and beneficial for elderly people. However, to the best of our knowledge, only a limited number of studies have investigated the reliability and feasibility of the VTC-administered MoCA in elderly populations.^19,20 These two studies looked at patients diagnosed with Parkinson's disease and Huntington's disease (one study looked at both diseases¹⁹; the other looked only at those with Parkinson's²⁰) with small sample sizes (N = 17 and 11), both of which showed feasibility and good reliability (intraclass correlation coefficient [ICC] = 0.59)¹⁹ and a mean difference of 2 (interquartile range: 1–2.5) out of a possible 30 points in MoCA scores²⁰ when comparing VTC-administered MoCA to face-to-face (FTF)-administered MoCA.

There are no studies targeting Japanese populations. Given these facts, further research with a lager sample size is needed to accumulate evidence on the reliability and feasibility of the VTC-administered MoCA, and to demonstrate the clinical utility of the VTC-administered MoCA in elderly populations of various ethnicities. Therefore, the aims of this study were as follows: (1) to evaluate the reliability of VTC-administered MoCA compared to FTF-administered MoCA in elderly Japanese populations and (2) to evaluate participants' satisfaction with the VTC-administered MoCA using a Likert scale.

Methods

Participants

This study was conducted at three sites (Keio University Hospital in Tokyo, Shigakukan University in Kagoshima Prefecture, and Hoshi Clinic in Fukushima Prefecture) from June 2017 to March 2018 with the approval of each hospital's ethics committee; Keio University, Shigakukan University, and Hoshi Clinic are located in urban, suburban, and rural areas, respectively. Written informed consent was provided by all participants or their legal representatives after they received detailed information regarding the study protocol.

This study included outpatients with cognitive impairment (i.e., clinically diagnosed as MCI or dementia) and healthy controls (HC) without cognitive impairment, who were all ≥60 years of age and fluent in Japanese. We accessed medical records and contacted patients who were diagnosed with MCI or dementia (i.e., Alzheimer's disease [AD] or dementia with Lewy Bodies [DLB]) by clinicians at each site. Then, we confirmed their eligibility in this study. MCI was diagnosed based on Petersen's definition,²¹ and dementia subclass diagnoses followed the National Institute on Aging-Alzheimer's Association's (NIA-AA) definition²² for dementia due to AD, and the DLB Consortium's definition for DLB,²³ respectively. We also recruited HC at each site and confirmed their eligibility. No specific recruitment process was performed to balance the number of participants in each cognitive function group, although we tried to recruit patients with a wide range of cognitive functions. We excluded participants with severe auditory or visual sensory impairment, severe concurrent medical illnesses such as severe heart diseases and malignant tumors, or major psychiatric disorders such as major depression, bipolar disorder, and schizophrenia.

Test Procedure

The study design comprised the administration of the MoCA-J by VTC and FTF testing for the entire sample. The validity and reliability of the MoCA-J were previously evaluated through an FTF session.^16,17 Lower scores indicate poorer levels of cognitive function in the MoCA-J. To avoid the learning effect as well as additional cognitive decline over a long period, each evaluation was performed with an interval of >2 weeks and <3 months between VTC and FTF sessions. The VTC and FTF assessments in the study were conducted by 11 trained clinical raters. Among them, six clinical raters worked at Keio University Hospital and Hoshi Clinic. These raters administered FTF MoCA-J to participants in their respective facilities, as well as administering VTC MoCA-J to participants at all of the facilities. Five clinical raters were located at Shigakukan University. They administered FTF MoCA-J at their facility, as well as administering VTC MoCA-J to all of the facilities. Two sets of VTC systems were installed in separate rooms both at Keio University and Shigakukan University. In this study, there were two types of procedures for how to conduct the VTC MoCA-J; in one type, VTC MoCA-J was conducted in two separate rooms within the same facility, and in the other type, the assessment was done across two different facilities. When conducting VTC MoCA-J within one facility, the remote rater spoke to the participants from the other room. Although we attempted to distribute the number of ratings equally among raters, there were no strict allocations made as deploying a specific rater to a specific assessment was not always possible. However, to prevent bias from the previous rating in the second assessment, we made sure that the FTF and VTC assessments were conducted by different raters. Furthermore, participants were randomized into two groups by order of assessment administration: VTC-first and FTF-first groups. Additional information on participants' age, sex, years of education, Mini-Mental State Examination (MMSE),²⁴ and prescription medications was collected from their medical records.

For the VTC sessions, we used Cisco TelePresence^® System EX60, DX80, SX20, and Roomkit (Tokyo, Japan) connected to the Internet. With the assistance of a local staff member, participants were positioned 0.5 m away from a 21.5-, 23-, 33-, or 40-inch LCD monitor in an examination room set up for VTC. Participants were shown by the staff member how the VTC equipment and procedure would work, and then the staff initiated the VTC call to connect with the rater, who was located at another room within the same facility or another facility. The rater could simultaneously view the participant and materials for neuropsychological tests using a picture-in-picture display. When assessing the “naming” task in the MoCA-J, we displayed pictures of animals one by one on the participant's monitor. During this assessment, the rater's face was still displayed on the upper part of the participant's monitor using a two-screen display mode. When assessing the drawing tasks (i.e., cube and clock drawing in the visual/executive function), we observed the participant drawing a cube and clock on paper by using the document camera device, which was placed in the upper frame of the participant's monitor. The participant's drawing motions were displayed on the rater's monitor with vertical and horizontal inversions so the rater could assess the task correctly and smoothly. When administrating the neuropsychological test, the local staff was not present in the room, but was available on-site to assist with VTC equipment. The communication system used was based on the SIP and H.323 protocols. The video resolutions were set up to 1080p30/WUXGA.

Satisfaction Survey

We conducted a participant satisfaction survey of VTC-administered MoCA-J compared to FTF-administered MoCA-J, using the 7-point Likert scale with seven questions regarding participants' VTC-administered MoCA experience (see Table 3 for details). Higher scores indicated greater levels of satisfaction with VTC-administered MoCA-J in each item (Table 3). Each participant answered this survey following their VTC-administered MoCA-J assessment. If participants could not read the questions, the clinical raters read the survey aloud and wrote the answers on behalf of the participant.

Statistical Analysis

Demographics and characteristics of participants in the three subgroups (i.e., MCI, dementia, and HC) were compared using analysis of variance (ANOVA) or the Kruskal-Wallis test for continuous variables and chi-squared test for categorical variables. Likewise, scores in the participants' satisfaction survey using the Likert scale were compared among subgroups (i.e., diagnosis, sex, years of education, and videoconference system experience), using ANOVA followed by post-hoc Bonferroni tests or the Mann-Whitney U test. The Shapiro-Wilk test was used to test continuous variables for normality.

We calculated ICCs using a mixed-effects model to assess the agreement between the two (VTC- vs. FTF-administered) groups. We controlled for the order effect (VTC first vs. FTF first) and time effect (first vs. second testing session) when testing the method effect (VTC- vs. FTF-administered group), and covariates were age, years of education, presence/absence of any antidementia drugs, and study sites. We conducted these analyses in the entire sample and subgroups (i.e., MCI, AD, and HC). Moreover, for the entire sample, we calculated the ICCs of seven different subdomains, in accordance with a recent classification of the 12 subscales of MoCA-J into the following seven subdomains: (1) visual/executive function (trail making, cube, and clock), (2) naming, (3) attention (number, vigilance, and calculation), (4) language (repeat and word fluency), (5) abstraction, (6) delayed recall, and (7) orientation. As post-hoc analyses, we calculated the ICCs of the two language domain subscales (repeat and word fluency) and compared the scores of the repeat subscale between VTC and FTF groups using Wilcoxon signed-rank test. Likewise, MoCA-J total scores and the scores of the other 11 subscales of MoCA-J were also compared using Wilcoxon signed-rank test. We also applied Bonferroni corrections to the 12 subscales.

Statistical analyses were performed using the SPSS^® version 24.0 (IBM Corp. Released 2016. IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp.). Figure 1 was created using the ggplot2 package in R. A p-value of <0.05 was considered statistically significant (two tailed). Based on the guidelines provided by Koo and Li,²⁵ we regarded ICC values of less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.9 as indicative of poor, moderate, good, and excellent reliability, respectively.

Fig. 1.

MoCA-J total scores by face-to-face assessment and VTC (N = 73). FTF, face-to-face; HC, healthy control; MCI, mild cognitive impairment; MoCA-J, Japanese version of Montreal Cognitive Assessment Tool; VTC, video teleconferencing.

Results

We recruited a total of 82 participants (MCI: n = 15, dementia [AD: n = 17 and DLB: n = 2], and HC: n = 48) for this study. Thirty-nine participants were included in the VTC-first group and 44 were included in the FTF one. Of them, nine participants who had missing values in the MoCA-J scores assessed by VTC or FTF were excluded from the analysis. Specifically, in the VTC-first group, two participants had missing values from both their VTC and FTF MoCA-J scores, one had missing values from their VTC MoCA-J score, and two had missing values from their FTF MoCA-J score; and in the FTF-first group, three had missing values from their VTC MoCA-J score and one had missing values from their FTF MoCA-J score. Thus, 73 participants (age: mean ± standard deviation [SD], 76.3 ± 7.5; male: n = 36 [49.3%]; MCI: n = 15, dementia [AD: n = 13 and DLB: n = 1], and HC: n = 44) were included in the analysis. Demographic and clinical characteristics of the participants are summarized in Table 1.

Table 1.

Demographic and Clinical Characteristics of Participants (N = 73)

CHARACTERISTICS	TOTAL (N = 73)	MCI (n = 15)	DEMENTIA (AD: n = 13, DLB: n = 1)	HC (n = 44)	p-VALUE
Age, years	76.3 ± 7.5	81.4 ± 6.8	80.8 ± 6.1	73.2 ± 6.6	<0.001^a, ^*
Males	36 (49.3)	5 (33.3)	5 (35.7)	26 (59.1)	0.119^b
Years of education	14.0 (12.0–16.0)	12.1 ± 3.0	12.3 ± 3.1	16.0 (12.0–16.0)	0.009^c, ^*
MMSE scores	27.0 (23.3–29.0)^#	24.5 ± 3.7	19.8 ± 3.6^##	29.0 (27.0–30.0)	<0.001^c, ^*
Use of any antidementia drugs^###	20 (27.4)	8 (53.3)	12 (85.7)	0 (0)	<0.001^b, ^*

Values are shown as mean ± SD (range), median (interquartile range), or n (%).

p-Value of <0.05.

n = 72.

n = 13.

###

Antidementia drugs used are donepezil, galantamine, rivastigmine, and memantine.

Based on ANOVA.

Based on Chi-squared test.

Based on Kruskal-Wallis test.

AD, Alzheimer's disease; ANOVA, analysis of variance; DLB, Dementia with Lewy Bodies; HC, healthy control; MCI, mild cognitive impairment; MMSE, Mini-Mental State Examination; SD, standard deviation.

The scatter plot for FTF and VTC MoCA-J is shown in Figure 1. The ICC of MoCA-J total scores of the entire sample (N = 73) was high (0.85).

In subgroup analyses based on diagnoses, the MoCA-J total scores in the MCI and dementia (AD and DLB) groups [mean ± SD (range)] were as follows: 18.4 ± 4.6 (10.0–25.0) for VTC and 19.5 ± 3.9 (13.0–26.0) for FTF in the MCI group, and 15.1 ± 4.0 (8.0–21.0) for VTC and 15.2 ± 5.7 (5.0–27.0) for FTF in the dementia group (AD and DLB). The MoCA-J total scores in the HC group were 24.1 ± 3.6 (17.0–30.0) for VTC and 25.6 ± 3.3 (14.0–30.0) for FTF. The ICCs were moderate to good in each subgroup (MCI [n = 15]: 0.82, dementia [AD: n = 13 and DLB: n = 1]: 0.82, and HC [n = 44]: 0.53). All ICCs for the subdomains of the MoCA-J were moderate to good (visual/executive function: 0.74, naming: 0.71, attention: 0.63, abstraction: 0.52, delayed recall: 0.57, and orientation: 0.81), except language (0.45) being poor. These results are summarized in Table 2. The results of the post-hoc analyses gave ICCs of 0.32 and 0.48 for the repeat and word fluency subscales of the language domain, respectively. The score of the repeat subscale for VTC MoCA-J is significantly lower than that for FTF MoCA-J (0.63 vs. 1.10, p < 0.001). The results of the difference in MoCA-J total score and each subscale score of MoCA-J between VTC and FTF are summarized in Supplementary Table S1.

Table 2.

Intraclass Correlation Coefficients for MoCA-J Total Score, and MoCA-J Subgroup Scores Based on Diagnosis and Domain

	MoCA-J TOTAL SCORE				MoCA-J SUBDOMAIN SCORE
	WHOLE SAMPLE (N = 73)	MCI (n = 15)	DEMENTIA (n = 14)	HC (n = 44)	VISUAL/EXECUTIVE (N = 73)	ATTENTION (N = 73)	NAMING (N = 73)	LANGUAGE (N = 73)	ABSTRACTION (N = 73)	DELAYED RECALL (N = 73)	ORIENTATION (N = 73)
ICC	0.85	0.82	0.82	0.53	0.74	0.63	0.71	0.45	0.52	0.57	0.81

These values are calculated using a mixed-effects model, where the method effect was tested, while controlling the order and time effects, with age, years of education, presence/absence of any antidementia drugs, and study site as covariates.

HC, healthy control; ICC, intraclass correlation coefficient; MCI, mild cognitive impairment; MoCA-J, Japanese version of Montreal Cognitive Assessment Tool.

A total of 70 out of 73 participants completed the satisfaction survey because three participants did not fill out the form. The number of participants who had experience with videoconferencing before this study was 15 (21%). Participant satisfaction with VTC-administered MoCA-J documented by the Likert scale is summarized in Table 3. The overall satisfaction with VTC-administered MoCA-J was good in the whole sample (N = 70, mean ± SD: 5.0 ± 1.1), although there are score differences in some questions among the three groups. Briefly, there were significant differences in participants' level of satisfaction with visibility (i.e., of the monitor screen used in the assessment), voice clarity (i.e., of the rater), expression (i.e., asking if the participants could express what they wanted to say to the rater well), and overall satisfaction among diagnoses (i.e., MCI, dementia, and HC) (all p-values <0.05). In addition, the level of satisfaction with the voice clarity and expression between male and female, and the satisfaction with expression between participants whose years of education ≥13 years or <13 years were significantly different (all p-values <0.05). Further details are summarized in Table 3.

Table 3.

Participants' Satisfaction with Video Teleconferencing-Administered MoCA-J

QUESTIONS^#	WHOLE SAMPLE (N = 70)	DIAGNOSIS^##				SEX^###			YEARS OF EDUCATION^###			EXPERIENCE OF ANY VIDEOCONFERENCE SYSTEMS^###
QUESTIONS^#	WHOLE SAMPLE (N = 70)	MCI (n = 13)	DEMENTIA (n = 14)	HC (n = 43)	p-VALUE	MALE (n = 35)	FEMALE (n = 35)	p-VALUE	≥13 YEARS (n = 39)	<13 YEARS (n = 31)	p-VALUE	NO (n = 55)	YES (n = 15)	p-VALUE
Visibility	4.4 ± 1.2	4.6 ± 1.1	5.2 ± 1.7	4.0 ± 1.0	0.004^a, ^*	4.1 ± 0.9	4.7 ± 1.5	0.067	4.2 ± 1.1	4.6 ± 1.4	0.235	4.4 ± 1.3	4.5 ± 0.9	0.540
Voice clarity	3.9 ± 1.5	4.8 ± 1.4	4.7 ± 2.0	3.7 ± 1.0	<0.001^b, ^*	3.5 ± 1.2	4.3 ± 1.6	0.013^d, ^*	3.6 ± 1.3	4.2 ± 1.6	0.110	4.1 ± 1.6	3.7 ± 1.0	0.127
Relaxation	3.9 ± 1.2	3.8 ± 1.3	4.5 ± 1.7	3.7 ± 0.8	0.086	3.9 ± 1.0	3.9 ± 1.4	0.830	4.0 ± 1.1	3.7 ± 1.3	0.283	3.9 ± 1.3	3.8 ± 0.9	0.687
Expression	4.2 ± 1.4	4.5 ± 1.4	5.4 ± 1.6	3.8 ± 1.2	0.001^c, ^*	3.9 ± 1.1	4.6 ± 1.7	0.032 ^d, ^*	3.8 ± 1.2	4.7 ± 1.6	0.017 ^e, ^*	4.4 ± 1.5	3.8 ± 1.3	0.166
Understanding	4.4 ± 1.3	4.2 ± 1.1	5.1 ± 1.7	4.2 ± 1.1	0.059	4.2 ± 1.0	4.6 ± 1.5	0.132	4.1 ± 1.1	4.7 ± 1.4	0.059	4.5 ± 1.3	4.2 ± 1.3	0.470
Privacy	4.5 ± 1.0	4.3 ± 0.7	4.9 ± 1.5	4.4 ± 1.0	0.209	4.3 ± 1.0	4.7 ± 1.1	0.195	4.4 ± 1.0	4.7 ± 1.2	0.244	4.6 ± 1.1	4.3 ± 1.0	0.327
Overall	5.0 ± 1.1	5.2 ± 1.0	5.9 ± 1.2	4.7 ± 1.0	0.002^a, ^*	5.0 ± 1.0	5.1 ± 1.0	0.759	5.0 ± 1.0	5.2 ± 1.2	0.589	5.0 ± 1.1	5.2 ± 0.7	0.575

Values are shown as mean ± SD.

Visibility: Were you satisfied with the visibility of the monitor?; Voice clarity: Were you satisfied with the voice clarity?; Relaxation: Were you relaxed during the interview?; Expression: Were you able to express what you wanted to say to the rater well?; Understanding: Did you understand what the rater wanted to tell you well?; Privacy: Were you satisfied with the protection of your privacy?; Overall: Overall, were you satisfied with the VTC experience?

Each question's score ranged from 1 to 7 (7: VTC is much better than FTF, 6: VTC is moderately better than FTF, 5: VTC is slightly better than FTF, 4: VTC is as good as FTF, 3: VTC is slightly worse than FTF, 2: VTC is moderately worse than FTF, and 1: VTC is much worse than FTF).

ANOVA is used to compare scores among the three groups with Bonferroni correction (uncorrected p-value is shown in table).

###

Mann-Whitney U test was used to compare scores between two groups.

p-Value is significant after Bonferroni correction.

The dementia group shows a significantly higher score than the HC group (p < 0.017).

The MCI and dementia groups show significantly higher scores than the HC group (p < 0.017, respectively).

The dementia group shows significantly higher scores than the MCI and HC groups (p < 0.017, respectively).

The female group shows a significantly higher score than the male group (p < 0.05).

The group with shorter years of education (<13 years) shows a significantly higher score than the group with longer years of education (≥13 years) (p < 0.05).

ANOVA, analysis of variance; FTF, face-to-face; HC, healthy control; MCI, mild cognitive impairment; VTC, video teleconferencing.

Discussion

In this study, we evaluated the reliability and feasibility of VTC-administered MoCA-J compared with FTF-administered MoCA-J across elderly populations with various cognitive functions. Overall, a good agreement was found between the two administration methods in terms of high ICCs, whereas the subgroup analyses found that ICCs varied across each subgroup (i.e., MCI, dementia [AD and DLB], and HC).

Our findings for the reliability and feasibility of VTC-administered MoCA-J were consistent with the findings of the previous studies, although they focused on patients with Parkinson's disease and Huntington's disease.^19,20 Furthermore, our finding of a high ICC in the entire sample is consistent with previous findings that showed high ICCs in several neuropsychological tests, including MMSE, when comparing VTC-administered and FTF-administered tests.^12,14 In our sample, ICCs varied among diagnoses (MCI: 0.82, dementia [AD and DLB]: 0.82, and HC: 0.53), although these values are considered to have moderate or good reliability.²⁵ Although ICCs generally reach a higher value when the distribution of the scores is large, a restricted range of the scores (e.g., by focusing on a specific population with similar cognitive function) could result in lower ICCs. In fact, the MoCA-J total scores in the HC group [mean ± SD (range)] were as follows: 24.1 ± 3.6 (17.0–30.0) for VTC and 25.6 ± 3.3 (14.0–30.0) for FTF. In contrast, the MoCA-J total scores in the MCI and dementia (AD and DLB) groups were as follows: 18.4 ± 4.6 (10.0–25.0) for VTC and 19.5 ± 3.9 (13.0–26.0) for FTF in the MCI group, and 15.1 ± 4.0 (8.0–21.0) for VTC and 15.2 ± 5.7 (5.0–27.0) for FTF in the dementia group (AD and DLB). Therefore, such a relatively narrow range of scores between VTC and FTF in the HC group might affect the results.

Furthermore, in our results, ICCs varied with cognitive domains in the entire sample (N = 73): ICCs of visual/executive function (0.74), naming (0.71), attention (0.63), abstraction (0.52), delayed recall (0.57), and orientation (0.81) were moderate to good, whereas that of language (0.45) was poor. The language domain has two subscales (repeat and word fluency), and the ICCs of repeat and word fluency were 0.32 and 0.48, respectively. In addition, the score of the repeat subscale for VTC-administered MoCA-J is significantly lower than that for FTF-administered MoCA-J (0.63 vs. 1.10, p < 0.001). The sentences in the repeat subscale consist of 7 to 12 words, which the participants have to listen to and repeat correctly. Here, we must consider the high prevalence of hearing loss in the elderly; the prevalence of hearing loss for people in their 70s is ∼50%.²⁶ We excluded participants with severe hearing loss from the study; however, some of the participants may have had hearing loss to some extent, given that the mean age of the participants in this study was 76.3 years. Thus, one possible reason for the lower ICC and lower score in VTC compared to FTF in the repeat subdomain might be that the participants had difficulty hearing what the clinical raters said through VTC. In fact, the satisfaction survey showed that the participants were less satisfied with voice clarity compared to other items. Thus, it is important to improve clinical raters' speech clarity in future studies and to develop technologies to compensate for difficulties in hearing high frequency sounds, which is an auditory characteristic of age-related hearing loss.

Nevertheless, the participants were overall satisfied with VTC. We consider that this satisfaction level was nearly ideal as we asked the comparative satisfaction of VTC compared to FTF and the means for the lowest items (voice clarity and relaxation) were 3.9, where 4.0 represents the equal satisfaction between VTC and FTF. This finding is consistent with a previous study that reported a positive experience completing VTC-administered MoCA overall in patients with Parkinson's disease (N = 11; median age of 69.0 years).²⁰ Although the HC group scored lower than the other groups on some questions, which reached a statistical significance (p < 0.05) (Table 3), there were only ∼1-point differences in the Likert scale on average between the HC group and the other groups, and the average scores on some questions were ∼4 in the HC group, which represents that VTC is considered to be as good as FTF even in the HC group. Nevertheless, individuals without cognitive decline might be sensitive to some factors such as visibility and voice clarity, which indicates the importance of improving clinical raters' techniques and the VTC system in future studies and clinical practice.

A previous study, which compared an electronic tablet-based version of MoCA to the paper-based MoCA, indicated that some factors, including age and computer literacy, had an effect on MoCA scores.²⁷ Likewise, computer literacy might have an impact on the results in this study. However, in this study, despite the high rate of participants who had no experience using a VTC system (79%), generally good agreements were found between the two administration methods (i.e., VTC and FTF) in terms of high ICCs, and participants' satisfaction for VTC-administered MoCA-J was high. Given these results, VTC-administered MoCA-J could be a feasible alternative method to FTF-administered MoCA.

The strengths of the study include that (1) we calculated ICCs, taking the effects of time and order into consideration; (2) we evaluated the consistency of the MoCA-J subdomain scores as well as the total score between the VTC and FTF groups; (3) we included not only patients with dementia (AD and DLB) but also those with MCI and HC participants; and (4) we evaluated the participants' satisfaction with VTC-administered MoCA-J. Considering these strengths, our study will foster the implementation of VTC-administered MoCA-J in elderly populations.

This study has some limitations. First, some participants' evaluation sessions may have occurred too close together, which may have caused the learning effect to influence the results. However, we ensured that sessions were held a minimum of 2 weeks apart to avoid this problem. Second, only patients with MCI and dementia (AD and DLB) who showed mild to moderate cognitive decline (i.e., MMSE total score: 24.5 ± 3.7 for MCI and 19.8 ± 3.6 for dementia [AD and DLB]) were included in this study, so we are not sure if these results can be applied to individuals with severe cognitive decline. Finally, this study had a relatively small sample size, particularly in the MCI and dementia (AD and DLB) subgroups, which could potentially lead to type II error. However, to the best of our knowledge, the sample size of this study was the largest among the studies that have investigated the reliability and feasibility of VTC-administered MoCA so far.^19,20

In conclusion, VTC-administered MoCA-J could be used as an alternative method to traditional FTF-administered MoCA-J in clinical practice based on our results, which reflect the reliability of and participants' satisfaction with VTC. Nevertheless, given the relatively low ICC for the language domain, especially the item “repeat,” clinical raters need to improve their speech clarity as much as possible when they perform VTC-administered-MoCA-J. Our findings may require replications with larger sample sizes that include participants with a variety of cognitive functions and those who live in various geographic areas. Such future studies and more advanced VTC equipment will expedite the development of the telepsychiatry field and the clinical application of several neuropsychological tests, including MoCA.

Footnotes

Disclosure Statement

K.Y. has received article fees from Sumitomo Dainippon Pharma, fellowship grants from the Japan Research Foundation for Clinical Pharmacology, and consultant fees from Bracket and VeraSci within the past 3 years. Y.E. has received grants from Mitsubishi Foundation, Daiwa Securities Health Foundation, and Okawa Foundation within the past 3 years. M.M. has received grants and/or speaker's honoraria from Daiichi Sankyo, Dainippon-Sumitomo Pharma, Eisai, Eli Lilly, Fuji Film RI Pharma, Janssen Pharmaceutical, Mochida Pharmaceutical, MSD, Nippon Chemipher, Novartis Pharma, Ono Yakuhin, Otsuka Pharmaceutical, Pfizer, Takeda Yakuhin, Tsumura, and Yoshitomi Yakuhin within the past 3 years. T.K. has received consultant fees from Dainippon Sumitomo, Novartis, and Otsuka, and speaker's honoraria from Banyu, Eli Lilly, Dainippon Sumitomo, Janssen, Novartis, Otsuka, and Pfizer within the past 3 years. He also received royalties from UpToDate. The other authors have nothing to disclose.

Contributors

K.I., K.Y., Y.Y., Y.E., D.S., M.K., K.F., M.M., and T.K. contributed to and approved the design and the protocol of the study and the literature search. Y.Y., K.I., Y.E., and K.F. recruited the participants and collected the data. DS contributed to the development of the database for the study. K.I., K.Y., and T.K. undertook the statistical analysis. K.I., K.Y., and T.K. wrote the first draft of the article. All authors contributed to and have approved the final article.

Funding Information

This research was supported by the Japan Agency for Medical Research and Development (AMED) under grant number 161k1010023h0001. AMED did not have any influence on designing the study, conducting the study, or analyzing the results.

Supplementary Material

Supplementary Table S1

References

Alzheimer's Disease International. The World Alzheimer Report. 2015. Available at www.worldalzreport2015.org/downloads/world-alzheimer-report-2015-media-release-FINAL.pdf (last accessed May 5, 2019 ).

Langa

, Levine

. The diagnosis and management of mild cognitive impairment: A clinical review. JAMA, 2014; 312:2551–2561.

Bruscoli

, Lovestone

. Is MCI really just early dementia? A systematic review of conversion studies. Int Psychogeriatr, 2004; 16:129–140.

Peterson

, Roberts

, Knopman

, Boeve

, Geda

, Ivnik

, et al. Mild cognitive impairment: Ten years later. Arch Neurol, 2009; 66:1447–1455.

Sherman

, Mauser

, Nuno

, Sherzai

. The efficacy of cognitive intervention in mild cognitive impairment (MCI): A meta-analysis of outcomes on neuropsychological measures. Neuropsychol Rev, 2017; 27:440–484.

Chandler

, Parks

, Marsiske

, Rotblatt

, Smith

. Everyday impact of cognitive interventions in mild cognitive impairment: A systematic review and meta-analysis. Neuropsychol Rev, 2016; 26:225–251.

Chandler

, Locke

, Crook

, Fields

, Ball

, Phatak

, et al. Comparative effectiveness of behavioral interventions on quality of life for older adults with mild cognitive impairment: A randomized clinical trial. JAMA Netw open, 2019; 2:e193016.

World Population Aging. United Nations. Available at https://www.un.org/en/development/desa/population/publications/pdf/ageing/WPA2017_Highlights.pdf (last accessed July 20, 2019 ).

Morgan

, Crossley

, Kirk

, D'Arcy

, Stewart

, Biem

, et al. Improving access to dementia care: Development and evaluation of a rural and remote memory clinic. Aging Ment Health, 2014; 13:17–30.

10.

Castanho

, Sousa

, Santos

. When new technology is an answer for old problems: The use of videoconferencing in cognitive aging assessment. J Alzheimers Dis Rep, 2017; 1:15–21.

11.

Carotenuto

, Rea

, Traini

, Ricci

, Fasanaro

, Amenta

. Cognitive assessment of patients with Alzheimer's disease by telemedicine: Pilot study. J Med Internet Res, 2018; 20:e31.

12.

Wadsworth

, Galusha-Glasscock

, Womack

, Quiceno

, Weiner

, Hynan

, et al. Remote neuropsychological assessment in rural American Indians with and without cognitive impairment. Arch Clin Neuropsychol, 2016; 31:420–425.

13.

Brearly

, Shura

, Martindale

, Lazowski

, Luxton

, Shenal

, et al. Neuropsychological test administration by videoconference: A systematic review and meta-analysis. Neuropsychol Rev, 2017; 27:174–186.

14.

Munro Cullum

, Hynan

, Grosch

, Parikh

, Weiner

. Teleneuropsychology: Evidence for video teleconference-based neuropsychological assessment. J Int Neuropsychol Soc, 2014; 20:1028–1033.

15.

Timpano

, Pirrotta

, Bonanno

, Marino

, Marra

, Bramanti

, et al. Videoconference-based mini mental state examination: A validation study. Telemed e-Health, 2013; 19:931–937.

16.

Nasreddine

, Phillips

, Bédirian

, Charbonneau

, Whitehead

, Collin

, et al. The Montreal cognitive assessment, MoCA: A brief screening. J Am Geriatr Soc, 2005; 53:695–699.

17.

Freitas

, Simões

, Alves

, Santana

. Montreal cognitive assessment: Validation study for mild cognitive impairment and Alzheimer disease. Alzheimer Dis Assoc Disord, 2013; 27:37–43.

18.

Damian

, Jacobson

, Hentz

, Belden

, Shill

, Sabbagh

, et al. The Montreal Cognitive Assessment and the mini-mental state examination as screening instruments for cognitive impairment: Item analyses and threshold scores. Dement Geriatr Cogn Disord, 2011; 31:126–131.

19.

Abdolahi

, Bull

, Darwin

, Venkataraman

, Grana

, Dorsey

, et al. A feasibility study of conducting the Montreal Cognitive Assessment remotely in individuals with movement disorders. Health Informatics J, 2016; 22:304–311.

20.

Stillerova

, Liddle

, Gustafsson

, Lamont

, Silbum

. Could everyday technology improve access to assessments? A pilot study on the feasibility of screening cognition in people with Parkinson' s disease using the Montreal Cognitive Assessment via Internet videoconferencing. Aust Occup Ther J, 2016; 63:373–380.

21.

Petersen

. Mild cognitive impairment as a diagnostic entity. J Intern Med, 2004; 256:183–194.

22.

McKhann

, Knopman

, Chertkow

, Hyman

, Jack

Jr ., Kawas

, et al. The diagnosis of dementia due to Alzheimer's disease: Recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease. Alzheimers Dement, 2011; 7:263–269.

23.

McKeith

, Boeve

, Dickson

, Halliday

, Taylor

, Weintraub, et al. Diagnosis and management of dementia with Lewy bodies Fourth consensus report of the DLB Consortium. Neurology, 2017; 89:88–100.

24.

Folstein

, Folstein

, McHugh

. “Mini-mental state.” A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res, 1975; 12:189–198.

25.

Koo

, Li

. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med, 2016; 15:155–163.

26.

Goman

, Lin

. Prevalence of hearing loss by severity in the United States 2016;106:1820–1822.

27.

Snowdon

, Hussein

, Kent

, Pino

, Hachinski

. Comparison of an electronic and paper-based Montreal Cognitive Assessment Tool. Alzheimer Dis Assoc Disord, 2015; 29:325–329.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.03 MB

0.00 MB