Korean Version of the Mini-Mental State Examination Using Smartphone: A Validation Study

Abstract

Background:

Stroke often leads to disability, and poststroke survivors often have limited accessibility to medical facilities.

Introduction:

For such patients, mobile videoconferencing technology offers an opportunity to perform follow-up assessment and appropriate management of cognitive impairment. We aimed to determine the validity of the Korean version of the Mini-Mental State Examination (MMSE-K) when administered using a smartphone.

Materials and Methods:

Thirty patients with ischemic or hemorrhagic stroke were included in this study (20 males, 10 females; mean age, 69.8 ± 12.9 years). Both face-to-face and remote assessments of cognitive function through MMSE-K were performed for each patient at an interval of at least 3 days. Additionally, an in-person collaborator evaluated the MMSE-K score during the remote assessment. A smartphone and a tablet were used by the patient and the examiner, respectively, and remote connection was mediated using a dedicated videoconferencing application. The MMSE-K scores obtained through face-to-face, remote, and in-person assessments were compared using the Wilcoxon signed rank test and the Spearman correlation analysis.

Results:

There was good agreement between face-to-face and remote assessments, as well as between remote assessment and in-person collaborator's evaluation regarding total MMSE-K score and subscores for each MMSE-K domain (orientation, memory, attention/calculation, language, and visuospatial function).

Discussion:

Remote assessment can be a useful clinical evaluation method, and this study confirmed the validity.

Conclusions:

The smartphone represents a promising tool for the assessment of cognitive function in clinical practice, but further research into the intra- and inter-rater reliability of observations is warranted.

Introduction

Stroke is an important cause of disability and represents a common cause of death around the world. Most stroke survivors experience cognitive dysfunction, and stroke has been reported to increase the incidence of dementia six times.¹ Therefore, it is important to routinely and consistently evaluate cognitive impairment in poststroke patients to maintain and improve functional outcome.² However, most stroke survivors with disability have poor access to medical healthcare services. While there is no statistical analysis regarding accessibility to medical services, a large-scale Korean study reported significant levels of dissatisfaction among Korean adults with disabilities.³ Specifically, of the 4,580 community-dwelling adults (>18 years old) with disabilities, surveyed as part of the National Health Insurance Service database, 19.7% reported not being able to visit the hospital when they needed to during the previous year, and 51% reported not having regular health examinations during the previous 2 years.

Some studies have investigated the feasibility and utility of remote medical services for patients who have difficulty in accessing medical services.^{4

–15} In a study done by Hubble et al., a high concordance rate between scores obtained through in-person evaluation and interactive videoconference was reported for the unified Parkinson's disease rating scale and the Hoehn and Yahr score in patients with Parkinson's disease.¹⁰ Similarly, good agreement between remote (videoconferencing) and in-person assessment was noted for diagnosing Alzheimer's disease in elderly individuals (> 65 years) with cognitive dysfunction living in the rural area.⁹ Furthermore, audio–visual telecommunication is known to be useful in making decisions for thrombolysis in acute stroke patients.^14,15 On the other hand, telemedicine regarding poststroke care and rehabilitation, and neurology in general, is yet to be more studied.^12,14,15 In addition, no other study has since evaluated the feasibility of videoconference-based cognitive assessment in poststroke survivors.

In the early years of telemedicine, medical services would be provided through telephone or telehealth programs using video or television. These telemedicine systems based on conventional techniques relied on wired communication, involved the use of a fixed camera or desktop, and required a significant amount of resources. In 2006, the cost of the equipment necessary to establish a telehealth program based on wired communication was reported at approximately $12,000.⁸ Therefore, at that time, such a system was not accessible to many patients.

With rapid advances in technology, mobile telehealth technologies have become available, especially with the development of the smartphone, which is convenient to use and available worldwide. Moreover, the smartphone represents a cost-effective alternative to conventional telemedicine systems. In the past several years, smartphones have found application in areas such as delivering medical education, evaluating medical clinical skills, and conveying patient instructions,^16

–19 and it is expected that the increase in use of smartphones may help improve accessibility to medical services.

The Mini-Mental State Examination (MMSE) is commonly used in the early diagnosis and follow-up of patients with cognitive dysfunction. The MMSE has adequate quality in detecting mild to moderate cognitive impairment in stroke patients, with a low MMSE score being predictive of long-term cognitive impairment.²⁰ Furthermore, MMSE has high inter- and intrarater reliability. Moreover, the examination is brief and does not require special tools.

In the present study, we used a smartphone and a mobile videoconference tool to perform MMSE in stroke patients. We aimed to assess whether MMSE can be applied effectively using a smartphone to evaluate cognitive function in stroke patients.

Materials and Methods

Subjects

Between December 2014 and August 2015, 30 patients with ischemic or hemorrhagic stroke, admitted to Seoul St. Mary's Hospital of the Catholic University of Korea, were recruited to participate in our study. Patients with severe aphasia or dysarthria,^20,21 as well as those with auditory or visual impairment, were excluded from our analysis.

Korean Version of the MMSE

The Korean version of the Mini-Mental State Examination (MMSE-K) was used to evaluate cognitive function in each patient. The MMSE-K comprised 30 questions (total, 30 points), including items related to time and place orientation (10 points), memory (registration, 3 points; recall, 3 points), attention/calculation (5 points), language (8 points), and visuospatial function (1 point). For patients with low levels of education (<1 year), 1 point was added to the subtotal scores for orientation, attention/calculation, and language, without exceeding the maximum total score achievable.²²

Test Environment

The smartphone (iPhone 5S, Apple, Inc., California) was set up in front of the patient using a tripod. A paper and a pen were prepared on the table for the patient. The smartphone was set up so that the patient's upper half of the body, including the paper and pen, was in full view of the camera. During the face-to-face assessment (T_f), the examiner was in the same room as the patient. During the remote assessment (T_m), the examiner was located in another room and used a tablet (iPad mini 2, Apple, Inc.), while an additional evaluator located in the same room as the patient also observed and recorded the patient's answers (T_ma) (Fig. 1). A mobile videoconference application (FaceTime^®; Apple, Inc.) was used to connect the equipment on the patient side (smartphone) to that on the remote examiner side (tablet). No additional video or auditory equipment, such as earphones, speaker, or external display, was used. In a stable network environment, the videoconference application used in this study provides high-definition resolution (1,280 by 720 pixels) and frame rate (30 frames/s). We used the hospital's wireless network system, with a maximum data transfer rate of 100 megabits per second. Throughout the entire remote assessment, there was no interruption or delay due to loss of wireless signal or other technical problems.

Fig. 1.

Concept of remote assessment of cognitive function using a smartphone. The remote examiner performs the assessment from a different location, while an in-person collaborator is located in the same room as the patient.

Assessment

We performed MMSE-K twice for each patient, namely through face-to-face (T_f) and remote (T_m) assessments. The order of the assessments was randomized, and there was an interval of least 3 days between the tests. Both the face-to-face (T_f) and remote (T_m) assessments were performed by the same examiner, who is a doctor of physical medicine and rehabilitation, with ample experience in administering MMSE-K. The additional examiner who served as the in-person collaborator and performed the in-person evaluation (T_ma) during the remote assessment is also a doctor of physical medicine and rehabilitation. The setup of the remote assessment is presented in Figure 1. Regarding the questionnaire, the examiner used real objects and showed them to the patient through the camera built into the tablet. A similar approach was followed for the assessment of visuospatial function as the examiner showed drawn models to the patient through the tablet camera.

The functional grade in activities of daily living was evaluated based on the Korean version of the modified Barthel index (K-MBI). The severity of aphasia was measured in terms of the aphasia quotient (AQ) score for the Korean version of the Western Aphasia Battery (K-WAB) published by the Paradise Welfare Foundation.

Statistical Analysis

We used SPSS, version 21 (IBM Corp., Armonk, NY), for the analysis. To compare face-to-face assessment (T_f), remote assessment using mobile phone (T_m), and the in-person evaluation during remote assessment (T_ma), the Wilcoxon signed rank test was used. Furthermore, the Spearman correlation coefficient was used to compare MMSE-K total score between groups as well as to compare the subtotal scores of various cognitive domains. p Values <0.05 were considered statistically significant.

Ethics Statement

The design and protocol of this study were approved by the review board of Seoul St. Mary's Hospital (KC16RISI0556).

Results

Thirty patients (20 males, 10 females) were included in the study. The average age of the patients was 69.83 ± 12.95 years, and the average level of education was 11.70 ± 5.38 years (range, 0–16 years). Of the 30 patients included in the study, 19 and 11 patients had ischemic and hemorrhagic stroke, respectively. The locations of the stroke included the supratentorial (n = 25) and infratentorial (n = 5) regions. The functional grade in terms of K-MBI was 48.97 ± 24.52 (range, 2–90) (Table 1). There were two patients with aphasia, and nine with dysarthria (11 patients in total), but their symptoms were mild, and their average AQ score following K-WAB was 90.15 ± 6.95. Among the 30 patients included in the study, 19 scored ≥25 points, 7 scored from 18 to 24 points, and 4 scored <18 points in the face-to-face MMSE-K.

Table 1.

Characteristics of Stroke Patients Included in the Study (n = 30)

VARIABLE	VALUE^a
Age, years	69.83 ± 12.95
Education, years	11.70 ± 5.38
K-MBI, score	48.97 ± 24.52
AQ, score	90.15 ± 6.95
MMSE-K, score (%)
≥25	19 (63.3)
≥18 and <25	7 (23.3)
<18	4 (13.3)
Sex (%)
Male	20 (66.6)
Female	10 (66.6)
Stroke type (%)
Ischemic	19 (63.3)
Hemorrhagic	11 (36.6)
Stroke lesion (%)
Supratentorial	25 (83.3)
Infratentorial	5 (16.6)

Values given as mean ± standard deviation or number (percentage).

AQ, aphasia quotient score for the Korean version of the Western Aphasia Battery published by the Paradise Welfare Foundation; K-MBI, Korean version of the modified Barthel index; MMSE-K, Korean version of the Mini-Mental State Examination.

The average total MMSE-K scores obtained through face-to-face assessment (T_f), remote mobile phone assessment (T_m), and the evaluation of the in-person collaborator during the remote assessment session (T_ma) were 24.95 ± 4.76, 24.36 ± 4.85, and 24.01 ± 4.98, respectively (Table 2). There was no significant difference in total MMSE-K scores between T_f and T_m or between T_m and T_ma. Spearman correlation analysis of the MMSE-K total scores revealed strong and significant positive linear correlation between T_m and T_f (ρ = 0.949, p < 0.001) (Fig. 2A) and between T_m and T_ma (ρ = 0.986, p < 0.001) (Fig. 2B).

Fig. 2.

(A) Agreement (Spearman correlation) between MMSE-K scores obtained through face-to-face and remote assessment and (B) between MMSE-K scores obtained through remote assessment and in-person evaluation during the remote assessment. MMSE-K, Korean version of the Mini-Mental State Examination.

Table 2.

Comparison Between MMSE-K Scores Obtained Through Face-to-Face Assessment, Remote Assessment Using Smartphone, and In-Person Collaborator's Evaluation During the Remote Assessment

	T_f	T_m	T_ma	T_f VERSUS T_m		T_ma VERSUS T_m
	SCORES^a			Z SCORE	p	Z SCORE	p
Orientation	8.53 ± 1.87	8.43 ± 1.85	8.38 ± 1.92	−0.787	0.431	−0.333	0.739
Memory	4.85 ± 1.13	4.77 ± 1.19	4.73 ± 1.20	−0.213	0.831	−1.000	0.317
Attention/calculation	3.30 ± 1.84	2.77 ± 1.83	2.80 ± 1.86	−1.027	0.305	−1.414	0.157
Language	7.63 ± 0.76	7.73 ± 0.52	7.53 ± 0.78	−0.816	0.414	−0.707	0.480
Visuospatial function	0.63 ± 0.49	0.67 ± 0.48	0.57 ± 0.50	−0.378	0.705	−0.577	0.564
Total score	24.95 ± 4.76	24.36 ± 4.85	24.01 ± 4.98	−1.574	0.116	−0.405	0.685

All comparisons were performed using the Wilcoxon signed rank test with the significance threshold set at p < 0.05.

Scores given as mean ± standard deviation.

T_f, face-to-face assessment; T_m, remote assessment using smartphone; T_ma, in-person collaborator's evaluation during the remote assessment.

When analyzing the subtotal scores for each cognitive domain (orientation in time and place, memory registration and recall, attention/calculation, language, and visuospatial function), no significant differences were found between the different types of assessments (Table 2). There were strong positive correlations between T_f and T_m for the MMSE-K scores regarding orientation (ρ = 0.840, p < 0.001), memory (ρ = 0.762, p < 0.001), attention/calculation (ρ = 0.947, p < 0.001), language (ρ = 0.767, p < 0.001), and moderate positive correlation regarding visuospatial function (ρ = 0.508, p = 0.004) (Table 3 and Fig. 3A). The situation regarding the correlation between T_m and T_ma was similar as there were positive correlations for the MMSE-K scores regarding orientation (ρ = 0.953, p < 0.001), memory (ρ = 0.992, p < 0.001), attention/calculation (ρ = 0.990, p < 0.001), visuospatial function (ρ = 0.791, p < 0.001), and language (ρ = 0.680, p < 0.001) (Table 3 and Fig. 3B).

Fig. 3.

Statistical correlation (Spearman correlation) between subtotal scores for various domains of the MMSE-K. (A) Spearman correlation between scores obtained through remote and face-to-face assessments. (B) Spearman correlation between scores obtained through remote assessment and in-person evaluation during the remote assessment.

Table 3.

Correlation Between MMSE-K Scores Obtained Through Face-to-Face Assessment, Remote Assessment Using Smartphone, and In-Person Collaborator's Evaluation During the Remote Assessment

	T_f VERSUS T_m		T_ma VERSUS T_m
	ρ	p	ρ	p
Orientation	0.840	<0.001^*	0.953	<0.001^*
Memory	0.762	<0.001^*	0.992	<0.001^*
Attention/calculation	0.947	<0.001^*	0.990	<0.001^*
Language	0.767	<0.001^*	0.680	<0.001^*
Visuospatial function	0.508	0.004^**	0.791	<0.001^*
Total score	0.949	<0.001^*	0.986	<0.001^*

All results reflect Spearman correlation.

p < 0.001; ^** p < 0.05.

Moreover, no significant differences between T_f and T_m or between T_m and T_ma were noted in terms of MMSE-K scores for patients with mild aphasia or dysarthria (n = 11; T_f: 24.90 ± 4.81, T_m: 24.09 ± 4.77, T_ma: 23.90 ± 4.85; T_f vs. T_m: p = 0.395; T_m vs. T_ma: p = 0.157) or for patients with cognitive deficit (MMSE-K score <25 through face-to-face assessment; n = 11; T_f: 19.68 ± 2.95, T_m: 19.27 ± 3.16, T_ma: 18.90 ± 3.36; T_f vs T_m: p = 0.104; T_m vs. T_ma: p = 0.157).

Discussion

The use of video teleconferencing systems to improve access to medical services is expanding around the world, and significant research is underway to provide improved solutions. Several studies focusing on telemedicine-based cognitive function evaluation in the elderly or in patients with dementia^7
–9 suggest that this method is effective and especially advantageous for patients who cannot visit the in-patient clinic due to limitations related to distance, cost, and mobility.

Recently, Kim et al. used a conventional telemedicine videoconferencing system for MMSE-K-based cognitive assessment of dementia patients in a rural area of Korea and demonstrated the usefulness of this remote approach.²³ However, no report has described the effectiveness of remote assessment of cognitive function in Korean poststroke patients, especially in terms of applying MMSE-K using smartphones. The present study confirmed the validity of MMSE-K performed remotely through videoconferencing and based on the widely available smartphone technology instead of on a conventional videoconferencing system. We expect that this approach will help stroke patients with cognitive dysfunction to gain better access to medical services.

We found good agreement between the results of the face-to-face assessment and those of the remote assessment performed using smartphones regarding MMSE-K total score and subtotal scores for each cognitive domain. Furthermore, the same was true for the agreement between the results of the remote assessment and those of the evaluation performed by the in-person collaborator during the remote assessment session. Taken together, these results suggest that remote assessment using smartphones represents a valid approach to evaluate cognitive dysfunction, which is in agreement with previous research regarding patients with Parkinson's disease or dementia, where the remote assessment was based on conventional videoconferencing systems.^8
–10

In 2013, a videoconference-based MMSE (V-MMSE) was developed, which enables the use of a laptop or workstation computer for cognitive evaluation. The V-MMSE was successfully validated in a sample of healthy and cognitive impairment adults aged over 50 years old. Some items are thought to be difficult to answer during videoconferencing evaluations, and therefore a few items such as interlocking pentagons and writing tasks are excluded in the V-MMSE version.²⁴ Interestingly, in our present study, all conventional MMSE-K items were used, which did not affect the agreement between the remote and face-to-face assessments. Specifically, there was significant positive linear correlation between the scores obtained through face-to-face and remote assessments, suggesting that the current level of technology, such as smartphone, is sufficient to overcome various limitations of videoconferencing, including display, sound, or network problems. Therefore, no other special videoconferencing system with expensive external equipment or program is currently necessary for cognitive evaluation through MMSE-K.

In our study, although the difference was not statistically significant, remote assessment generally provided lower scores than the face-to-face assessment in terms of the attention/calculation items. In addition, the scores for visuospatial function showed moderately positive correlation, while other domains showed strongly positive correlation (Fig. 3A). These results may be related to the fact that the display monitor of the smartphone is small, which likely has a negative effect on the ability of stroke patients to focus on visual details. Furthermore, there was a difference between the score provided by the remote examiner and that provided by the in-person collaborator, although the difference was not statistically significant. Specifically, remote examiners tended to assign higher scores in the language and visuospatial function domains, which may be caused by possible errors on the examiner side, such as confusing right and left or miscalculating the number of pentagon edges. Therefore, when performing a remote examination, special consideration should be applied when interpreting scores for the attention/calculation and visuospatial functions.

Remote assessment can be a useful clinical evaluation method, but has some possibilities of errors when compared with the conventional in-person assessment. In particular, it is difficult to perform all points of the cognitive examination, and the evaluator must pay special attention when interpreting the results of a remote assessment. However, by using an adequate and widely available device, accessibility to medical services can be improved, leading to successful early diagnosis and management of cognitive dysfunction.

This study shows the utility of using a smartphone for cognitive testing. However, this study has several limitations. First, the primary targets for cognitive assessment using the mobile teleconference tool in the clinical field are community-dwelling poststroke patients and people who are at high risk of senile cognitive dysfunction; however, this study evaluated only inpatients. Second, patients with severe cognitive impairment were excluded, and only a small proportion of the subjects scored below 18 points.²⁵ Finally, we only used a single specific mobile teleconferencing application, FaceTime^®; however, the current worldwide market share of Apple's iOS is 13.9%, whereas that of Android is 82.8%.²⁶ We believe that for this purpose, other applications can provide equivalent audio–visual quality provided that the network environment is adequate and results equivalent to those found in this study would be expected.

Conclusions

We found good agreement between remote assessment using smartphone and face-to-face assessment, suggesting that the smartphone can be a useful tool for assessing cognitive function through MMSE-K in poststroke patients. Based on our results, we expect that mobile remote assessment could be implemented in centers where expensive equipment is not available. Further study should focus on testing different smartphone systems and including specific patient groups (e.g., with severe cognitive impairment or aphasia).

Authors' Contribution

Research conception and design: H.Y.P., S.S.J., J.Y.L., and J.H.P. Data acquisition: J.H.P and S.S.J. Data analysis and interpretation: J.H.P, S.S.J., A.R.C., and J.H.P. Manuscript preparation: all authors. Approval of final manuscript: all authors.

Footnotes

Disclosure Statement

No competing financial interests exist.

References

Desmond

, Moroney

, Sano

, Stern

. Incidence of dementia after ischemic stroke: Results of a longitudinal study. Stroke, 2002; 33:2254–2260.

Greenberg

, Treger

, Ring

. Post-stroke follow-up in a rehabilitation center outpatient clinic. Isr Med Assoc J, 2004; 6:603–606.

Shin

. [Determinants of unmet healthcare needs among the community dwelling Korean adults with disabilities: A case of the National Health Insurance beneficiaries.]. Health Soc Welf Rev, 2013; 33:549–577. [Article in Korean]

Poon

, Hui

, Dai

, Kwok

, Woo

. Cognitive intervention for community-dwelling older persons with memory problems: Telemedicine versus face-to-face treatment. Int J Geriatr Psychiatry, 2005; 20:285–286.

Mattioli

, Klutke

, Baruffaldi

, Villar-Guzman

, Toni

, Englmeier

. Technical validation of low-cost videoconferencing systems applied in orthopaedic teleconsulting services. Comput Methods Programs Biomed, 1999; 60:143–152.

Ball

, Scott

, McLaren

, Watson

. Preliminary evaluation of a Low-Cost VideoConferencing (LCVC) system for remote cognitive testing of adult psychiatric patients. Br J Clin Psychol, 1993; 32(Pt 3):303–307.

Ciemins

, Holloway

, Coon

, McClosky-Armstrong

, Min

. Telemedicine and the mini-mental state examination: Assessment from a distance. Telemed J E Health, 2009; 15:476–478.

Cullum

, Weiner

, Gehrmann

, Hynan

. Feasibility of telecognitive assessment in dementia. Assessment, 2006; 13:385–390.

Loh

, Donaldson

, Flicker

, Maher

, Goldswain

. Development of a telemedicine protocol for the diagnosis of Alzheimer's disease. J Telemed Telecare, 2007; 13:90–94.

10.

Hubble

, Pahwa

, Michalek

, Thomas

, Koller

. Interactive video conferencing: A means of providing interim care to Parkinson's disease patients. Mov Disord, 1993; 8:380–382.

11.

Loh

. Teledementia. In: Jack

, Bart

, eds. Teleneurology in practice, A comprehensive clinical guide. New York: Springer. 2015:147–157.

12.

Rubin

, Wellik

, Channer

, Demaerschalk

. Role of telemedicine in providing tertiary neurological care. Curr Treat Options Neurol, 2013; 15:567–582.

13.

Rubin

, Wellik

, Channer

, Demaerschalk

. Systematic review of teleneurology: Methodology. Front Neurol, 2012; 3:156.

14.

Rubin

, Wellik

, Channer

, Demaerschalk

. Systematic review of telestroke for post-stroke care and rehabilitation. Curr Atheroscler Rep, 2013; 15:343.

15.

Wechsler

, Tsao

, Levine

, Swain-Eng

, Adams

, Demaerschalk

, et al. Teleneurology applications: Report of the Telemedicine Work Group of the American Academy of Neurology. Neurology, 2013; 80:670–676.

16.

Bilgi

, Erol

, Gullu

, Sezgin

, Hamad

, Bilgel

, et al. Teleconsultation of coronary angiograms using smartphones and an audio/video conferencing application. Technol Health Care, 2013; 21:407–414.

17.

Matera

, Boonyasirikool

, Saggini

, Pozzi

, Pegoli

. The new smartphone application for wrist rehabilitation. J Hand Surg Asian Pac Vol, 2016; 21:2–7.

18.

Waki

, Fujiu

, Hayashi

, Kimura

, Kobayashi

, Nangaku

, et al. DialBetics: Smartphone-based self-management for type 2 diabetes patients on insulin injections. J Diabetes Sci Technol, 2016; 10:804–805.

19.

Miyashita

, Iketani

, Nagamine

, Goto

. FaceTime® for teaching ultrasound-guided anesthetic procedures in remote place. J Clin Monit Comput, 2014; 28:211–215.

20.

Bour

, Rasquin

, Boreas

, Limburg

, Verhey

. How predictive is the MMSE for cognitive performance after stroke?. J Neurol, 2010; 257:630–637.

21.

Nys

, van Zandvoort

, de Kort

, Jansen

, Kappelle

, de Haan

. Restrictions of the Mini-Mental State Examination in acute stroke. Arch Clin Neuropsychol, 2005; 20:623–629.

22.

Kwon

, Park

. Korean version of Mini-Mental State Examination (MMSE-K). Part I: Development of the test for the elderly. J Korean Neuropsychiatr Assoc, 1989; 28:125–135.

23.

Kim

, Jhoo

, Jang

. The effect of telemedicine on cognitive decline in patients with dementia. J Telemed Telecare, 2017; 23:149–154.

24.

Timpano

, Pirrotta

, Bonanno

, Marino

, Marra

, Bramanti

, et al. Videoconference-based mini mental state examination: A validation study. Telemed J E Health, 2013; 19:931–937.

25.

Tombaugh

, McIntyre

. The mini-mental state examination: A comprehensive review. J Am Geriatr Soc, 1992; 40:922–935.

26.

International Data Corporation.. Smartphone OS market share,. 2015. Available at www.idc.com/prodserv/smartphone-os-market-share.jsp (last accessed December 3, 2015 ).