Abstract
Summary
We compared the telemedicine assessment of 20 patients with multiple sclerosis (MS) with the findings of a hands-on examiner. The remote specialist was a neurologist with expertise in MS; the hands-on examination was performed by an experienced mid-level practitioner. We also compared the findings of a second specialist viewing the examination in the room with the patient. The videoconference link operated at a bandwidth of 384 kbit/s. All three examiners independently completed a standardized rating scale for neurological functions. Cronbach's α for the three raters' total expanded disability status scale (EDSS) score was 0.99 with individual correlations ranging from 0.96–0.97. Agreement between raters for individual neurological domain scores was more variable. The most consistent assessments were for optic, bowel and bladder, and cerebral functions. The least consistent were for cerebellar and brain stem functions. Agreement between the remote and local examiners was similar to that reported for different neurological examiners directly assessing the same patient using the EDSS rating system.
Introduction
There have been a number of studies assessing the feasibility of remote consultation in neurology. Some reports have been anecdotal, where remote consultation and examination were deemed to have contributed to patient management. 1–3 Other studies have compared remote and traditional hands-on examination with respect to diagnoses, 4,5 investigations undertaken, 3,5,6 and length of hospitalization and post-hospital course. 5 The general consensus has been that a neurologist can obtain information for diagnosis and management at a distance, especially when aided by a trained individual who performs the hands-on examination being transmitted over a video link.
A few studies have investigated specific aspects of the neurological examination. These have included the remote implementation of the National Institutes of Health Stroke Scale to inform decisions regarding the use of a recombinant tissue plasminogen activator (rtPA) to reduce morbidity following stroke 7–11 and the use of the Unified Parkinson's Disease Rating Scale for remote patient assessment. 12 All studies produced positive results. While the literature supports the use of telemedicine to bring expert neurological assessment to locations where it is not readily available, there has been no systematic study of the use of teleneurology for assessing patients with multiple sclerosis (MS).
We therefore compared the assessment made by a specialist viewing the neurological examination remotely with the findings of a hands-on examiner. To simulate conditions under which telemedicine would probably occur, the hands-on examination was performed by an experienced mid-level practitioner. We also compared the findings of a second specialist viewing the examination in the room with the patient.
Methods
Participants
Twenty patients with MS participated in the study (Table 1). Ten patients were seen at the Baltimore VA Medical Center and 10 at the Washington, DC VA Medical Center. All patients had a diagnosis of MS based on McDonald International Panel Criteria 13 and were actively followed. Patients were selected based on their willingness to participate and to represent a range of MS related disability. The study was approved by the appropriate ethics committees.
Participant characteristics (n = 20)
Procedure
Patients were brought into the examining room and introduced to the investigators in the room and those observing remotely. The first 10 patients were seen at the Washington, DC VA Medical Center. The hands-on examination was performed by a physician's assistant experienced in the neurological examination of MS patients. A neurologist with expertise in MS observed and guided the examination remotely. A second neurologist experienced in MS was also in the room observing the examination.
The second 10 patients were examined at the Baltimore VA Medical Center. The hands-on examination was performed by a master's level nurse practitioner also experienced in performing neurological examinations with MS patients. The neurologist in Washington, DC who previously acted as the local observing neurologist served as the remote examiner. The neurologist in Baltimore who had previously acted as the remote examiner served as the local observer in the room with the patient.
In all cases, the examination was directed by the remote MS specialist observing the neurological examination via a videoconferencing system. The mid-level practitioners (MLPs) performing the hands-on examination were limited with respect to the comments they were allowed to make while examining the patient. For example, the MLPs were allowed to say that they could overcome a muscle group easily or with effort, but were not allowed to indicate a specific rating.
At the completion of the examination, all three clinicians independently completed the Kappos Neurostatus rating system, 14 a scored neurological examination based on the Kurtzke EDSS 15 producing scores for eight neurological systems along with a total score. The EDSS rating system includes an evaluation of walking and visual acuity. A measured distance walk was accomplished prior to the remote examination and visual acuity was measured using a standard eye chart. Results from the distance walk and visual acuity assessment were available to all examiners. Since all examiners were provided with results from the measured distance walk, we did not include this dimension when comparing scores between examiners.
The specialist in the room with the patient was asked to respond to a single question regarding whether or not he felt he was able to obtain adequate information about the patient's neurological status through direct observation.
The specialist examining the patient remotely completed a questionnaire indicating his confidence in the final EDSS score along with his confidence when rating each of the seven separate neurological systems assessed, using a 10-point scale with higher scores indicating greater satisfaction or confidence.
Following the examination, patients completed a questionnaire indicating their comfort, satisfaction and confidence in the telemedicine based neurological examination.
Telemedicine equipment
Each site was equipped with a pan/tilt/zoom camera under the control of the neurologist assessing the patient remotely. All patients assessed in Baltimore were viewed in Washington, DC on a 122 cm plasma screen monitor. Patients assessed in Washington were viewed from one of two rooms in Baltimore on a 81 cm CRT monitor or a 127 cm plasma screen monitor. Connections were via three ISDN lines transmitting at 384 kbit/s.
Data analysis
Ratings from all three examiners were compared for all patients with respect to total EDSS score and for scores within each functional domain making up the total EDSS score. Reliability among examiners was computed using Cronbach's alpha 16 and at various EDSS levels using Bland-Altman plots. 17
The remote examiners were asked to indicate their confidence in their assessments of the total EDSS score and scores for each rated neurological system. Ratings were made on a 10-point scale, with 10 indicating the greatest degree of confidence.
Results
Cronbach's alpha for the total EDSS score was 0.99 and ranged from 0.83–0.99 for the neurological systems (Table 2). The average correlations (Cronbach's alpha) for all functional domains, excluding ambulation and total EDSS scores, were:
remote specialist and hands-on examiner = 0.93; remote specialist and local specialist = 0.91; local specialist and hands-on examiner = 0.98.
Cronbach's alpha for general reliability and individual correlations between examiners (observations on 20 patients)
There were strong correlations between all examiners (α coefficients between 0.96 and 0.99) with respect to the EDSS total score. However, there were variations depending on which domain was assessed and the location of the examiner. The most consistent ratings between examiners were for optic, pyramidal, bowel/bladder and cerebral functions. There were lower correlations for brain stem, cerebellar and sensory functions. Generally, correlations were higher between the hands-on examiner and the MS specialist in the room with the patient.
While correlations were high among all three examiners for the total EDSS score, some variation was noted when looking at a patient by patient comparison. Agreement for the local and remote specialist versus the hands-on examiner are shown in Figure 1 as Bland-Altman plots. In these plots the mean of the examiners' ratings is plotted against the difference between examiners. The reference line at zero denotes no difference between examiners and the limits of agreement (broken line) were set at +/−1 point. The limits of agreement were set according to the consensus among our clinicians that a 1-point difference or less was unlikely to cause a change in treatment strategy. Ratings were nearly identical for both examiners in the room with the patient (local specialist versus hands-on examiner). There was some variation in the EDSS score ranges between the remote and the hands on examiner (remote specialist versus hands-on examiner). However, both the local and remote specialists' ratings were at or within the limits of agreement suggesting good agreement between examiners.
Bland-Altman plot of the total EDSS scores (i.e. difference between the two raters' scores vs average of the two raters' scores). The limits of agreement were set at +/–1. The sizes of the dots are proportional to the number of cases. (a) comparison of scores from the local specialist [L] and the hands-on examiner [M]. (b) comparison of the scores from the remote specialist [R] and the hands-on examiner
The highest correlation among examiners (α = 0.99) for a specific neurological domain was for optic functions. The poorest correlation was for cerebellar functions (α = 0.83) (Figures 2 and 3). For optic functions, the scores were within the limits of agreement for both the local and remote specialist compared with the hands-on examiner. However, for cerebellar functions the remote specialist and the hands-on examiner exceeded the limits of agreement on four patients, whereas the local specialist and hands-on examiner exceeded limits on only one patient.
Bland-Altman plot of the optic function scores. (a) comparison of scores from the local specialist [L] and the hands-on examiner [M]. (b) comparison of the scores from the remote specialist [R] and the hands-on examiner
Bland-Altman plot of the cerebellar function scores. (a) comparison of scores from the local specialist [L] and the hands-on examiner [M]. (b) comparison of the scores from the remote specialist [R] and the hands-on examiner
Almost all of the local specialists (19 of 20 cases) said that they were able to obtain sufficient information to evaluate the patients. The one ‘no’ response pertained to a sensory examination.
On average, the remote examiners' confidence ratings were high for the rated neurological domains. Median ratings for the different domains ranged from 9.0 to 7.5. The maximum score for each scale was 10 with minimum scores ranging from 7.0 to 4.0.
In general, patients reported a positive experience and confidence in the telemedicine based neurological examination. However, not all patients were satisfied with all aspects of the examination. For example, to the statement ‘My needs could be met using telemedicine’, 16 patients strongly agreed or agreed, but five strongly disagreed.
Discussion
The results of the present investigation were mostly encouraging and support the use of teleneurology in MS. In general, ratings of neurological functions made by a remote specialist agreed well with those of a MLP performing a hands-on examination and with the assessment made by a specialist observing in the room with the patient. Correlations were higher between assessments made by the hands-on examiner and the specialist observing in the same room as the patient. Hence, there appeared to be some loss of information when the examination was performed remotely. Certain aspects of the neurological examination appeared easier to assess remotely than others, with the lowest correlations between the remote and local examiners occurring for brain stem, cerebellar and sensory functions.
The Kappos Neurostatus rating system for the Kurtzke EDSS 16 is quantifiable and widely used with MS patients. In the present telemedicine study, some aspects of this system were not tested: visual acuity was assessed using a standard eye chart and this information was made available to all examiners for the visual system score. Data regarding bowel and bladder functioning was primarily from patient report. Despite an assessment of walking during the examination, information regarding distance walked was obtained off camera.
Although there appeared to be some loss of information when the examination was observed by video link, the agreement between the remote and local examiners was similar to reports of inter-rater agreement when different neurological examiners directly assessed the same patient using the EDSS rating system. A number of studies have reviewed the reliability of the Kurtzke EDSS. 18–26 Overall inter-rater reliability for the total EDSS as measured by the kappa statistic ranged from 0.1923 to 0.95. 19 Much of the variability in agreement was related to the degree of disability, with less disabled MS patients producing more variation in scoring. Most studies recorded at least moderate inter-rater agreement when the total EDSS was above 4.0. 19−21,24–26 When the criterion for agreement was set at +/−1.0 point on the EDSS scale, as in the present study, agreement indices improved. 20–22,24–26 In general, when scoring the Kurtzke Functional Systems, higher inter-rater agreement occurs for pyramidal, visual, brain stem, and bowel and bladder systems and lower inter-rater agreement occurs for mental, sensory and cerebellar systems.
Intra-rater agreement on the EDSS has not been reported as frequently, but has generally been better than inter-rater agreement. Kappa statistics for intra-rater agreement have been 0.60 or better 19,24–26 and have improved when intra-rater agreement is defined as a difference of 1.0 point on the EDSS. In these previous studies, lower agreement levels were seen in sensory and cerebellar functional system scores.
The findings from our study employing telemedicine and a remote examiner were similar to those obtained with different examiners doing in-person examinations: visual, pyramidal and bowel/bladder functions were most consistently rated, while cerebellar and sensory functions showed the least consistency between examiners. With some exceptions, patients in the present study rated their telemedicine experience positively. For all patients, it was their first experience with telemedicine.
The findings from our study support the continued development of a telemedicine-based neurological examination for MS patients who live considerable distances from specialized care. The fact that discrepancies in ratings comparing remote and local examiners were similar to those among clinicians performing hands-on examinations is encouraging. However, the findings from our small pilot study are less supportive of the use of telemedicine for the initial diagnostic workup. Future studies are therefore required of diagnosis, treatment decision-making and the overall quality of care that results from telemedicine-based neurological examination in MS.
Footnotes
Acknowledgements
This study was sponsored by and carried out under the direction of the US Department of Veterans Affairs East Coast Multiple Sclerosis Center of Excellence.