Validation of a Smartphone Application Measuring Motor Function in Parkinson’s Disease

Abstract

Background: Measurement of motor function is critical to the assessment and management of Parkinson’s disease. Ambulatory motor assessment has the potential to provide a glimpse of the patient’s clinical state beyond the consultation. We custom-designed a smartphone application that quantitatively measures hand dexterity and hypothesized that this can give an indication of a patient’s overall motor function.

Objective: The aims of this study were to (i) validate this smartphone application against MDS-UPDRS motor assessment (MDS-UPDRS-III) and the two-target tapping test; (ii) generate a prediction model for MDS-UPDRS-III; (iii) assess repeatability of our smartphone application and (iv) examine compliance and user-satisfaction of this application.

Methods: 103 patients with Parkinson’s disease were recruited from two movement disorders clinics. After initial assessment, a group of patients underwent repeat assessment within two weeks. Patients were invited to use the smartphone application at home over three days, followed by a survey to assess their experience.

Results: Significant correlation between key smartphone application test parameters and MDS-UPDRS-III (r = 0.281–0.608, p < 0.0001) was demonstrated. A prediction model based on these parameters accounted for 52.3% of variation in MDS-UPDRS-III (R² = 0.523, F(4,93) = 25.48, p < 0.0001). Forty-eight patients underwent repeat assessment under identical clinical conditions. Repeatability of key smartphone application tests parameters and predicted MDS-UPDRS-III was moderate to strong (intraclass correlation coefficient 0.584–0.763, p < 0.0001). The follow-up survey identified that our patients were very comfortable with the smartphone application and mobile technology.

Conclusions: Our smartphone application demonstrated satisfactory repeatability and validity when measured against MDS-UPDRS-III. Its performance is acceptable considering our smartphone application measures hand dexterity only.

Keywords

Parkinson’s disease bradykinesia measurement motor validation

INTRODUCTION

Measurement of motor function is integral to the clinical assessment of Parkinson’s disease. It heavily influences management and is an important end-point in clinical trials. Developed in 1987, the Unified Parkinson’s Disease Rating Scale (UPDRS) is a standard instrument used for measuring a patient’s daily and motor functions [1]. This was revised in 2007 to the Movement Disorders Society-sponsored revision of the UPDRS (MDS-UPDRS) [2]. While the UPDRS motor examination (UPDRS-III) demonstrated strong inter- and intra-rater reliability [1 , 4], it is limited by the need of a trained assessor [5].

Limited medical resources and unprecedented growth in the number of patients with Parkinson’s disease especially in developing countries [6], is paving the way for innovative methods to assess and monitor patients. This resulted in the development of telemedicine and novel remote monitoring devices [7]. Even as far back as the early 1990 s, remote assessment of motor function had already been proven as a valid approach [8]. A tool that is easy-to-use, quick to perform, economical and accurately measures motor function without the need of a trained assessor is valuable in both clinical practice and research. Attempts at developing such devices saw the creation of continuous electromyography [9], various computer- and mobile device-based motor assessments [10 –12] and wearable automated devices [7 , 14]. In the past decade, increased coverage of broadband Internet and boom in mobile technology, both in sophistication and number, provide the ideal platforms to evolve the field of ambulatory motor measurement. By 2017, there will be more than three billion smartphones and mobile devices, covering nearly half the world’s population [7]. In developing countries where mobile phone ownership rate is high and conventional health care infrastructure is poorly developed, mobile devices will become increasingly important in health care [7].

We have designed a user-friendly smartphone application that quantitatively measures hand dexterity. We postulate that measurement of hand dexterity alone can give an indication of a patient’s overall motor function. The aims of this study were:

To demonstrate construct and criterion validity of the smartphone application against MDS-UPDRS-III and the two-target tapping test [15 –17];

To generate and assess the performance of a prediction model of MDS-UPDRS-III based on key smartphone application test parameters;

To assess the repeatability of smartphone application tests and

To examine the practicality, compliance and user satisfaction of our smartphone application and mobile technology in clinical practice.

Materials and methods

Overview

This study was approved by the Alfred and Royal Melbourne Hospitals’ Human Research and Ethics Committees. All eligible patients provided informed consent.

Smartphone application

The smartphone application was custom-designed for quantitative measurement of hand dexterity and attempts to strike a balance between user-friendliness and clinical usefulness. It consists of four tests (Fig. 1A–D):

Timed Tapping Test (TTT): This is based on the two-target tapping test [15 –17]. The subject alternately touches the center of two targets as many times as possible within ten seconds. A cumulative score is calculated based on the accuracy of each tap with reference to the center of the target.

Rapid Alternating Movements (RAM): This is based on item 3.6 (pronation-supination movements of hands) of MDS-UPDRS-III [2]. The subject holds and rotates the smartphone as many times as possible within ten seconds and the number of rotations per second (Hertz) is calculated.

Tremor Tracker: The subject traces a finger as fast and as accurately as possible between two parallel lines. This is repeated five times, with the distance between the two lines successively reduced. The accuracy and time taken are measured. Tremor Tracker endeavors to capture the presence of kinetic tremor as rest tremor is difficult to measure with the hardware used.

Cognitive Interference Test (CIT): This is based on the Stroop test [18], useful in assessing cognition in Parkinsonian disorders [19] and suitable for the context of a smartphone application. The subject is presented with a five by four grid of randomly assigned words (“red”, “blue” and “green” in equal proportions). Grids are presented in grey then color (random proportional mix of red, blue and green) in three separate sets. The subject is asked to select the word “red” in set 1, “blue” in set 2 and “green” in set 3. The accuracy and time taken are measured. (Validation of CIT will be covered in another article.)

Each hand is tested separately in all tests except for CIT, for which the subjects use their preferred hand. Auditory cue is incorporated in TTT and RAM to indicate start and end of testing.

Subjects and data collection

Consecutive patients from two movement disorders clinics were screened. All subjects fulfilled the United Kingdom Parkinson’s Disease Society Brain Bank diagnostic criteria [20]. Patients who were not on pharmacotherapy were excluded. Patients with severe Parkinson’s disease (Hoehn and Yahr stages 4 and 5) and physical or cognitive impairment deemed by the clinician to affect their ability to use a smartphone were also excluded.

Procedure

Baseline characteristics were documented. Levodopa equivalent daily dose was calculated [21]. Assessments included MDS-UPDRS-III, the two-target tapping test [15 –17], Montreal Cognitive Assessment (MOCA) [22] and Victoria Stroop Test (VST) [23]. All clinical assessments were administered (by Author WL) in the “on” motor state as judged by the patients. A standard demonstration of the smartphone application was presented to all participants, who were instructed to complete each test as quickly as they could. To avoid learning effect, subjects were allowed to practice, completing all tests twice. Data was collected on the third occasion. A subgroup of patients underwent repeat assessment one to two weeks after their initial visit. Time from last levodopa dose at repeat assessment was mirrored as close as practical to that of the initial assessment. To assess compliance and user satisfaction, patients were also invited to use the smartphone application at home over three days, taking measurements three times daily in both “on” and “off” motor states. At the end of the test period, a survey about their experience was completed.

Statistical analysis

All statistical analysis was performed using IBM SPSS version 22 (IBM Corporation, Armonk, New York, 2013).

Background characteristics of patients were analysed by descriptive statistics. To assess criterion and construct validity of our smartphone application, Pearson’s or Spearman’s correlation between test parameters, MDS-UPDRS-III and other assessments were calculated as appropriate. In addition, patients were categorized according to MDS-UPDRS-III severity (each category representing an increment of 5 points in MDS-UPDRS-III total score) and their results of key smartphone application test parameters were compared by Pearson Chi Square.

Repeatability of the smartphone application was assessed by Bland-Altman plot (≤5% of differences lying outside the repeatability coefficient is considered acceptable [24]) and intraclass correlation coefficient (ICC).

To enhance the practicality of the smartphone application, reliable change [25] of key test parameters was calculated. The reliable change criterion was set at 80%. This cutoff was calibrated on the basis of published moderate clinically important difference of UPDRS-III (5 points) [26]. (The clinically important difference for MDS-UPDRS-III is not available in the literature.) Moderate rather than minimal clinically important difference was chosen in order to reduce the likelihood of overestimating the significance of a change in smartphone application test result. Based on this cutoff, the likelihood of measurement unreliability alone accounting for a change in result in the same subject is less than 20%. For practical purposes, a change in result greater than the calculated reliable change will be considered clinically significant.

The prediction model for MDS-UPDRS-III total score was generated through a two-step process. Firstly for each of the four smartphone application tests, the ten parameters bearing the strongest correlation to MDS-UPDRS-III total score were put through multiple linear regression. The least significant parameters were eliminated sequentially until only statistically significant parameters remained. Secondly, IBM SPSS implementation of bootstrap multiple regression, known as automatic linear modeling (ALM), was performed on all test parameters [27]. One thousand regressions based on the Akaike Information Criterion, corrected for smaller sample (AICc), were performed. The AICc is a measure of both model performance (e.g., amount of variation in outcome variable accounted for by the model) and model complexity (e.g. number of variables included in the model). Automated regression methods may capitalise heavily on chance relationships in the data, therefore the resulting models may not generalise beyond the dataset employed to create them [28]. To increase generality, bootstrapped regression is a highly iterative procedure, in which random ‘bootstrap’ samples of the same size as the original are drawn, with replacement, from the original data. Thus, the first bootstrap sample might consist of four copies of the first observation, one copy of the second observation, no copies of the third observation and so on [28]. Bootstrap regression studies generally recommend choosing those variables that have been selected (e.g. on the basis of the AICc described above) in at least 50% of the bootstrap regressions [29, 30]. This criterion was found to be too liberal in the present study however, as most of the variables would then be selected. It was therefore decided to take the top 12 variables in terms of the number of times they were selected across the 1000 iterations. From the pool of variables identified through multiple linear regression and automatic linear modeling, one item from each test that was felt to best reflect the patient’s motor state was included in the final prediction model.

Information on compliance and user satisfaction of the smartphone application generated from the final survey was analysed by descriptive statistics.

RESULTS

One hundred and three patients were recruited over an eighteen-month period. All patients were taking levodopa and demonstrated motor fluctuation. Baseline characteristics as well as performance on key smartphone application test parameters are summarised in Table 1.

Validation of smartphone application

Over 150 test parameters are generated each time the smartphone application is completed. All test parameters were correlated against MDS-UPDRS-III total and sub scores and the two-target tapping test. Overall, the smartphone application demonstrated convergent quality to MDS-UPDRS-III and the two-target tapping test. Moderate (r = 0.3–0.49) to strong (r = 0.5–1.0) [31] correlation was seen between key smartphone application test parameters and the two-target tapping test (r = 0.339–0.729, p < 0.007). The magnitude of correlation was comparable and in some cases, superior to the correlation seen between MDS-UPDRS-III total score and the two-target tapping test (r = 0.523, p < 0.0001). Correlation of key smartphone application test parameters and MDS-UPDRS-III total score were small (r = 0.1–0.29) [31] to moderate (r = 0.281–0.608, p < 0.0001) (Fig. 2). The smartphone application test parameters that bear the strongest correlation to MDS-UPDRS-III and the two-target tapping test are presented in Table 2. When patients were separated according to MDS-UPDRS-III severity, the results for Tremor tracker–Time, left hand, Part 4 was significantly different across severity categories (p < 0.0001), while TTT–Mean total score approached statistical significance (p = 0.057).

Repeatability of smartphone application

To test repeatability of the smartphone application, 48 patients underwent repeat assessment under identical clinical conditions (mean time from last levodopa dose–112 minutes on both occasions). The mean time to repeat assessment was eight days (range 4–22 days). On Bland-Altman plot (Fig. 3) and ICC (Table 3), TTT–Mean total score demonstrated comparably strong repeatability as the two-target tapping test and MDS-UPDRS-III, with 98% of differences lying within the repeatability coefficient and ICC of 0.763 (p < 0.0001). Tremor tracker–Time, left hand, part 4; RAM–Mean frequency of rotation; CIT–Time, set 1, color and predicted MDS-UPDRS-III demonstrated moderate repeatability, with 92–96% of differences lying within the repeatability coefficient and ICC of 0.584–0.710 (p < 0.0001). Reliable changes in smartphone application key test parameters are presented in Table 4.

Prediction model for MDS-UPDRS-III total score

Items generated from automated linear modeling and multiple linear regressions were pooled (supplementary 1a and 1b). From TTT and RAM items, Mean total score and frequency of rotation were logically selected over other more specific parameters. From tremor tracker items, Time, Left hand, Part 4 was selected. The other Tremor Tracker items measure relationship of the subject’s finger movement to the two parallel lines on screen, e.g. distance from margin and number of times margin crossed. Time, Left hand, Part 4 was selected in preference as time was felt to be a better reflection of hand dexterity. From CIT, Time, Set 1, Color was selected. Of all the CIT items identified, most were related to Color sets. Again, time to task completion was felt to be a better indicator of hand dexterity, hence Time, Set 1, Color was selected over items examining other aspects e.g. variance of time and time between selection. In addition, both Tremor Tracker-Time, Left hand, Part 4 and CIT-Time, Set 1, Color consistently appeared in a number of other test models generated for MDS-UPDRS-III bradykinesia and tremor sub-scores (not included in this article). A prediction model including these four items accounted for 52.3% of variation in MDS-UPDRS-III total score (R² = 0.523, F(4,93) = 25.48, p < 0.0001).

Qualitative assessment of smartphone application

(Table 5) To assess compliance and user satisfaction of the smartphone application, 95 patients participated in home testing. Ninety-one patients (96%) returned with complete data and undertook the follow-up survey. Four patients dropped out due to difficulties using the smartphone. Amongst the patients who completed home testing, mobile device and computer ownership rate was 54% (n = 49) and 89% (n = 81) respectively. Of the patients with Internet access (92%, n = 84), 71% (n = 60) used the Internet daily. Common purposes of Internet usage were emails (82%), news and information (52%), online shopping (43%) and finance management (37%). Close to 40% access social media and 57% have experience making video calls. The majority of patients (83%) reported being at least fairly comfortable with mobile technology. Comfort level with mobile devices increased in 36%, including 11/13 patients who reported feeling uncomfortable with the technology at study initiation. However, 40% (n = 36) of patients did experience difficulties with the smartphone application, with the most commonly cited reasons being poor hand dexterity (50%, 18/36), fear of new technology (33%, 12/36) and reduced clarity of thoughts (31%, 11/36). Cost was perceived as a potential limitation for future mobile device usage in 8%. Over 90% found the smartphone application useful and almost all participants (97%) were interested in taking part in similar future developments. Of those who did not own mobile devices, 38% reported that they would consider purchasing one after their study experience.

DISCUSSION

MDS-UPDRS-III is the “gold standard” measurement of motor function that is broadly adopted in both clinical and research settings worldwide. We demonstrated that through four simple smartphone-based tests measuring hand dexterity, a patient’s motor function could be gauged. Repeatability and validity of key smartphone application test parameters against MDS-UPDRS-III total score and the two-target tapping test were satisfactory. A prediction model generated from these test parameters performed well and accounted for over 50% of variation in MDS-UPDRS-III total score. Good consistency of measurement was demonstrated across time. In addition, reliable change for key test parameters was calculated, thereby enabling our smartphone application to be used for monitoring purposes. Our patients were very familiar and comfortable with mobile technology and routinely use the Internet, reinforcing the feasibility of incorporating such technologies into clinical care and research.

In this study, we validated our smartphone application against MDS-UPDRS-III. Although it may seem logical to assess the performance of our smartphone application against other ambulatory devices, which may be more relevant “gold standards” for our tests, this will not necessarily be as appropriate given their limited usage in clinical practice. Our aim was to gauge a patient’s motor state through the smartphone application. Therefore in order to establish its uniqueness as a simple tool that can be widely adopted, assessing the performance of our smartphone application against MDS-UPDRS-III, a commonly used clinical tool, is more relevant. In addition, in light of the novelty of our smartphone application, there is a lack of well accepted “criterion measurement” that it can be validated against. Of the key test parameters, TTT-Mean total score performed best overall, demonstrating the strongest correlation with MDS-UPDRS-III and approached statistical significance when compared across severity categories of MDS-UPDRS-III. Furthermore, it has the best repeatability out of the four tests. An interesting observation was how CIT correlated with MDS-UPDRS-III total and bradykinesia scores despite being designed predominantly as a cognitive test. The design of CIT is heavily dependent on hand dexterity. We speculate that this is the main reason for its correlation with motor function. Furthermore, this observation probably also reflects the natural course Parkinson’s disease, whereby increasing disease severity is associated with more cognitive problems [32]. CIT was also observed to correlate stronger with motor parameters than other cognitive tests. This is also likely due to the confounding factor of heavy reliance on hand dexterity, which is not an element in MOCA andVST.

While our smartphone application may lack the complexity of measurement methodology compared to other devices, the prediction model generated still accounted for over 50% of variation in the MDS-UPDRS-III total score. It must be emphasized that our smartphone application was not designed to be a diagnostic tool nor was it intended to provide precise measurement of a patient’s motor function. Hence if the intention is to replace MDS-UPDRS-III, then this prediction model is perhaps not robust enough. However, this is hardly surprising given that upper limb bradykinesia and tremor items only account for about one third of the MDS-UPDRS-III [2]. Alternatively, if the intention were to generate a surrogate marker of motor function, then we would argue that the moderate strength of R² supports our hypothesis. Despite the limited representation of upper limb bradykinesia in the MDS-UPDRS-III, correlation of key smartphone application test parameters with MDS-UPDRS-III total score was slightly better than correlation with MDS-UPDRS-III bradykinesia sub-scores. Moreover, correlation of key test parameters with MDS-UPDRS-III upper limb or total bradykinesia sub-scores did not differ greatly. These observations, together with the moderate correlation results of our smartphone application and R² of 0.523 achieved by the prediction model, support our hypothesis. Hand dexterity measurement does give an indication of a patient’s overall motor function beyond its weighting in the MDS-UPDRS-III or measurement of upper limb bradykinesia alone. A recent pilot study comparing office versus web-based MDS-UPDRS-III yielded a mean ICC and correlation coefficient of 0.63 [33], similar to our results. Considering web-based motor assessment is common practice nowadays, the comparable performance of our smartphone application justifies its place in clinical practice. The value of our smartphone application lies in its ability to frequently and longitudinally capture data on a patient’s motor state to provide insights that cannot be acquired in the clinic.

A high level of compliance and user satisfaction was found with our smartphone application. While our study has an inherent selection bias towards patients with less advanced disease and cognitive impairment, the mean age of our patients is comparable to other Parkinson’s disease studies. Our patients demonstrated a high level of familiarity with computer technologies and this is likely translatable to other patients with similar demographics. The traditional model of face-to-face consultation will need to change to cope with increasing demand. Integrating mobile technology into clinical care and research will only enhance efficiency, accessibility and overall quality of care [34]. The feasibility of this approach is supported by the relatively high proportion of patients who own mobile devices (54%), have experience in making video calls (60%) and access social media (40%).

Quantitative assessment of Parkinson’s disease motor symptoms was historically performed under laboratory conditions, utilising optoelectronic, magnetic or ultrasound techniques [35]. The development of motion sensors enabled ambulatory measurement of motor symptoms. Previous approaches adopted complex algorithms to generate continuous and precise recordings of the patient’s motor states [7 , 35]. However, the requirement for specialized and expensive equipment meant widespread use is not feasible. Mobile technology provides the ideal platform to overcome such shortcomings with sophisticated built-in motion sensors that are well suited to motor measurement in Parkinson’s disease [35]. They provide ready-made hardware with Internet connectivity and once software is developed, the purchasing cost to the patient is generally small. In addition, the worldwide smartphone ownership rate is growing, with good penetrance even in developing countries [7]. The only prerequisite for our smartphone application is smartphone ownership. The overriding advantages that follow are the minimal training required for its use and the likely insignificant cost to the patient. Extensive and detailed information of the patient’s motor state such as the degree of bradykinesia and dyskinesia, tremor frequency, falls and gait freezing can now be measured using various devices [7 , 35]. Ambulatory measurement can generate an enormous amount of data and it is not known whether this actually improves patient outcome. With education and simple training, patients can reliably recognize and report their motor symptoms with good agreement to the clinician’s assessment [36]. Therefore, restraint must be in place to only selectively collect information that compliments rather than duplicates what can be achieved through good history taking and patient education.

There are several limitations to this study. A relatively selected group of patients were recruited. Those with severe disease (Stages 4 and 5) were excluded as it was felt that the associated physical and cognitive impairment would confound the patients’ ability to use a smartphone adequately. As such, extensibility of our results to the wider population of Parkinson’s disease patients may need additional investigation. This can be addressed by further validation in a larger, unselected cohort of patients and against other ambulatory measurement devices. Although the design of our smartphone application tests was intended to be simple, accuracy of measurement nonetheless still depends on the patient’s understanding, motivation and active participation. In addition, our application assesses hand dexterity and not other aspects of motor function such as gait. One of the main difficulties reported by patients was that of hand clumsiness in the setting of a small screen, particularly with CIT. This problem will likely be resolved by future generations of improved hardware.

Ambulatory quantitative measurement of motor function obviates the need of a trained assessor, avoids subjectivity of clinical rating scales and facilitates out-of-office assessment. Mobile technology provides the ideal platform to further develop this field due to its sophistication, user-friendliness and pervasiveness. Harnessing this technology, our smartphone application demonstrates repeatability and validity that are at least comparable to web-based assessment of motor function. Bearing in mind that our smartphone application measures hand dexterity predominantly, its performance is acceptable and provides a reasonable indication of a patient’s motor function. It has the potential to evolve into a useful tool in both clinical and research settings.

ACKNOWLEDGMENTS

The authors thank Dr Dean McKenzie and Ms Catherine Smith from the Department of Epidemiology and Preventative Medicine, Monash University, for their assistance with statistical analysis of data.

CONFLICT OF INTEREST

The authors have no conflict of interest to report.

FINANCIAL DISCLOSURES

Will Lee

Employment: Alfred Health

Grants: Movement Disorders Society of Australia Fellowship

Andrew Evans

Stock Ownership in medically-related fields: CSL and GKC

Intellectual Property Rights: Parkinson’s Disease Assessment App

Consultancies: GKC

Advisory Boards: GKC, Abbvie, UCB

Employment: the Royal Melbourne Hospital

Honoraria: UCB, Abbvie, GKC

David Williams

Intellectual Property Rights: Parkinson’s Disease Assessment App

Consultancies: Abbvie, Ipsen, Allergan

Advisory Boards: Abbvie, Ipsen, Allergan

Grants: Abbvie

Footnotes

Appendix

References

Siderowf

, McDermott

, Kieburtz

, Blindauer

, Plumb

, Shoulson

, & Parkinson Study G (2002) Test-retest reliability of the unified Parkinson’s disease rating scale in patients with early Parkinson’s disease: Results from a multicenter clinical trial. Mov Disord, 17, 758–763.

Goetz

, Fahn

, Martinez-Martin

, Poewe

, Sampaio

, Stebbins

, Stern

, Tilley

, Dodel

, Dubois

, Holloway

, Jankovic

, Kulisevsky

, Lang

, Lees

, Leurgans

, LeWitt

, Nyenhuis

, Olanow

, Rascol

, Schrag

, Teresi

, Van Hilten

, & LaPelle

(2007) Movement Disorder Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS): Process, format, and clinimetric testing plan. Mov Disord, 22, 41–47.

Metman

, Myre

, Verwey

, Hassin-Baer

, Arzbaecher

, Sierens

, & Bakay

(2004) Test-retest reliability of UPDRS-III, dyskinesia scales, and timed motor tests in patients with advanced Parkinson’s disease: An argument against multiple baseline assessments. Mov Disord, 19, 1079–1084.

Richards

, Marder

, Cote

, & Mayeux

(1994) Interrater reliability of the Unified Parkinson’s Disease Rating Scale motor examination. Mov Disord, 9, 89–91.

Goetz

, & Stebbins

(2004) Assuring interrater reliability for the UPDRS motor section: Utility of the UPDRS teaching tape. Mov Disord, 19, 1453–1456.

Dorsey

, Constantinescu

, Thompson

, Biglan

, Holloway

, Kieburtz

, Marshall

, Ravina

, Schifitto

, Siderowf

, & Tanner

(2007) Projected number of people with Parkinson disease in the most populous nations, 2005 through 2030. Neurology, 68, 384–386.

Achey

, Aldred

, Aljehani

, Bloem

, Biglan

, Chan

, Cubo

, Dorsey

, Goetz

, Guttman

, Hassan

, Khandhar

, Mari

, Spindler

, Tanner

, van den Haak

, Walker

, Wilkinson

, International

, & Movement Disorder Society Telemedicine Task F (2014) The past, present, and future of telemedicine for Parkinson’s disease. Mov Disord, 29, 871–883.

Hubble

, Pahwa

, Michalek

, Thomas

, & Koller

(1993) Interactive video conferencing: A means of providing interim care to Parkinson’s disease patients. Mov Disord, 8, 380–382.

Askari

, Zhang

, & Won

(2010) An EMG-based system for continuous monitoring of clinical efficacy of Parkinson’s disease treatments. Conf Proc IEEE Eng Med Biol Soc, 2010, 98–101.

10.

Fazekas

, Voros

, Keresztenyi

, Kozmann

, & Laczko

(2002) Computer aided interactive remote diagnosis of Parkinsonians. Stud Health Technol Inform, 90, 572–576.

11.

Lemoyne

, Mastroianni

, Cozza

, Coroian

, & Grundfest

(2010) Implementation of an iPhone as a wireless accelerometer for quantifying gait characteristics. Conf Proc IEEE Eng Med Biol Soc, 2010, 3847–3851.

12.

Lemoyne

, Mastroianni

, Cozza

, Coroian

, & Grundfest

(2010) Implementation of an iPhone for characterizing Parkinson’s disease tremor through a wireless accelerometer application. Conf Proc IEEE Eng Med Biol Soc, 2010, 4954–4958.

13.

Griffiths

, Kotschet

, Arfon

, Xu

, Johnson

, Drago

, Evans

, Kempster

, Raghav

, & Horne

(2012) Automated assessment of bradykinesia and dyskinesia in Parkinson’s disease. J Parkinsons Dis, 2, 47–55.

14.

Yang

, Hsu

, Shih

, & Lu

(2011) Real-time gait cycle parameter recognition using a wearable accelerometry system. Sensors (Basel), 11, 7314–7326.

15.

Hogl

, Peralta

, Wetter

, Gershanik

, & Trenkwalder

(2001) Effect of sleep deprivation on motor performance in patients with Parkinson’s disease. Mov Disord, 16, 616–621.

16.

Nutt

, Gancher

, & Woodward

(1988) Does an inhibitory action of levodopa contribute to motor fluctuations?. Neurology, 38, 1553–1557.

17.

Defer

, Widner

, Marie

, Remy

, & Levivier

(1999) Core assessment program for surgical interventional therapies in Parkinson’s disease (CAPSIT-PD). Mov Disord, 14, 572–584.

18.

Jensen

, & Rohwer

Jr (1966) The Stroop color-word test: A review. Acta Psychol (Amst), 25, 36–93.

19.

Lee

, Williams

, & Storey

(2012) Cognitive testing in the diagnosis of parkinsonian disorders: A critical appraisal of the literature. Mov Disord, 27, 1243–1254.

20.

Hughes

, Daniel

, Kilford

, & Lees

(1992) Accuracy of clinical diagnosis of idiopathic Parkinson’s disease: A clinico-pathological study of 100 cases. J Neurol Neurosurg Psychiatry, 55, 181–184.

21.

Tomlinson

, Stowe

, Patel

, Rick

, Gray

, & Clarke

(2010) Systematic review of levodopa dose equivalency reporting in Parkinson’s disease. Mov Disord, 25, 2649–2653.

22.

Gill

, Freshman

, Blender

, & Ravina

(2008) The Montreal cognitive assessment as a screening tool for cognitive impairment in Parkinson’s disease. Mov Disord, 23, 1043–1046.

23.

Hanes

, Andrewes

, Smith

, & Pantelis

(1996) A brief assessment of executive control dysfunction: Discriminant validity and homogeneity of planning, set shift, and fluency measures. Arch Clin Neuropsychol, 11, 185–191.

24.

Bland

, & Altman

(1986) Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 1, 307–310.

25.

Evans

C MF

, & Barkham

(1998) The contribution of reliable and clinically significan change methods to evidence-based mental health. Evidence Based Mental Health, 1, 70–72.

26.

Shulman

, Gruber-Baldini

, Anderson

, Fishman

, Reich

, & Weiner

(2010) The clinically important difference on the unified Parkinson’s disease rating scale. Arch Neurol, 67, 64–70.

27.

Norusis

(2011) IBM SPSS Statistics 19 statistical procedures companion, Addison-Wesley, Boston, Massachusetts.

28.

Efron

, & Tibshirani

(1993) An introduction to the bootstrap, Chapman and Hall, New York.

29.

Austin

, & Tu

(2004) Bootstrap methods for developing predictive models. Am Stat, 58, 131–137.

30.

Billah

, Reid

, Shardey

, & Smith

(2010) A preoperative risk prediction model for 30-day mortality following cardiac surgery in an Australian cohort. European J Cardiothorac Surg, 37, 1086–1092.

31.

Cohen

(1988) In Statistical power analysis for the sciences.. Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 79–81.

32.

Hely

, Reid

, Adena

, Halliday

, & Morris

(2008) The Sydney multicenter study of Parkinson’s disease: The inevitability of dementia at 20 years. Mov Disord, 23, 837–844.

33.

Cubo

, Gabriel-Galan

, Martinez

, Alcubilla

, Yang

, Arconada

, & Perez

(2012) Comparison of office-based versus home Web-based clinical assessments for Parkinson’s disease. Mov Disord, 27, 308–311.

34.

Pasluosta

, Gassner

, Winkler

, Klucken

, & Eskofier

(2015) An emerging era in the management of Parkinson’s disease: Wearable technologies and the Internet of things. IEEE J Biomed Health Inform, 19, 1873–1881.

35.

Hobert

, Maetzler

, Aminian

, & Chiari

(2014) Technical and clinical view on ambulatory assessment in Parkinson’s disease. Acta Neurol Scand, 130, 139–147.

36.

Reimer

, Grabowski

, Lindvall

, & Hagell

(2004) Use and interpretation of on/off diaries in Parkinson’s disease. J Neurol Neurosurg Psychiatry, 75, 396–400.