Digital Health Technology to Measure Drug Efficacy in Clinical Trials for Parkinson’s Disease: A Regulatory Perspective

Digital health technology (DHT), that includes wearable and environmental sensors, video cameras and other electronic tools to evaluate disease remotely, has provided new opportunities for the measurement of movement and functionality. Accelerometers are present in our actigraphy gadgets, smart watches and smart phones. They can also be customized to provide a continuous 3-dimensional measurement of limb and trunk movement in patients that would not be observed during an examination [1]. Some commercial systems combine accelerometers, gyroscopes and magnetometers into an algorithm, known as inertial measurement units (IMUs) to analyse spatio-temporal parameters [2]. Video cameras, wearable systems, gloves, and other environmental sensors can capture activity and movement.

The impact of Parkinson’s disease (PD) on movement is profound and protean. Current standards for evaluating disease severity (e.g., Movement Disorder Society - Unified Parkinson’s Disease Rating Scale, MDS-UPDRS) rely on subjective reporting, and some disagreement has been shown between assessments made by investigators and those made by study subjects [3, 4]. Disease rating scales are also limited by the periodicity of the measurement and recall bias. DHTs now offer the potential for objective measurement of tremor, gait deficits, freezing of gait, postural instability, upper limb motion, leg agility, rigidity, and motor fluctuations. Besides movement, abnormalities in cadence, tonal variation and fluency of speech in PD patients can also be analysed by DHTs. Table 1 describes the spectrum of sensors that are being investigated to capture symptoms and other assessments of PD [5, 6]. In addition to these passive measurements, sensors have been used to challenge patient performance. Tapping tests on a cell phone have distinguished patients with PD from healthy controls while motion detectors can be used to perform timed up-and-go tests [7, 8].

There is a large spectrum of DHTs available for use in a clinical trial. Some DHTs meet the definition of a medical device under the Federal Food, Drug and Cosmetic Act while some DHTs do not [9, 10]. Generally, clearance or approval of the DHT for use in a clinical trial conducted under an Investigational New Drug (IND) application or an Investigational Device Exemption (IDE) [11, 12] is not a requirement unless the DHT will be marketed independently or as a combination product.

With seemingly limitless combinations of measurements, sensors, tests and algorithms, how are we to choose those that are most useful for drug development?

The first goal is to obtain measurements that have been demonstrated to be accurate and reliable over time and across patients, leading to solid scientific conclusions on drug efficacy. For the approval of drugs, the Food, Drug and Cosmetic Act requires that substantial evidence of effectiveness be provided that would allow experts to conclude that a drug would have the effect described in labelling [13]. To satisfy this regulation, DHTs should allow for a well-defined and reliable assessment of a patient’s response to treatment. Verification and validation are important to confirm the accuracy and precision of measurements. Analytical validation and clinical validation ensure the reliability of algorithms that translate accelerometry or other sensor readings into clinical observations (e.g., tremor, falls, steps) [14]. Current standards for measurement of drug effect rely largely on patient reported outcomes, neurological examinations and face-to face consultations and these are useful benchmarks against which to evaluate new measurements.

Not all the manifestations of PD are equally amenable to measurement by DHTs. Some aspects of functionality may be best assessed during in-person visits. On the other hand, for some manifestations, DHT measurements may outperform the assessments of observers. Experiments manipulating the intensity of deep-brain stimulation in PD patients have shown that some sensor measurements are more sensitive and less variable than human scoring on components of the UPDRS. Some disease features such as freezing of gait are unpredictable and difficult to assess during clinic visits. These have been successfully evaluated using DHT in laboratory conditions and DHT may allow for greater opportunities for detection when used to monitor patients at home [15].

Clinical validation of DHT should ensure that individuals with and without PD can be clearly distinguished. Depending on the characteristic being measured, specificity of measurements may be challenging. Tremor in PD varies in frequency and amplitude. A study by Hossen et al. showed that accelerometers failed to distinguish between the tremor of PD and essential tremor in 10%of subjects [16]. In a review of studies comparing accelerometers to video-recordings to capture freezing of gait, validity values ranged from 73 to 100%for sensitivity, and from 67 to 100%for specificity. The authors concluded that there is a lack of consistency in outcomes measured, methods of assessing validity, and reported results. Given these limitations, the validation of sensor-derived assessments of PD features would benefit from increased collaboration among researchers, aligning data collection protocols, and sharing data sets [17]. Many DHT measurements are in the early stages of research and warrant larger sample sizes with patients with varying stages of disease progression.

Shortfalls in the specificity of sensors suggest that using more than one modality of measurement may be an important strategy, just as current clinical scoring systems measure many facets of the disease. In a study using smartphones to measure five different tasks (voice test, posture test, gait test, finger tapping test and reaction time test) performed by 10 patients with PD and 10 healthy controls, the mean sensitivity of the smartphone measurements for detection of PD was 96.2%(SD 2%) and mean specificity was 96.9%(SD 1.9%) [18]. While multiple measurements add to the richness and specificity of the assessment, the statistical plan used to determine the outcome of a trial using multiple measurements must be prespecified to avoid the pitfalls of multiplicity and the increasing risk of false positive findings when many tests of efficacy are combined [19].

The ability to capture the impact of known effective treatments is another indication that the DHT will be useful in evaluating new treatments. Using wrist and ankle sensors, Pulliam et al. were able to quantify the effect of a dose of levodopa on tremor, bradykinesia and dyskinesia in 13 patients with PD. The measurements made by these sensors correlated with video-recording evaluations made by clinicians [20]. Investigators have reported the ability to distinguish on from off periods, which is another indication the drug effect is captured [21].

A second goal is to ensure that trial endpoints involving DHT measurements represent clinically meaningful responses to a drug; interpreted in FDA regulations as those with an impact on how patients feel, function or survive. The clinical benefit of some sensor readings is self-evident. Weiss et al. found that a 3-day sample of gait recordings using a 3-D accelerometer placed in the middle of the back served as a predictor of falls within the next year [22]. The clinical meaningfulness of other sensor measurements may be less obvious. For example, measurements of tremor may not reflect the functional impairments that patients find most disabling. Early engagement of patients is a cornerstone in determining the relevance of endpoints that involve functional measurements made by DHT [23]. “The Voice of the Patient” is part of FDA’s Patient-Focused Drug Development initiative to incorporate perspectives from patients, caretakers and other patient representatives on the most significant effects of PD on their daily lives and experiences with currently available therapies [24]. In addition to patients, engagement of a variety of stakeholders, including caregivers, disease experts and regulatory authorities would be necessary to determine the meaningfulness of certain measurements in a clinical trial.

Challenge tests are helpful to assess activities of daily living in patients with PD. Using a mobile app, Zhan et al. challenged individuals with and without PD to perform various tasks reflecting speech, dexterity, gait, balance, and reaction time and used machine-learning on these tasks to create a PD severity score. The authors aimed to provide a clinically meaningful assessment of patients in their real-world environments [25]. Extensive research has been conducted with machine learning to analyse and predict freezing of gait, tremors and falls. Machine learning algorithms provide new opportunities for long-term monitoring of a drug’s effectiveness as well as disease progression [26, 27].

Selection of the metrics best suited to disease evaluation presents another challenge. Just looking at gait characteristics using machine learning, Rehman et al. identified five different clinical characteristics (step velocity, mean step length, step length variability, mean step width, and step width variability) that classified PD [28]. Among the plethora of possible measurements, principle component analyses have been helpful to whittle down to those that account for most of the variance in the data [29].

Given the complexity of PD and the innumerable possible measurements that can be made, the challenge is to find those that best reflect meaningful responses to treatment and that can be used as a standard in clinical studies. What is an optimal sampling interval to obtain a stable estimate of function? Do we focus on average measurements or outlying measurements? Where do we position sensors, and how many do we need? Not all the pathological features will be captured even when multiple sensors are used, and drugs may also only affect some of these features. Different measurements may be needed for different stages of the disease, and for drugs with different mechanisms of action.

Finally, DHTs should be useable and safe for study participants. In general, large uncomfortable wearables, or DHTs that require fine-motor skills to use them and those that need technological know-how are unlikely to get the necessary cooperation from patients. DHTs need to be physically safe to use, electronically secure, and trustworthy when recording personally identifiable information. FDA regulations are designed to ensure the safety and welfare of subjects enrolled in clinical investigations, detailing requirements for safety reporting and Institutional Review Board supervision and allowing clinical holds when “human subjects are or would be exposed to an unreasonable and significant risk of illness or injury” [30].

From a regulatory perspective, adequate and well-controlled studies are the basis of determining whether there is “substantial evidence” to support the claims of effectiveness for new drugs [31]. The comparative structure of a clinical trial is important to be able to conclude that the measured effect can be attributed to the drug. Randomized and blinded trials showing superiority of the investigational treatment to control, inherently confirm that the sensor is detecting an effect. Such studies may involve parallel arm controls or crossover within individuals. Absent substantial experience with sensors, non-inferiority studies are likely to be difficult to interpret since the effects of comparator drug on the proposed sensor measurements may not be known. Consequently, we may not know whether both arms were effective or ineffective.

Besides their potential for scientific improvements in measurement, DHTs may be able to gather much of the needed study measurements from participants in their home environments, offering a new dimension of convenience for patients. Such decentralized clinical trials may make it much easier for patients with mobility challenges and other personal and practical obstacles in getting to study sites, to participate in clinical research.

Sensors cannot capture certain aspects of a face-to-face interview, a physical examination of patient’s balance or muscle tone. With increasing use of digital measurements in clinical trials, it will be important to ensure that we do not ignore these aspects of the disease. There are situations where sensors are more accurate and sensitive than human raters, and situations where human raters are more discerning and specific than sensors. Careful studies will be needed to demonstrate when digital measurements succeed or fall short of capturing meaningful benefits during drug development.