Validation of Freezing-of-Gait Monitoring Using Smartphone

Abstract

Background:

Freezing of gait (FOG) is a commonly observed motor symptom for patients with Parkinson's disease (PD). The symptoms of FOG include reduced step lengths or motor blocks, even with an evident intention of walking. FOG should be monitored carefully because it not only lowers the patient's quality of life, but also significantly increases the risk of injury.

Introduction:

In previous studies, patients had to wear several sensors on the body and another computing device was needed to run the FOG detection algorithm. Moreover, the features used in the algorithm were based on low-level and hand-crafted features. In this study, we propose a FOG detection system based on a smartphone, which can be placed in the patient's daily wear, with a novel convolutional neural network (CNN).

Methods:

The walking data of 32 PD patients were collected from the accelerometer and gyroscope embedded in the smartphone, located in the trouser pocket. The motion signals measured by the sensors were converted into the frequency domain and stacked into a 2D image for the CNN input. A specialized CNN model for FOG detection was determined through a validation process.

Results:

We compared our performances with the results acquired by the previously reported settings. The proposed architecture discriminated the freezing events from the normal activities with an average sensitivity of 93.8% and a specificity of 90.1%.

Conclusions:

Using our methodology, the precise and continuous monitoring of freezing events with unconstrained sensing can assist patients in managing their chronic disease in daily life effectively.

Introduction

Parkinson's disease (PD) is a chronic neurological disorder affecting 10 million people worldwide.¹ The disease results from the loss of dopamine-producing nerve cells in the basal ganglia.² PD impairs the motor functions of patients and has few symptoms, including tremors, bradykinesia, rigidity, postural instability, and the freezing of gait (FOG). FOG is a frequently observed gait disturbance among PD patients. The motor symptom of FOG is defined as a profound decrease in the step length or an inability to walk.³ Freezing events occur occasionally and intermittently, lasting for a few seconds to minutes, when patients attempt to start walking, turning, or entering doors.⁴ FOG during walking restricts daily activities and increases the risk of falling, which can cause personal limitations and various health problems.⁵

FOG can be monitored conventionally by motor examination in outpatient clinics; however, going to a hospital is cumbersome at times due to distance and mobility impairments. This results in infrequent symptom measurements and improper treatment. With rapid technological development, the telemonitoring of FOG events is possible, with sensors near the patients. The feasibility of FOG event detection has been investigated in many studies utilizing wearable sensors, such as force-sensitive shoe insoles,⁶ video cameras with 3D markers,⁷ and inertial sensors.^{8

–18} Among them, the accelerometer and gyroscope have been intensively studied because of their low cost and compact size that enables easy wearing. However, patients had to attach several sensors to the body and another computing device was needed to run the detection algorithm; hence, they were not practical for real-life application.

To address these issues, a smartphone as a sensing device was suggested as a solution for FOG detection, in recent studies.^19,20 The accelerometer and gyroscope, which can measure user motion, are embedded in the smartphone. Smartphones have practical and economic advantages over the other sensors because additional hardware is not needed owing to a penetration rate of more than 80% in advanced economies.²¹ The smartphone can be used as a computing device to run the FOG detection algorithm. In addition, as people always carry Internet-connected smartphones in daily life, unconstrained and cumulative monitoring is possible.

The detection algorithm is a crucial component for adopting the technology-based service. To achieve similar or better performances with a single device, the development of sophisticated method is essential. However, the algorithms investigated in previous literature have several limitations. Many of these studies have identified freezing symptoms by establishing specific thresholds, which can discriminate them from normal walking.^8,9,19 This method is manual and subject dependent; therefore, it prevents the algorithm from adding complexity above a certain degree, to offer a better performance. Furthermore, the features used in previous studies are based on domain-specific knowledge; hence, the determination of the discriminative features in such a handcrafted method is limited. In addition, previous reports^9,20 have already proved that a classifier using common features exhibited varied performances, based on the patients' walking and freezing styles. As these issues can adversely affect the performance, another methodology for detecting the FOG is required.

Convolutional neural networks (CNNs), which are driving the artificial intelligence boom, of late, are the key to solving the abovementioned problems. CNN extracts the best discriminative features and learns the concepts automatically by applying convolution filters to raw data, instead of using predesigned “shallow” features.²² It is suitable, if the detection algorithm is developed, without prior knowledge of individual data. As the CNN framework exploits several neural network layers, which can extract the features from raw sensor signals, it can effectively learn the various, complex walking styles of patients. This structure has not only outperformed many of the established methods, but has also demonstrated the results surpassing human-level performance in image classification²³ and speech recognition.²⁴

In this study, we present a system that can remotely monitor the FOG events of PD patients, using sensors embedded in a single smartphone. We collected the movements of PD patients from a smartphone located in their trouser pockets, in a natural position. The measured signals were transferred to a remote server and saved as 2D images, which enable the CNN to learn the classification of FOG patterns. CNN was used to classify walking and freezing events with a high accuracy, through validation. To evaluate our approach objectively, the performance of the proposed structure was compared with the results in previous literature.

Materials and Methods

Data Collection

The walking data of PD patients was collected from Android smartphones (Nexus 5; Google, Inc.); the system was equipped with internal sensors, a tri-axial accelerometer, and a gyroscope. Each sensor has measurement ranges of ±2 G and ±250 °/s, respectively. Motion signals were sampled at a rate of ∼200 Hz, the fastest sampling mode supported by the mobile OS. During the experiment, videos were recorded using another smartphone (Galaxy S5; Samsung Electronics, Inc.) for freezing assessment and posterior referral review. Two more smartphones were attached to the patient to compare the detection performances, according to the measurement positions. To synchronize the time differences across the devices, custom applications that can allow socket communication were developed. Server and client applications were installed in all the smartphones. The smartphone recording the video sends trigger messages, which activate the motion sensors automatically in the devices, while recording. The collected data on the smartphone were transferred to a remote server that can perform the classification task using our suggested algorithm through Wi-Fi.

Experimental Setup

Thirty-two PD patients (20 males, 12 females) participated in this study voluntarily. All the participants wore their daily clothes during the experiments. To measure the movements of the patients, a smartphone was casually located in the trouser pocket, as usually placed. The trouser pocket of each patient was sufficiently tight, which did not affect the measurements significantly. The other smartphones were located at the lower limbs to compare the performances with those of previous studies. The most common placement for the sensors was on the ankle, where the smartphone was attached using an elasticized armband. We also attached another smartphone on the lateral side of the waist with a belt clip.

The experiments were carried out in a hospital corridor. Initially, the subjects were made to sit on a chair, after which they were made to walk 3 m along the taped line on the floor. They turned around at the end of the line and returned to the original position. After walking, the patients were asked to perform additional tasks, such as turning around, and opening and entering doors, to provoke a FOG. All the procedures were performed twice to acquire sufficient data and their average duration was 153 s. The overview of the experimental protocols is described graphically in Figure 1. The study was approved by the Seoul National University Hospital Ethics Committee and written informed consent was obtained.

Fig. 1.

Overview of the experimental environment. The smartphones were placed in the patient's pocket and attached additionally to the ankle and the waist for reference. Another smartphone videorecorded the entire experiment, controlling the start and stop of all the smartphones with a Wi-Fi connection.

Algorithm

The measured walking data were segmented into 2.5-s windows and transformed into the frequency domain using discrete Fourier Transform for utilizing the gait frequency information. A specialized CNN structure was designed to detect FOG patterns. We could determine the optimal architecture, consisting of two convolution layers with 20 convolution filters, through parameter optimization processes. The network was trained with walking data and the corresponding annotations of experienced doctors. Detailed explanation on the training and establishment of the CNN structure is presented in the Appendix.

Evaluation

The entire data were divided into 90% for training and 10% for the test dataset. A 10-fold cross-validation was performed using the training data for performance comparison. To determine the best model configuration, a single-number evaluation metric is required. An F1-score was selected for considering both the precision and the recall [F1-score = 2 (precision recall)/(precision + recall)]. The sensitivity and specificity were also computed on a test set to compare the classification results with those in previous literature. Several machine learning classifiers, listed in Table 2 were also tested to verify the effectiveness of the CNN. All features used in literature for training conventional classifiers are shown in Table 1.

Table 1.

Shallow Features Used in Previous Studies for FOG Detection

DOMAIN	FEATURE	DESCRIPTION
Time	Mean¹¹	Average value of its signal
	Variance¹¹	Average of the squared deviation of a signal from its mean
	Standard deviation¹¹	Square root of its variance
Frequency	Freezing index⁸	Power in the “freeze” band (3–8 Hz) divided by the power in the “locomotion” band (0.5–3 Hz)
	Step frequency¹⁹	Second power spectrum harmonic
	Entropy¹²	Quantitative measure of dispersion in the power spectrum
	Energy¹¹	Sum of the squared FFT-transformed magnitudes of the signal, divided by the window length
	Power⁹	Sum of the power in the “freeze” and “locomotion” band

FFT, fast Fourier transform; FOG, freezing of gait.

Results

Dataset

Seventeen out of the 32 patients exhibited freezing symptoms during the experiments in the clinic. A total of 110 FOG episodes were observed with an average duration of 57.8 s. The total number of labeled images was 5,839 and freezing episodes accounted for 41.7%, when a 2.5 s window size was used.

Classification Result

Table 2 lists the comparative results between our CNN approach and a few other machine learning classifiers with shallow features (Table 1), presented in literature. The CNN diagnosed the FOG patterns with a sensitivity of 93.8% and a specificity of 90.1%. Compared with our approach, the other conventional classifiers showed significantly lower F1 scores.

Table 2.

Performance Comparison Between the Convolutional Neural Network Architecture and the Machine Learning Classifiers with Shallow Features

APPROACH	F1-SCORE (%)	SENSITIVITY (%)	SPECIFICITY (%)
CNN (Ours)	91.8	93.8	90.1
Random forest	73.5	70.8	89.1
Multilayer perception	72.1	71.8	83.1
Decision tree	68.7	70.6	81.6
Support vector machine	67.5	62.2	89.5
Naive Bayes	56.6	60.3	74.8

CNN, convolutional neural network.

We have referred to the recent comprehensive review²⁵ on FOG detection and have summarized the comparative statistics with the previous studies, in Table 3. The type of sensors used for gathering the movements of the patients and their measured positions are listed. In the table, the sensitivity and specificity are presented with the classification metrics, as in most studies.

Table 3.

Comparison of Our Proposed Solution with the Methods Suggested in Previous Literature

	SENSORS (NO. OF SENSORS)	PLACEMENT	SENSITIVITY (%)	SPECIFICITY (%)
Our solution	Smartphone (1)	Trouser pocket	93.8	90.1
Bachlin et al.⁹	Accelerometer (3)	Ankle, thigh, lower back	73.1	81.6
Zach et al.¹⁶	Accelerometer (1)	Waist	78	76
Tripoliti et al.¹²	Accelerometer (4), IMU (2)	Leg (ankle), waist, chest, wrist	81.9	98.7
Cole et al.¹⁰	Accelerometer (2), EMG (1)	Forearm, thigh	82.9	97.3
Moore et al.¹³	Accelerometer (7)	Foot, ankle, thigh, waist	84.3	78.4
Rezvanian and Lockhart¹⁷	Accelerometer (1)	Ankle	84.9	81
Kwon et al.¹⁵	Accelerometer (2)	Shoe	86	86
Pepa et al.¹⁹	Smartphone (1)	Waist	89	97
Mazilu et al.¹⁸	IMU (1)	Wrist	90	66–80
Djurić-Jovicić et al.¹⁴	IMU (1)	Ankle	100	99

EMG, electromyography; IMU, inertial measurement unit.

We compared the performances between the measurement positions, for comparison with previous studies. Table 4 presents the FOG pattern classification results, when the smartphone is placed at different locations. We obtained an F1-score of 90.2% with the smartphone in the pocket, which is the most convenient placement. For reference, when we tested the detection accuracies at the ankle and waist, the average F1-scores were 87.8% and 90.5%, respectively.

Table 4.

Classification Results of the 10-Fold Cross-Validation at Different Measurement Sites

PLACEMENT	F1-SCORE (%)	SENSITIVITY (%)	SPECIFICITY (%)
Pocket	90.2	89.5	92.7
Ankle (reference)	87.8	88.1	91.1
Waist (reference)	90.5	89.4	93.5

Discussions

In this study, we proposed a novel single smartphone-based system to monitor the walking status of PD patients remotely. We applied the CNN-based method to the data collected from the internal sensors of a single smartphone, casually placed in the pocket of a PD patient simulating daily life motions. The measured signals were transformed into images containing frequency components for training and testing the CNN structure. This study significantly improved the convenience, as the smartphone was placed casually in the patient's pocket, unlike most previous studies, which utilized and fixed many sensors artificially. We applied the CNN algorithm, which extracts discriminative features automatically from raw data, to analyze various types of walking signals. In addition, the approach can classify symptoms with a high accuracy, owing to its immense learning capacity. To verify the effectiveness of our system, we extensively compared our performance with the previously reported results.

Our system suggested the use of the CNN algorithm to detect freezing events, using the walking data acquired from a smartphone. Previous studies were mainly conducted in two ways: Most of them^8,9,13,14,19 manually designed the fixed-threshold algorithms; the methods detected FOG events, when some of the features exceeded a certain threshold. Another strategy for FOG detection was to use machine learning classifiers.^11,12 However, both approaches require a manual feature extraction process, by observing the raw signals acquired from sensors. These methods cannot achieve full performance and applied to many patients because researcher intervention is needed.

The proposed CNN architecture automatically finds the optimal parameters directly from the raw data of each patient, as shown in the successful applications, in image recognition.^22,26 To examine the performance of the CNN in FOG detection, we tested machine learning algorithms with shallow features, on the same dataset. In Table 2, the CNN sensitivity was 20% or higher than that of the Random Forest, which showed the highest accuracy among the classifiers. As per these results, the recognition performance of the deep learning approach outperforms conventional algorithms. In addition, it can adapt to various types of patient data and remotely distributed with minimal researcher intervention.

We also compared the configuration of our system with those in previous studies. The type and number of sensors, and their placement, in literature, are summarized in Table 3. Many studies^9,12,13 attached several sensors to the body for detecting freezing events, causing discomfort to the patient, in daily life. However, we demonstrated a superior sensitivity over most of those results, even when only a single device was used. Recent studies have considered the convenience of patients; hence, the use of a single sensor is being investigated. In^14,19 FOG detection was possible using a sensor, but it was fixed at the ankle and waist as in previous studies, which is not a natural position. Some researchers¹⁸ explored the use of a smart watch that can monitor the motion of the wrist naturally, but it is impractical for usage in daily life because of its degraded and inconsistent specificity.

As described in Table 3, previous studies explored various sensor positions for FOG detection. To evaluate the effect of the sensor positions, we tested the classification performance of the smartphone located in the pocket, using simultaneously collected data at the ankle and waist, during the experiment. Each row in Figure 2 represents the acceleration and gyroscope signals collected using smartphones attached to the abovementioned locations. The walking-signal measurements were sufficiently robust to detect the changes in signal shapes, when the patient experienced a freezing event, even when the smartphone was placed in the pocket. From Table 4, the smartphone in the pocket shows comparable classification results over other positions. The waist position presented a slightly better accuracy than that of the pocket case. This is understandable because acceleration at the waist level can measure the movement of the torso, close to the center of mass of the human body. As a result, it is the common location adopted by systems based on a single accelerometer in human activity recognition (HAR).^27
–29 Moreover, some researchers, who studied FOG detection,¹³ recommended the waist position as a configuration because it shows robust performance. Nonetheless, the most convenient placement for smartphone sensing is the pocket because it can be located naturally. Unlike other wearable sensors, a smartphone is carried by people and this contributes to the acceptance of the walking activity monitoring. Furthermore, it is always connected to the Internet; hence, the progress of the disorder can be tracked quantitatively, by communicating information remotely between patients and doctors.

Fig. 2.

Y-axis acceleration signals (left side) and X-axis angular velocity signals (right side) measured from each body position from patients who experienced freezing of gait. Normal walking and freezing-of-gait events were performed on the sections colored gray and black, respectively.

FOG detection with a smartphone is a promising approach because it can increase the frequency of monitoring without burdening professional resources. However, the cost of training and testing the detection algorithm was too high to be performed on a smartphone; hence, the results were calculated using a remote server and transferred to the device. In future, a mobile application that performs all the processes using a single smartphone must be developed. Moreover, the proposed architecture should be tested with a large cohort dataset of the motion of PD patients, to generalize our results for unseen patient data in telemonitoring. Thereby, the program can be widely distributed and the feasibility of home monitoring in daily life, investigated.

Footnotes

Acknowledgment

This work was supported by grant no. 3420140050 from the SNUH Research Fund and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. NRF-2017R1A5A1015596).

Disclosure Statement

No competing financial interests exist.

References

Dorsey

, Constantinescu

, Thompson

, Biglan

, Holloway

, Kieburtz

, Marshall

, Ravina

, Schifitto

, Siderowf

, Tanner

. Projected number of people with Parkinson disease in the most populous nations, 2005 through 2030; Neurol, 2007; 68:384–386.

Jankovic

Parkinson's disease: Clinical features and diagnosis. J Neurol Neurosurg Psychiatry, 2008; 79:368–376.

Nutt

, Bloem

, Giladi

, Hallett

, Horak

, Nieuwboer

. Freezing of gait: Moving forward on a mysterious clinical phenomenon. Lancet Neurol, 2011; 10:734–744.

Schaafsma

, Balash

, Gurevich

, Bartels

, Hausdorff

, Giladi

. Characterization of freezing of gait subtypes and the response of each to levodopa in Parkinson's disease. Eur J Neurol, 2003; 10:391–398.

Bloem

, Hausdorff

, Visser

, Giladi

. Falls and freezing of gait in Parkinson's disease: A review of two interconnected, episodic phenomena. Mov Disord, 2004; 19:871–884.

Hausdorff

, Schaafsma

, Balash

, Bartels

, Gurevich

, Giladi

. Impaired regulation of stride variability in Parkinson's disease subjects with freezing of gait. Exp Brain Res, 2003; 149:187–194.

Delval

, Snijders

, Weerdesteyn

, et al. Objective detection of subtle freezing of gait episodes in Parkinson's disease. Mov Disord, 2010; 25:1684–1693.

Moore

, MacDougall

, Ondo

. Ambulatory monitoring of freezing of gait in Parkinson's disease. J Neurosci Methods, 2008; 167:340–348.

Bachlin

, Plotnik

, Roggen

, et al. Wearable assistant for Parkinson's disease patients with the freezing of gait symptom. Inf Technol Biomed IEEE Trans, 2010; 14:436–446.

10.

Cole

, Roy

, Nawab

Detecting freezing-of-gait during unscripted and unconstrained activity. Med Biol, 2011:5649–5652.

11.

Mazilu

, Hardegger

, Zhu

, et al. Online detection of freezing of gait with smartphones and machine learning techniques. In: 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth). San Diego, CA: IEEE, 2012; 123–130.

12.

Tripoliti

, Tzallas

, Tsipouras

, et al. Automatic detection of freezing of gait events in patients with Parkinson's disease. Comput Methods Programs Biomed, 2013; 110:12–26.

13.

Moore

, Yungher

, Morris

, et al. Autonomous identification of freezing of gait in Parkinson's disease from lower-body segmental accelerometry. J Neuroeng Rehabil, 2013; 10:19.

14.

Djurić-Jovicić

, Jovicić

, Radovanović

, Stanković

, Popović

, Kostić

. Automatic identification and classification of freezing of gait episodes in Parkinson's disease patients. IEEE Trans Neural Syst Rehabil Eng, 2014; 22:685–694.

15.

Kwon

, Park

, Kim

, Ho

. A practical method for the detection of freezing of gait in patients with Parkinson's disease. Clin Interv aging, 2014; 9:1709–1719.

16.

Zach

, Janssen

, Snijders

, et al. Identifying freezing of gait in Parkinson's disease during freezing provoking tasks using waist-mounted accelerometry. Park Relat Disord, 2015; 21:1362–1366.

17.

Rezvanian

, Lockhart

. Towards real-time detection of freezing of gait using wavelet transform on wireless accelerometer data. Sensors, 2016; 16:475.

18.

Mazilu

, Blanke

, Calatroni

, Gazit

, Hausdorff

, Tröster

. The role of wrist-mounted inertial sensors in detecting gait freeze episodes in Parkinson's disease. Pervasive Mob Comput, 2016; 33:1–16.

19.

Pepa

, Ciabattoni

, Verdini

, Capecci

, Ceravolo

. Smartphone based Fuzzy Logic freezing of gait detection in Parkinson's Disease. In: 2014 IEEE/ASME 10th International Conference on Mechatronic and Embedded Systems and Applications (MESA). Senigallia-Ancona, Italy: IEE/ASME, 2014; 1–6.

20.

Kim

, Lee

, et al. Unconstrained detection of freezing of Gait in Parkinson's disease patients using smartphone. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, vol. 2015. Milan, Italy: IEEE, 2015; 3751–3754.

21.

Poushter

Smartphone Ownership and Internet Usage Continues to Climb in Emerging Economies. Pew Res Cent Glob Attitudes Trends, 2016; 22:1–44.

22.

Le Cun

, Boser

, Denker

, Henderson

, Howard

, Hubbard

, Jackel

. Handwritten Digit Recognition with a Back-Propagation Network. In: Advances in Neural Information Processing Systems. 1990; 396–404. DOI:10.1111/dsu.12130.

23.

Krizhevsky

, Sutskever

, Hinton

. Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems. Lake Tahoe, NV: Curran Associates, 2012; 1097–1105.

24.

Hinton

, Deng

, Yu

, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Process Mag, 2012; 29:82–97.

25.

Silva de Lima

, Evers

LJW

, Hahn

, et al. Freezing of gait and fall detection in Parkinson's disease using wearable sensors: A systematic review. J Neurol, 2017; 264:1642–1654.

26.

Deng

, et al. Construction and analysis of a large scale image ontology. Vision Sciences Society, 186 (2009): 2.

27.

Sekine

, Tamura

, Akay

, Fujimoto

, Togawa

, Fukui

. Discrimination of walking patterns using wavelet-based fractal analysis. IEEE Trans Neural Syst Rehabil Eng, 2002; 10:188–196.

28.

Long

, Yin

, Aarts

. Single-accelerometer-based daily physical activity classification. In: Proceedings of the IEEE 2009 Annual International Conference of the Engineering in Medicine and Biology Society (EMBC 2009). 2009. Minneapolis: IEEE, 2009; 6107–6110.

29.

Rodriguez-Martin

, Samà

, Perez-Lopez

, Català

, Cabestany

, Rodriguez-Molinero

. SVM-based posture identification with a single waist-located triaxial accelerometer. Expert Syst Appl, 2013; 40:7203–7211.

30.

Capela

, Lemaire

, Baddour

. Feature selection for wearable smartphone-based human activity recognition with able bodied, elderly, and stroke patients. PLoS One, 2015; 10:e0124414.

31.

Glorot

, Bordes

, Bengio

Deep sparse rectifier neural networks. In: AISTATS. Ft. Lauderdale, FL: PMLR, 2011; 15:275.

32.

Simonyan

, Zisserman

Very deep convolutional networks for large-scale image recognition. CoRR, 2014;abs/1409.1.

33.

Szegedy

, et al., Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015, pp. 1–9. doi: 10.1109/CVPR.2015.7298594

34.

, Zhang

, Ren

, Sun

Identity mappings in deep residual networks. In: European Conference on Computer Vision. 2016; 630–645.

35.

Chollet

Keras. 2015. https://github.com/fchollet/keras

36.

Abadi

, Agarwal

, Barham

, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv Prepr arXiv160304467. 2016.

37.

Kingma

, Ba

Adam: A method for stochastic optimization. arXiv Prepr arXiv14126980. 2014.

38.

Karpathy

Cs231n convolutional neural networks for visual recognition. Neural networks, 2016, 1.

39.

Goodfellow

, Bengio

, Courville

. Deep learning, (Vol 1). Cambridge: MIT Press, 2016.