How to Annotate Freezing of Gait from Video: A Standardized Method Using Open-Source Software

Freezing of gait (FOG) is a leading cause for falls and a poor quality of life for many patients with Parkinson’s disease and related disorders [1, 2]. It can be defined as a “brief, episodic absence or marked reduction of forward progression of the feet despite the intention to walk” [3]. Current treatments for FOG offer limited symptomatic relief [4], so there is strong need for clinical trials aimed primarily at reducing FOG severity [5]. To date, such trials have been scarce, in large part due to the lack of a reliable outcome measure that is sensitive to detect changes in FOG severity. As a result, FOG has most frequently been assessed as a secondary- or even tertiary outcome using freezing of gait questionnaires [6, 7]. These subjective scales have been shown to be good screening tools for detecting patients who experience FOG, yet they might poorly depict its severity [8]. The ‘old’ freezing of gait questionnaire in particular includes two general gait-related items that are not specific to FOG [6]. Lastly, the minimal detectable change of these scales following an intervention remains to be determined, as is the precise clinical impact a reduction in the total score of these scales has on the wellbeing of patients.

Visually scoring FOG episodes from video recordings has therefore been put forward as the gold-standard for measuring the actual severity of FOG [8, 9]. The percentage of time spent with FOG (% FOG) can be obtained as an outcome from standardized gait assessments that are video-taped and scored offline for FOG episodes by a trained expert. The % FOG can then be calculated using the following formula:

% FOG = ( Total duration of FOG observed during the gait task * 100 ) Total duration it took the participant to perform the gait task

Depending on the study objectives, the % FOG can be calculated from all FOG episodes taken together, per FOG ‘phenotype’ based on the leg motion observed (shuffling, trembling, complete akinesia) [10], as well as per trigger of FOG (e.g., turning, doorways) [3]. A strength of the % FOG is that it is an objective outcome of ratio measurement level that directly reflects the severity of FOG at the time of testing. It can be obtained from any gait task, as long as the legs and feet of the participant have been recorded on video. Importantly, the % FOG can be scored offline by raters who are blinded to the intervention allocation, thus allowing it to be used as an outcome in double-blinded therapeutic trials. Finally, the % FOG can be used to validate other FOG measures, for example by correlating its values against subjective scales and diaries or other objective outcomes, such as derived from wearable sensors [9]. Participants can wear the sensors while performing the video-recorded gait assessments, thereby

allowing sensitivity and specificity of the wearable signals to detect FOG as seen on video to be calculated.

A recent randomized controlled trial set the stage by obtaining the % FOG as the primary outcome, showing a significant reduction in FOG severity following a cognitive training program as compared to a sham intervention in PD patients who experienced FOG at baseline [11]. Importantly, however, for % FOG to be implemented as a true gold-standard outcome measure, its scoring should be standardized across raters and studies. In addition, software to score FOG from video should be made widely accessible to encourage its implementation.

I therefore hereby present a free template that can be implemented in open source software to visually score % FOG from any video recording. I also provide a user guide on how to navigate the software and personalize the template, as well as guidelines and definitions (see Table 1) on how to visually score FOG during frequently recorded gait assessments. These definitions are provided in an attempt to standardize the visual scoring of FOG from video footage across raters and studies. Regardless, researchers might choose to apply different definitions. It is therefore advised to clearly state the definitions applied when disseminating % FOG outcomes.

OPEN IN VIEWER

Table 1

Definitions to standardize the visual scoring of FOG from video recordings of gait assessments that are commonly used in the clinical and research setting

Task/Event	Start Time	End Time
Gait tasks that start- and end in a seated position (e.g., Timed up and Go -TUG).	The moment the subject lifts off from the chair (i.e., the buttocks no longer touches the chair). *This is easier to score on video than the release of the participant’s back from the chair when the camera is positioned in front of the participant.	The moment the subject touches back down on the chair (i.e., the buttocks touches the chair) after performing the gait task.
Gait tasks that start- and end from a standing position.	The moment the “GO” signal is given by the instructor. *This allows start hesitations to be scored.	The moment the participant returns to the starting position.
Turning after a period of walking (e.g., during a TUG).	The moment of toe-off of the first step that touches down in the area where the turn should be performed with the foot pointing towards the direction of the turn.	The moment of heel-strike of the first step that leaves the area of where the turn was performed with the foot pointing towards the end target of the gait task.
Turning on the spot (e.g., during a 360 degrees turning task)	The moment the “GO” signal is given by the instructor.	The moment the participant returns to the starting position.
Freezing of Gait (FOG) Based on the definition of FOG by Nutt et al., (2011) [3]	The moment when the foot of the participant is suddenly no longer producing an effective step forward and is displaying FOG-related features (trembling, shuffling, complete akinesia), despite the participant’s intention to continue walking.	The moment of initial toe-off after the FOG when the participant is again able to perform at least two effective alternating steps with both legs showing no FOG-related features. For example, if during a FOG episode one leg is suddenly moving forward but the other leg is still showing FOG-related features, than this can be counted as the same episode.

There are many different software packages that could be used to annotate FOG, but most of these require licensing. The software I present here is called ELAN, which is developed by The Language Archive of the Max Planck Institute for Psycholinguistics in Nijmegen, The Netherlands, as a professional tool for creating complex annotations on video and audio recordings. It may be used free of charge and is provided open-source. The ELAN software can be downloaded from the developers’ website at: https://tla.mpi.nl/tools/tla-tools/elan/. The benefit of ELAN is that users can create their own templates to score the onsets and durations of all sorts of tasks and performances. ELAN thus lends itself perfectly for scoring FOG from video. I therefore created a ‘FOG-scoring’ template containing task conditions and predefined annotations that are frequently used to score FOG in the clinic (Fig. 1A). This template along with the user-guide on how to score % FOG can be downloaded for free from: https://morangilat.com.

OPEN IN VIEWER

Fig.1

Overview of the FOG-scoring Template as implemented in the ELAN software. A) Layout of the FOG-scoring template with the video presented at the top and the area to place annotations below; B) Example output from the events scored in (A), including the calculation of the % FOG for that particular Timed up and Go (TUG) task with a freezing of gait (FOG) episode of the trembling type occurring when the participant was turning 180 degrees to the right.

Using ELAN, task conditions (e.g., Timed up and Go (TUG)) and events (e.g., FOG) can be annotated using simple mouse-clicks. The labels of the annotations can be pre-specified or written as free text, allowing for endless variations. ELAN outputs the onsets and durations of each annotation with millisecond precision. This allows the % FOG to be calculated from its output (Fig. 1B). The same software was used to calculate % FOG by Walton et al. (2018) [11], showing a high intraclass correlation coefficient of 0.902 for scoring FOG across six independent and blinded assessors.

The template I created allows for the annotation of: i) frequently used gait tasks, including different TUG variations; ii) different triggers of FOG such as turning, doorways and dual tasking [3]; iii) different phenotypes of FOG based on the leg motion detected such as shuffling, trembling or complete akinesia [10], and; v) free text notes. This template thereby provides a starting point for calculating % FOG, characterizing each FOG episode based on phenotype and trigger, as well as validating FOG detection algorithms from wearable sensor data [9]. Yet, users are free to modify the template, as explained in full in the user guide.

It is my hope that by disseminating this scoring method open-access, researchers and clinicians across the field will be encouraged to obtain the % FOG as a primary outcome in their studies and therapeutic trials. By providing these guidelines it is also my goal to standardize the scoring of % FOG, which would allow for the comparison of outcomes across studies and centers.

CONFLICT OF INTEREST

The author has no conflict of interest to report.