Clinical machine learning predicting best stroke rehabilitation responders to exoskeletal robotic gait rehabilitation

2.1 Study design

This was an observational cohort study that used secondary analysis of data from our previous ongoing projects. Data screening was performed by three investigators. Two investigators determined the eligibility and completeness of the data. Patients that completed the interventions and outcome measurements at pre- and post-intervention were included for analysis during 01.01.2022∼01.01.2023.

2.2 Participants

In total, 187 patients with subacute stroke (mean age, 64±13.33 years; sex, 111 male and 76 female individuals) were recruited from a major hospital in masked for review. Tables 2 and 3 outline the characteristics of patients with hemiparetic stroke. The inclusion and exclusion criteria are presented in Supplementary file 1. Interestingly, 51.3% (n = 91) of the 187 patients with stroke exhibited good improvement, while the rest demonstrated the poor improvement. An experienced physical therapist was consistently conducted all outcome measurements. All participants provided their informed consent prior to the initiation of clinical trials. This study was conducted in accordance with the tenets of the Declaration of Helsinki. The present study protocol was approved by the Institutional Review Board of Chungdam Rehabilitation Hospital Center (CDIRB-2023-004). The study flowchart is presented in Fig. 1.

OPEN IN VIEWER

Fig. 1

The study flowchart.

We selected the 18 potential predictors based on ICF to include “Body function” and “Activity.” We compiled data on 12 clinical outcomes and six demographic features, reflecting the patient’s body function and activity domain. The six demographic data were: (1) age, (2) sex, (3) side of the lesion, (4) onset time, (5) height, and (6) weight. The 12 clinical outcome data were: (1) initial FMA, (2) final FMA, (3) initial BBS, (4) final BBS, (5) initial MBI, (6) final MBI, (7) initial MMSE, (8) final MMSE, (9) initial trunk impairment scale (TIS), (10) final TIS, (11) initial modified Ashworth scale (MAS), and (12) final MAS scores (Tamura et al., 2022; Zeltzer, 2010; Verheyden et al., 2004; Loewen & Anderson, 1988; Blackburn et al., 2002; Bour et al., 2010). Interestingly, the MAS measurements specifically targeted the ankle plantar flexor muscles. In both research and clinical contexts, these features are commonly utilized to gauge the rehabilitation recovery of patients with stroke (Tamura et al., 2022; Zeltzer, 2010; Verheyden et al., 2004; Loewen & Anderson, 1988; Blackburn et al., 2002; Bour et al., 2010).

2.4 Feature importance analysis

To investigate the feature importance among the various input features, the five ML algorithms were used to forecast good or poor functional and structural improvement in individuals with subacute stroke who underwent RAGT (compared to the first session) and elucidate how certain features influence the improvement after RAGT. Feature importance is a valuable metric, offering a quantitative measure of the impact each predictor variable on the target outcome, extracting the clinical insights (Kuo et al., 2021). For a comprehensive evaluation, our study compared among the feature importance values derived from the decision tree (DT) model and ensemble model based on the DT model, including random forest (RF), eXtreme Gradient Boosting (i.e., XGBoost [XGB]), the light gradient boosting machine (i.e., LightGBM [LGB]), and the categorical boosting (i.e., Catboost [Cat]). Subsequently, a comparative analysis was performed to ascertain the consistency in feature importance rankings across the different models.

We also constructed partial dependence plots (PDP) to identify the most influential features and characterize the contribution of each feature to the initial ability of individuals with subacute stroke, visualizing the relationship between these features and the model’s predicted outcome (Kuo et al., 2021; Zheng et al., 2022).

2.5 Machine learning algorithms and hyperparameter tunning

In our analysis, we considered a set of 18 predictors. The primary objectives of this study were (1) to determine the best predictive model for categorizing stroke patients into “good” and “poor” functional and structural improvement groups based on their response to Walkbot RAGT intervention and (2) identify key features that significantly influence individuals with subacute stroke recovery who underwent RAGT. For our ML analysis, we deployed five distinct models. These models were chosen for their capability in handling the binary and multiple classification tasks effectively, as well as for their proven ability to discern feature importance. Such features are instrumental in understanding combined factors related to the prognosis of structural recovery, balance function, and overall independence-related ambulation in patients with subacute stroke (Fujita et al., 2020).

It is important to highlight that while RF adopts a bagging ensemble approach, XGB, LGB, and Cat use the boosting ensemble methodology. These ensemble models have been extensively applied in various clinical domains, showcasing features, such as quick training duration, minimal overfitting, and commendable accuracy (Al Daoud, 2019). To ensure a rigorous evaluation and guard against model overfitting, we adopted the 10-fold cross-validation technique. The dataset was partitioned into 10 subsets: nine of these were utilized for training, with the remaining one reserved for testing (Kuo et al., 2021; Rodríguez et al., 2010).

2.6 Classification of individuals with subacute stroke improvement

To effectively examine the degree of patient’s functional and structural improvement, including FMA, BBS, and MBI after RAGT, this study introduces two keys classification criteria: the minimal clinically important difference (MCID) and the four-effect level. MCID is defined as the minimal change in assessment scores that patients perceive as clinically significant in relation to their health status (Cook, 2018). In this research, ICF framework informed our selection of three primary outcomes for evaluation: the FMA, BBS, and MBI. These measures, extensively employed in similar studies, provide a comprehensive insight into a patient’s level of body function and activity domain (Patel et al., 2020). The FMA is a 226-point scale and is divided into five domains (i.e., motor and sensory function, balance, joint range of motion, and joint pain) developed as an evaluation tool for overall recovery from stroke (Pandian et al., 2016). The BBS is a 56-point balance function scale (Tamura et al., 2022). The MBI is a 100-point scale assessing the activities of daily living (Hsieh et al., 2007).

Another classification metric utilized is the four-effect level, which quantifies the impact of the Walkbot RAGT. Four-effect level ranges from 0 to 3, where each number represents the sum of MCID achieved across the FMA, BBS, and MBI: Effect Level 0, 1, 2, and 3 indicate poor, moderate, good, and excellent improvement, respectively. This grading provides a multi-faceted understanding of post-treatment progress, adding depth to the MCID’s binary classification.

2.7 Model evaluation metrics

The performance of ML models was evaluated using the standard ML performance metrics including (1) accuracy, (2) recall, (3) precision, and (4) F1 scores (Kuo et al., 2021). The accuracy, recall, precision, and F1 score were defined as follows: Accuracy = TP + TN TP + TN + FP + FN Sensitivity = TP TP + FN Specificity = TN TN + FP Precision = TP TP + FP F 1 - score = 2 X PrecisionXSensitivity Precision + Sensitivity

Table 1 represents the confusion matrix. Accuracy is an overall index of prediction performance. Sensitivity is the ratio of participants who were correctly identified as positive by the model to those who were actually positive. Precision is the ratio of participants who were correctly identified as positive by the model to those who were labeled as positive by the model. F1 scores are the harmonic mean of precision and recall and are a combination index. The F1 score, ranging between 0 and 1, indicates the classification performance, with a higher value approaching 1 indicating better classification accuracy (Hsieh et al., 2007).

The categorical variables were coded, and continuous variables were standardized. Google Colaboratory (Google LLC, Mountain View, CA, USA), which is a web integrated development environment for python, was employed for model development and statistical analysis (Sokolova et al., 2009). For continuous variables, means and standard deviation were calculated, and for categorical variables, frequencies and percentages were calculated. The chi-square test and independent t-test were used to compare the categorical and continuous variables between the good and poor improvement groups. The level of significance was set at p < 0.05.

3.1 Significant differences were observed in RAGT parameters in the two study groups

Descriptive statistical analyses of continuous and categorical variables of the 187 patients with hemiparetic stroke in Tables 2 and 3, respectively. Age, initial Mini-Mental State Examination (MMSE), final MMSE, initial FMA, final FMA, initial BBS, final BBS, initial MBI, final MBI, initial TIS, and final TIS scores were significantly different between the “good-improvement group” and the “poor-improvement group”. The onset time, weight, height, and all categorical variables did not significantly differ between the two groups.

The predictive performances of different ML algorithms with 10-fold cross-validation using different numbers of input sessions to predict the improvements in FMA, BBS, and MBI’s MCID of the 12th session are delineated in Table 4. The XGB algorithm resulted in achieving the high performance of 0.8015, 0.7910, 0.8344, and 0.8010 in accuracy, sensitivity, specificity, and F1 score, respectively (Table 4a). We investigated whether there were significant differences in the means of the accuracy, sensitivity, and specificity between the ML models. In Table 4b, the accuracies from different ML model evaluated by 10-fold-cross validation for four levels were ranked from the highest to the lowest as follows: RF, 0.5192; LGB, 0.5082; XGB, 0.5020; Cat, 0.4857; and DT, 0.4216. Comparing the results outlined in Table 4a, the experimental findings revealed that employing all sessions as input to forecast improvements in FMA, BBS, and BMI’s MCID specifically by the 12th session provided a more detailed estimation. This approach aimed to discern whether an outcome variable performed below a certain threshold, as opposed to a binary classification of improvement. This enhanced granularity in estimation was attributed to the utilization of a greater number of classes for prediction.

The contribution of features in the five ML models with best performance is summarized in supplementary file 2. The feature importance among the five models was similar. The average of the feature importance is presented in Fig. 2a and 2b. The primary four features in the MCID model, ranked by importance, are final BBS, initial BBS, initial MBI, and final MAS scores. In contrast, the effect level model prioritizes the following top four features by importance: final BBS, initial BBS, final MAS, and initial MBI scores.

OPEN IN VIEWER

Fig. 2a

Feature importance of MCID (binary classification).

OPEN IN VIEWER

Fig. 2b

Feature importance of four-effect level (multiclassiffication).

As shown in Fig. 3 and supplementary file 3∼6, in accordance with the PDP, we can see that the initial BBS, MAS, and TIS scores were negatively associated with ICF structural and functional domain improvements. We demonstrated a positive correlation, indicating that a higher median value of initial FMA, initial MMSE, and initial MBI corresponds to a greater likelihood of observing substantial improvement. The effect level model’s PDP is similar to that of the MCID model’s PDP in supplementary file 7.

OPEN IN VIEWER

Fig. 3

Partial dependence plot of initial clinical outcomes.

4.1 Study limitations

Our study had two main limitations. First, the focus was primarily on body function, structure, and activity domains of the ICF framework, suggesting the need for future research to explore “participation restriction” and other contextual factors. Second, the study did not explore a range of potential ML models that could further enhance prediction of gait-related impairments, indicating a direction for future studies to optimize predictive analytics in rehabilitation.