Predictive models for independence after stroke rehabilitation: Maugeri external validation and development of a new model

1 Introduction

Stroke remains one of the leading causes of disability worldwide, with the majority of stroke survivors requiring specialized inpatient rehabilitation (Harari et al., 2020). Prediction of functional outcome after stroke rehabilitation can be used by clinicians to improve the accuracy of prognoses, set attainable goals, reach shared decisions, personalize rehabilitation plans and inform patients and relatives (Bates et al., 2015). Thus, many efforts have been devoted to identify such predictors (Meyer et al., 2015). Unfortunately, most prediction research focuses on model development and there are relatively few external validation studies (Riley et al., 2019). Besides, a clear distinction should also be made between reporting a model’s external performance done by the authors who developed the prediction model and done by independent investigators. Only a small fraction of the predictive models ever get evaluated on its performance by other authors using external data (Collins et al., 2014).

As recently reported by Stinear and colleagues in a topical review addressing prediction tools for stroke rehabilitation, further work is needed to externally validate the majority of prediction tools currently available (Stinear et al., 2019). Specifically addressing functional independence, Stinear et al. identify only three studies (Scrutinio et al., 2017; Douiri et al., 2017 and de Ridder et al., 2018) that consider several types of factors (e.g. history of diabetes mellitus, stroke lesion location, unilateral neglect, aphasia, previous stroke, dyslipidemia, NIHSS severity, etc.) and finally combine a small number of them in an easy to use way).

In the present study we conduct an external validation of the Scrutinio et al. Maugeri models, after considering their higher discrimination power (retained when restricting to patients admitted within 30 days after stroke onset), all their patients were recruited in inpatient rehabilitation Italian centers (with southern European lifestyle similar to our case), their longer reported length of stay (LOS) (when compared e.g. to Dutch settings, but similar to ours) and the availability of a free online easy-to-use calculator.

As shown in Table 1, Scrutinio et al. used a derivation data set (n = 717) and a validation data set (n =875). The primary outcome measure was the achievement of an M-FIM score of > 61 points at discharge. The secondary outcome measure was a physical independence grade ≥5, as defined in the Functional Independence Staging (FIS) system developed by Stineman and colleagues (Stineman et al., 2003). A derivation dataset was used to build two separate models: model 1 for the primary outcome and model 2 for the secondary outcome. Model 1 included age, time from stroke occurrence to rehabilitation admission, admission motor and cognitive Functional Independence Measure scores and neglect. Model 2 included age, male gender, time since stroke onset, and admission motor and cognitive Functional Independence Measure score.

The aims of the present study are i) to externally validate Maugeri model 1 and model 2, ii) to develop and internally validate a modified version of these models (model 3) iii) to provide a free online calculator of model 3 and deliver a public version of it that can be easily tailored to other predictive models by updating only their main parameters.

The study population consisted of 1419 patients consecutively admitted for stroke rehabilitation from March 2006 to September 2019. Patients were included in the study if they had been admitted within 90 days of onset of an ischemic or hemorrhagic stroke and had an admission FIM score of < 80 points and a motor (M)-FIM score of < 61 points.

Patients who were admitted > 90 days after stroke (n = 209; 29%), with no FIM assessment within a week since stroke rehabilitation admission (n = 198; 28%), had missing data (n = 156; 22%), had an admission FIM score of ≥80 points (n = 58; 8%), a M-FIM score of ≥61 points (n = 45; 6%) and were transferred to an acute care facility or discharged against medical advice (n = 43; 6%) were excluded. Therefore, 710 patients were available for analysis.

The setting was the inpatient Acquired Brain Injury rehabilitation unit of Institut Guttmann, certified in quality of care and patient safety (Joint Commission International since 2005, consecutively recertified in 2009, 2012 and 2018) (Joint Commission International, 2020).

At admission, each patient is assigned a medical doctor, who coordinates the rehabilitation team (a nurse, a physiotherapist, an occupational therapist, a social worker, and a clinical psychologist or neuropsychologist -speech specialist if required, based on the characteristics of the case). Therefore, trained physiotherapists recorded admission and discharge FIM scores, as a part of the rehabilitation program. All the data were extracted from the electronic Hospital Information System.

This is a STROBE compliant study and follows the Helsinki Declaration of 1975, as revised in 2008 and was approved by the Ethics Committee of Clinical Research of Institut Guttmann. The participants were anonymized and non-identifiable.

3 Results

3.3 Discrimination and calibration (model 3)

Table 4 presents AUC and calibration results using our dataset for model 1, model 2 and model 3. For model 3 primary outcome, AUC was 0.894 (95% CIs, 0.857–0.929), detailed in Supplementary Figure 2, the Hosmer–Lemeshow χ² statistic was 10.40 (p = 0.23). For model 3 secondary outcome, AUC was 0.845 (95% CIs, 0.789–0.900), detailed in Supplementary Figure 3, the Hosmer–Lemeshow χ² statistic was 6.94 (p = 0.54). Table 4 may also show AUC and calibration results for model 1 and model 2 using Maugeri derivation and validation datasets for the primary and secondary outcomes after obtaining written permission from the Authors (Scruitinio et al., 2017), to facilitate comparison of results.

OPEN IN VIEWER

Table 4

Performance (discrimination and calibration) of models 1, 2 and 3 using our dataset (for primary and secondary outcomes)

Model	Outcome	AUC (95% CI)	χ2
Model 1	Primary	0.873 (0.833–0.915)	6.07 (P = 0.63)
Model 2	Secondary	0.803 (0.749–0.857)	8.91 (P = 0.34)
Model 3	Primary	0.894 (0.857–0.929)	10.40 (P = 0.23)
Model 3	Secondary	0.845 (0.789–0.900)	6.94 (P = 0.54)

Figure 1 presents the calibration plots using our dataset for model 1 primary outcome (top left), model 2 secondary outcome (bottom left), model 3 primary outcome (top right) and model 3 secondary outcome (bottom right).

OPEN IN VIEWER

Fig. 1

Calibration plots using our dataset for model 1 primary outcome (top left), model 2 secondary outcome (bottom left), model 3 primary outcome (top right) and model 3 secondary outcome (bottom right).

The “Ideal” line represents perfect prediction as the predicted probabilities equal the observed probabilities. The “Apparent” line is the in-sample calibration. The “Bias Corrected” line is derived via a resampling procedure (bootstrap 2000 repetitions) to help add uncertainty to the calibration plot to get an idea of how this might perform out-of-sample and adjusts for optimistic (better than actual) calibration that is in fact an artifact of fitting a model to the data at hand. This line provides a measure of generalization (until we have new data to try the model on). When either of the two lines is above the “Ideal” line, this tells us the model underpredicts in that range of predicted probabilities. When either line is below the “Ideal” line, the model overpredicts in that range of predicted probabilities. All four models appear to be reasonably well calibrated based on the Bias-Corrected line closely following the Ideal line; nevertheless there is some underprediction at lower predicted probabilities of model 2 secondary outcome because the Bias-Corrected line is below the Ideal line around < 0.1 predicted probability and above it < 0.3 when compared to model 3 secondary outcome which performs better at < 0.3.

The ticks across the x-axis represent the frequency distribution (a rug plot) of the predicted probabilities. This is a way to see where there is sparsity in our predictions and where there is a relative abundance of predictions in a given area of predicted probabilities. Model 2 secondary outcome and model 3 secondary outcome present higher frequency along < 0.3 predicted probability, remarking the better calibration of model 3 secondary outcome when compared to model 2 secondary outcome.

4 Discussion

We developed model 3 and validated Maugeri model 1 and model 2 to estimate the probability of achieving functional improvement, as defined by the achievement of an M-FIM > 61 points (primary outcome), or a level of independence requiring no more than supervision according to the FIS system (secondary outcome) after stroke rehabilitation. Maugeri model 1 and model 2 were selected for external validation after considering the recently reported state of the art in functional predictors of rehabilitation outcomes (Stinear et al., 2019).

Model 3 incorporates age, time from stroke occurrence to inpatient rehabilitation admission, admission M-FIM score, neglect and aphasia. Aphasia was included as candidate predictor in both Maugeri models but it was not found significant, as opposed to our case. Supplementary Figure 12 presents the 95% CI of motor FIM scores at discharge for patients with aphasia and patients without aphasia by age range. Supplementary Figure 12 allows us to visualize differences in motor FIM at discharge between patients with aphasia and patients without aphasia, regardless of the age group that patients belong to. When considering primary outcome, model 3 (AUC = 0.894 (0.857–0.929)) outperformed Maugeri model 1 in both derivation (AUC = 0.883 (0.858–0.910)) and validation (AUC = 0.866 (0.840–0.892)) datasets.

In relation to secondary outcome, model 3 (AUC =0.845 (0.789–0.900)) performed almost equal to Maugeri model 2 validation (AUC = 0.850 (0.815–0.885)) but was outperformed by Maugeri model 2 derivation (AUC = 0.913 (0.884–0.942)). Nevertheless, the Maugeri derivation dataset presents the lowest proportion of participants achieving the secondary outcome (13.9%), meanwhile these proportions were higher in the Maugeri validation (18.3%) and in our dataset (20.8%).

When considering only participants presenting an initial severe motor impairment (defined by an admission M-FIM score of < 37 points), model 3 primary outcome (AUC = 0.862 (0.810–0.914)) outperformed both Maugeri model 1 derivation (AUC = 0.833 (0.796–0.870)) and Maugeri model 1 validation (AUC = 0.826 (0.790–0.861)). Similarly, when considering initial severe motor impairment, secondary outcome, model 3 (AUC = 0.865(0.790–0.940)) outperformed Maugeri model 2 derivation (AUC = 0.857 (0.805–0.909)) and Maugeri model 2 validation (AUC = 0.789 (0.735–0.844)).

When examining the predictors of MCID for motor-FIM score. (set as ≥25 points), model 3 (AUC = 0.769 (0.714–0.825)) outperformed Maugeri model 4 (presented in Scrutinio et al., Supplementary Materials) for both derivation dataset (AUC = 0.754 (0.718–0.790)) and validation dataset (AUC = 0.757 (0.726–0.789)).

A reported limitation of the study by Scrutinio et al. (also presented in Table 1) was in relation to LOS, quite longer than comparable populations in other countries. Mean or median LOS varies considerably, ranging from 17 days in the United States (Reistetter et al., 2010) to 35 in Canada (Grant et al., 2014), to 44–66 in Europe (De Wit et al., 2007). Therefore, as presented in Table 5, we conducted an additional analysis considering only participants with LOS≤66 (48.2% of participants) in which model 3 yielded an AUC = 0.901 (0.849–0.952) for primary outcome and AUC = 0.844 (0.776–0.913) for secondary outcome. Another reported limitation of the study by Scrutinio et al. was the prediction of outcomes only at discharge, without considering specific time-points. Therefore, we conducted an additional analysis for model 3 to predict primary outcome at 6 months (up to 8 months) after injury, in this case we were able to include 52.4% of the initial 710 participants and model 3 yielded an AUC of 0.938 (0.90–0.97).

As stratified medicine research examines whether a treatment works better or worse for some subgroups than others (Hingorani et al., 2013), the use of external datasets allows prediction model research to examine whether a model is more accurate for some subgroups than others. As extensively reported in previous research, two of the main predictors of stroke rehabilitation outcomes are age and stroke severity (Forlivesi et al., 2020). Both of them mainly characterize the subgroup of participants included in our study. In Scrutinio et al., 46.2% of participants in the derivation cohort and 41% in the validation cohort were aged > 75 years, the mean age of participants were 72 and 70 years respectively. Meanwhile, in our study only 1.8% of participants were aged > 75 years, 90.7% of participants were aged < 65 and the mean age of our participants was 52 years. Scrutinio et al. reported 42.1% of severe stroke participants (NIHSS > 10) in the derivation dataset, meanwhile in our case it is 73.1%, as presented in Supplementary Table 4.

Besides, associations between factors for ischemic or hemorrhagic stroke and clinical outcomes have been analyzed predominantly in older rather than younger patients (e.g. Scrutinio et al., 2017, Douiri et al., 2017 and de Ridder et al., 2018). Nevertheless, the incidence of any stroke in the young (18–44 years) has increased by 23% during the past ten years (Ekker et al., 2019). Stroke incidence rates have increased in adults aged 55 years and under in the United States (Ramirez et al., 2016) and in Europe (Tibaek et al., 2016).

Although the current recommendations for treatment of young and old patients with stroke are similar, the optimal management of young adult patients with stroke is unknown. (Ekker et al., 2018). They are usually not included in trials, and specific sub-analyses limited to young adult patients with stroke are usually not done (Ekker et al., 2018).

Following the onset of stroke, spontaneous mechanisms of recovery at the cellular, molecular, and systems levels occur. These mechanisms are often compensatory and incomplete since chronic disabilities persist for many individuals. Restorative therapies that rely on neuroplasticity have been developed to ultimately improve motor, sensory, language, and cognitive impairments (Cramer et al., 2011). Often, the mechanisms underlying these therapies rely on similar mechanisms as observed in spontaneous recovery (Cassidy et al., 2017).

5 Conclusions

We developed a new model and contributed to the validation of Maugeri model 1 and model 2 in a southern European setting (similar to Maugeri) but with specific characteristics, such as age and severity of participants. Our external validation and the development of the new model contribute to supporting researchers and clinicians with easy-to-use, accurate, and validated predictive tools that may be applied in rehabilitation research and stroke management.