Comorbidity Scores and Machine Learning Methods Can Improve Risk Assessment in Radical Cystectomy for Bladder Cancer

INTRODUCTION

In 2020, more than 500,000 patients were diagnosed with bladder cancer and > 200,000 bladder cancer related deaths worldwide were reported [1]. For muscle-invasive bladder cancer (MIBC), (neo-)adjuvant chemotherapy and radical cystectomy (RC) is recommended as the curative therapy of choice in non-metastatic patients [2]. The operative technique of RC has significantly improved over the last decades so that today a 30-day mortality rate < 3.5–5% can be achieved [3]. However, the 90-day complication rate still is reported to be higher than 50%, even in high-volume hospitals [4]. Due to these relevant risks despite surgical improvement, it is necessary to conduct a thorough preoperative risk evaluation. Factors adversely influencing the early postoperative morbidity and mortality seem to be comorbidity, age and lower hospital volume among others [3, 5].

Comorbidity indices such as the Charlson Comorbidity Index (CCI) or its age-adjusted version (aCCI) [6, 7], Elixhauser Index (EI) [8] and the American Society of Anesthesiology classification system (ASA) [9] have been successfully investigated for their prediction of mortality after RC. These scores proved to be significant predictors of mortality after radical cystectomy [10, 11]. Recently, Gagne et al developed the combined comorbidity index (GCI), a combination of the aCCI and van Walraven’s modification of the EI [12]. The GCI performed better than the individual indices in the prediction of short- and long-term mortality in general emergency patients > 65 years [13]. However, its use has not been validated in the prediction of postoperative mortality after RC.

For the preoperative risk evaluation of postoperative morbidity, validated instruments are lacking. The use of comorbidity indices to predict complications showed only limited predictive value [14]. Further, risk calculators from general surgery, such as the National Surgical Quality Improvement Program risk calculator, have not shown a sufficient performance in RC patients [15]. Thus, a thorough preoperative risk assessment remains a challenging task in clinical practice. This leads to an underutilization of RC with MIBC, especially in comorbid or elderly patients [16, 17].

Hence, we evaluated the ability of the GCI in comparison to the aCCI, EI and ASA in the prediction of postoperative mortality after RC at a German high-volume tertiary referral university hospital. Further, we evaluated the performance of the comorbidity indices and state-of-the-art machine learning (ML) models in predicting complications after RC.

MATERIAL AND METHODS

RESULTS

Comorbidity indices to predict mortality and severe complications

The univariable logistic regression analysis showed significant results for the prediction of 90-day mortality for the ASA (χ² = 25.8, p < 0.001), the aCCI (χ² = 21.1, p < 0.001) and the GCI (χ² = 2.4, p = 0.04). AUROCs demonstrated good performance for the ASA (0.77) and for the aCCI (0.81) while the CGI’s AUROC (0.63) revealed only moderate performance, as demonstrated in Fig. 1A. EI did not show a significant prediction (χ² = 1.87, p = 0.17, AUROC = 0.58).

OPEN IN VIEWER

Fig. 1

ROC curves for the index prediction of 90-day mortality and severe 30-day complications. The figure shows the individual ROC curves and corresponding AUROCs of the four comorbidity indices for the respective predictions. A) Prediction of 90-day mortality by the four comorbidity indices in univariable logistic regression analysis. B) Prediction of severe 30-day complications (CDC IIIb-V) by the four comorbidity indices in univariable logistic regression analysis.

The prediction of severe 30-day complications showed significant results for the ASA (χ² = 15.7, p < 0.001), the aCCI (χ² = 8.8, p < 0.01) and the GCI (χ² = 4.6, p = 0.03). However, AUROCs showed only moderate to poor performance, 0.63, 0.60 and 0.57, respectively, as illustrated in Fig. 1B. Again, EI was no significant predictor of severe 30-day complications (χ² = 0.93, p = 0.3; AUROC = 0.53).

Machine Learning models to predict morbidity

Different ML models were developed to predict severe complications after RC. Due to the low number of deaths, the development of ML models to predict mortality was not considered expedient. The ASA as the best performing comorbidity index, body mass index, age and sex were used as variables. In 10-fold cross validation, all models showed an AUROC > 0.7 (Table S.1 in the supplementary). Interestingly, the performance of the NN, RF and DT dropped significantly when evaluated on the hold out test set (n = 98) while GNB and MLR showed slightly improved results. On the test set, the GNB showed the highest AUROC (0.79, χ² = 7.3, p < 0.01) followed by the MLR (AUROC 0.76, χ² = 9.0, p < 0.01), RF (AUROC 0.75, χ² = 7.43, p < 0.01) and NN (AUROC 0.74, χ² = 4.76, p = 0.03) as depicted in Fig. 2. The worst performance was seen for the DT (AUROC 0.58, χ² = 0.89, p = 0.57). The hyperparameters chosen for the different models and the results on the training set are stated in Table S.1 in the supplementary.

OPEN IN VIEWER

Fig. 2

ROC curves for the prediction of 30-day severe complications by the machine learning models. The figure shows the individual ROC curves and corresponding AUROCs of the five ML models for the prediction of severe 30-day complications (CDC IIIb-V) on the hold-out test set.

Interestingly, when replacing the comorbidity index with individual comorbidities as variables for the ML models, no outperformance was achieved.

DISCUSSION

The present study investigated the value of four comorbidity indices in their prediction of postoperative morbidity and mortality after RC. A good performance was shown for the prediction of mortality. However, prediction of complications was worse but can be improved using easily obtainable preoperative variables in state-of-the-art ML models.

The predictions of four different indices were evaluated in this study. The aCCI, EI and CGI are established indices that objectively indicate the severity of comorbidity by means of a sum score but are usually not collected in routine clinical practice. The ASA, on the other hand, is a simple and quick semi-subjective categorization score that is used almost ubiquitously. The indices showed their strength in predicting mortality. The in-hospital mortality in our cohort was 2.6% while the 30-day and 90-day mortality were 1.8% and 4.8% respectively. These results are in line with the mortality rate reported by other high-volume centers [3]. The best prediction of mortality was provided by the aCCI and the ASA followed by the GCI while the EI showed no significant prediction. A good performance of the CCI and aCCI in mortality prediction is well documented in the current literature. Froehner et al. reported an AUROC of 0.74 using the CCI to predict 90-day mortality in a larger cohort [25]. Prayer Galetti et al. showed that the aCCI as a significant predictor of mortality in elderly patients undergoing RC [11]. The AUROC of 0.81 in our analysis again underlines the predictive capability of the aCCI in RC candidates. Comparable results are also shown for the ASA. Our study revealed an AUROC of 0.77 which is concordant to recent studies investigating the predictive ability of the ASA [14]. Boorjian et al. found the ASA to be a strong risk factor of 90-day mortality after RC [10]. They additionally found the EI to be a risk factor. However, this could not be confirmed in our analysis. Regarding the CGI, a significant result was also shown, but the AUROC was worse than that of the ASA and aCCI. In our cohort, there was no advantage to be gained from using this index.

The index-based prediction of severe complications was also evaluated. Severe complications were defined as CDC≥IIIb. Although CDC IIIa complications are often included in the definition of severe complications, they were not included in our analysis. These complications, such as nephrostomy insertion for hydronephrosis, usually have a fundamentally different etiology and significantly less influence on the postoperative outcome than complications CDC≥IIIb. The preoperative prediction of CDC IIIb-V complication was therefore considered more clinically relevant. In the univariable analyses, the aCCI, ASA and GCI proved to be significant risk factors of severe complications which occurred in 17%. This again is in line with recent works. Roghman et al. showed a comparable complication rate [26]. Furthermore, they also identified the aCCI and ASA as risk factors for severe complications. However, the use of the indices as stand-alone models for the prognostication of complications seems to be not sufficient since AUROCs were < 0.70 for all indices in our study. A significant improvement could be achieved by introducing the ASA, sex, BMI and age as variables in different ML models. Specifically, the GNB, the MLR, the RF and the NN showed an AUROC above 0.70 on the hold-out test set. A similar approach was used by Taylor et al. [27]. They used ML models to predict adverse events after RC including multiple variables. They reported an AUROC of 0.67 for the prediction of serious complications. However, a direct comparison to our results seems difficult since a non-standardized definition of serious complications was used.

While the prognostication of complications after RC remains challenging, our results suggests that the combination of easily obtainable risk factors in combination with the ASA in ML models can provide a reasonable estimation of the risk of severe complications. Due to the monocenter design of our study, it can be assumed that an application of the presented models in other institutions would be associated with a drop in performance. Thus, for the development of a model that aims to be implemented in clinical practice, a multicenter cohort should be used. The most promising models were the GNB, the MLR, the RF and NN. All models showed stable results when evaluated on the hold-out test set while the DT model showed a drop in performance. Especially MLR has the advantage of being widely accepted, easy to interpret and implement in manually applicable risk models [28]. Therefore, it might be the most promising model for clinical implementation in this setting with a limited number of variables included. When a larger number of variables is to be analyzed, the more complex models such as GNB, NN and RF might show an advantage. The calculation of the risk score of these models is computer-based, so that an online application offers a good solution for easy access.

In recent years, frailty has emerged as another predictor of morbidity and mortality. In contrast to comorbidity indices, frailty focuses more on the actual physical performance and less on the pre-existing conditions [29]. Recent studies showed promising results for this concept in RC patients, especially since it can add additional value to the use of comorbidity scores [30, 31]. It will be exciting to see to what extent frailty establishes itself as a preoperative screening tool before RC or as a part of a multivariable model. As these scores usually need to be collected prospectively, an evaluation in this regard was not possible in our study but represents a highly interesting aspect for the further development and improvement of the models.

Of note, only patients undergoing open RC were included in our study. Overall, current studies suggest comparable mortality and severe complication rates for robotic-assisted RC [32]. However, blood loss, transfusion rate and length of stay have been reported to be lower when robotic-assisted RC is used, while open RC has significant shorter operating times [33]. The extent to which this reduction may improve postoperative morbidity and mortality in specific patient groups, such as cardiac high-risk patients, remains to be explored.

LIMITATIONS

First, although the data was mainly prospectively collected, naturally the power of the retrospective analysis of such database is limited. Second, due to the low absolute number of deaths, the mortality analyses must be confirmed in larger studies. Third, the ML models were not validated on an external dataset. To reduce the risk of overfitting, the cohort was split into a training and hold-out test set and results showed robustness of the model in our cohort. Still, only external validation can assess the generalizability of the models.

CONCLUSION

The ASA and the aCCI can help to assess the risk of mortality after RC with good accuracy. However, their prediction of postoperative severe complications is limited. State-of-the-art ML models show an improved prediction and thus might help to provide a clinically useful prognostication of severe complications.