Abstract
Introduction:
Sepsis is defined as a life-threatening organ dysfunction caused by a dysregulated host response to infection. Mortality rates are high, exceeding 50% in patients with septic shock. The sepsis severity score (SSS) was developed to determine the severity of sepsis and as a prognostic model. The aim of this study was to externally validate the SSS model.
Methods:
Calibration and discrimination of the SSS were retrospectively evaluated using data from a single-center sepsis registry.
Results:
Data from 156 septic patients were recorded; 56% of them had septic shock, 94% of patients required mechanical ventilation. The observed hospital mortality was 60.3%. The mean SSS value was 94.4 (95% CI 90.5–98.3). The SSS presented excellent discrimination with an area under the receiver operating characteristic curve (AUC) of 0.806 (95% CI 0.734–0.866). The pairwise comparison of APACHE II (AUC = 0.789; 95% CI 0.715–0.851) with SSS and 1st day SOFA (AUC = 0.75; 95% CI 0.673–0.817) with SSS revealed no significant differences in discrimination between the models. The calibration of the SSS was good with the Hosmer-Lemeshow goodness-of-fit H test 9.59, P > 0.05. Analyses of calibration curve show absence of accurate predictions in lower deciles of lower risk (2nd and 4th).
Conclusion:
The SSS demonstrated excellent discrimination. The calibration evaluation gave conflicting results; the H-L test result indicated a good calibration, while the visual analysis of the calibration curve suggested the opposite. The SSS requires further evaluation before it can be safely recommended as an outcome prediction model.
Introduction
Sepsis and septic shock, which are the most severe manifestations of acute infection, present a major public health challenge. The exact global burden of sepsis is unknown. Worldwide hospital-treated sepsis incidence was recently estimated at 31.5 million cases annually. This estimation, performed by the World Health Organization, is significantly underestimated, since it is based on data from seven developed countries and 87% of the world population was not included in this estimation. 1 However, regardless of the criteria used to identify sepsis from administrative data, various authors are reporting a rise in the incidence of sepsis.2 –6 A second common pattern that has emerged from sepsis epidemiological research is that there has been an annual decrease in mortality.7 –9 The absence of sepsis mortality data based on the Polish population makes benchmarking more difficult and hinders quality improvement. Lacking comparative data for observed hospital mortality, we decided to use a modern severity score to compare predicted outcomes with observed outcomes. This procedure, called external validation, could provide valuable results. If predictions based on the model are accurate, then the model’s prediction utility will be confirmed, while also confirming that severity-adjusted outcomes are similar to the outcomes that were observed in the dataset used to develop the severity score. This pragmatic way of obtaining benchmarking data has proven to be effective in view of the extreme variability in reported sepsis mortality.10 –15 The sepsis severity score (SSS) was developed using an international database created over a 5-year observation period on 27,836 patients (2005–2009). The SSS consists of 34 strictly defined predictor variables that were chosen while developing the model. 16 The aim of this study was to externally validate the SSS in a European intensive care unit.
Methods
The SSS was calculated retrospectively based on data prospectively entered into a sepsis registry database. The database contains the clinical and demographic data of patients treated in a 20-bed mixed intensive care unit which provides critical care service for the 1013-bed University Hospital in Wroclaw, Poland. When collecting data in the sepsis registry database, we used the SEPSIS-II definition, which was the most current at that time. 17 After the publication of the SEPSIS-III criteria, we retrospectively applied the new definitions. 18 In our database, all patients meeting the SEPSIS-II criteria also met the SEPSIS-III criteria (the minimum 1st-day SOFA score was 2 pts.). According to the SEPSIS-III definition, sepsis was defined as life-threatening organ dysfunction caused by a dysregulated host response to infection with a sequential organ failure assessment (SOFA) score of at least 2 pts. Septic shock was defined as sepsis with persistent hypotension requiring vasoactive agents to maintain a mean arterial pressure of at least 65 mmHg and having a serum lactate > 2 mmol/L. The study was approved by the local ethics committee from the Wroclaw Medical University, approval number KB 23/2015. The need for informed consent was waived because of the retrospective, observational nature of the study.
Inclusion criteria: all consecutive adult patients (over 18 years of age) diagnosed with sepsis were recorded in the sepsis registry database for the 12-month period from 1st of January to 31st of December 2014.
Exclusion criteria: patients who suffered from multiple sepsis episodes during a single ICU stay, had only the first episode analyzed in the study and data recorded for subsequent sepsis episodes were excluded from the final analysis.
All predictor variables required for the SSS calculation were available in the sepsis database or hospital medical records. All data necessary for the SSS calculations were available. The SSS was generated by summing the individual point values based on all patient clinical characteristics and whether patients had any goal-directed therapies. The SSS calculation formula from source publications with supplementary material was used.16,19 The SSS has not been previously validated or recalibrated for the Polish population. The SSS is calculated from 0 to 164 points; a higher score on the scale corresponds to a worse clinical condition of the patient.
In addition, variables for calculating the APACHE II and SOFA score were collected. The calculation was performed according to the original publications; there was no missing data for either score.20,21 APACHE II and SOFA scores have recently been externally validated in a single-center Polish study; recalibration was not performed. 22 Preliminary data from the Silesian registry of intensive care units showed a standardized mortality ratio for APACHE II of 0.93. No formal discrimination or calibration test was presented. 23 For the calculation of the SSS, APACHE II, and SOFA score the most abnormal values during the first 24h of ICU treatment were recorded. The type and number of organ dysfunctions were also registered. Organ dysfunction definitions provided by 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference were employed. 17
Quantitative data were summarized as a mean with 95% confidence intervals. Qualitative data were summarized as a percentage distribution and count. The normality of distribution was evaluated with the Shapiro-Wilk test. As the distributions were not normal, non-parametric tests were used for analysis. Quantitative data were compared with the Mann–Whitney U test. Qualitative data were compared with cross-tabulation tables and the Chi-square or Fisher’s exact test when appropriate. 24 The performance of severity scores was evaluated by two parameters: calibration and discrimination. Discrimination defines the score’s ability to distinguish between survivors and non-survivors. This parameter was evaluated with the area under the receiver operating characteristic curve (ROC), according to the method used by Hanley and McNeil. 25 The discrimination of the model ranged from acceptable, if 0.7 ⩽ AUC < 0.8, excellent, if 0.8 ⩽ AUC < 0.9, to outstanding, if AUC ⩾ 0.9. Discrimination below the AUROC of 0.7 was not acceptable by the severity model authors.26,27 We hypothesized that, in terms of discrimination, the SSS would outperform the 1st day SOFA. Based on the AUROC values from studies by Khwannimit et al. 28 (SSS AUROC 0.846) and Raith et al. 29 (1st day SOFA AUROC 0.75), we predicted the difference in the AUROC at 0.096 in favor of the SSS. At 80% power and 5% type I error, the minimum of sample size was 156 cases.
The DeLong method was used to perform a pairwise comparison of the ROC curves. 30 Calibration was referenced to the degree of agreement between the mortality predicted by the model and the observed mortality across all risk strata. The calibration was evaluated by the Hosmer-Lemeshow (H-L) goodness-of-fit test. 31 The calibration curve for the SSS was created by comparing predicted mortality versus observed morality in 10% increments. Medcalc software was used for power calculation and Delong’s test calculation. Microsoft Excel was used for the Hosmer-Lemeshow test calculation and STATISTICA 12 (StatSoft, Inc.) was used to perform other statistical analyses.
Results
Detailed patient characteristics stratified by hospital outcome are presented in Table 1. Data from 156 septic patients were recorded in the sepsis registry for 12 months of observation. Among these patients, 62 survived and 94 died. The observed ICU mortality was 47.4% and the observed hospital mortality was 60.3%. Sepsis and septic shock were diagnosed in 69 (44%) and 87 (56%) cases, respectively. Pneumonia (49%) and intraabdominal infection (35%) were the two most frequent causes of sepsis and septic shock, followed by urinary tract infection (7%), intravascular catheter associated infections (2%), device associated infection (2%), and other infections (15%). Sources of ICU admissions were as follows: general wards (48%), operating theater (38%), Emergency Department (7%), transfer from other hospital (6%), and high dependency unit (1%). The clinical condition of the patients was already critical on admission to the ICU, with a mean value of the SOFA score of 11.9 points in non-survivors and 8.1 points in survivors. More than half the patients (51%) had failure of four or more organs during the first 24 h of the ICU stay. Hospital mortality in this subgroup was very high (81%). 20% of patients had three failing organs on admission and mortality in this subgroup was 50%. In the subgroup of patients with two failing organs (19%) the mortality rate was 31% and in patients with one failing organ 8%. The most common type of organ failure was respiratory (87%), followed by cardiovascular (86%), and then renal (57%), metabolic (38%), hematologic (26%), liver (22%), and CNS (23%). A majority of patients (94%) required mechanical ventilation. 89% of patients received vasopressor infusion, with noradrenalin being the most commonly used drug (88%); 29% of patients also received adrenaline infusion. Renal replacement therapy was utilized in 44% of patients. 51% of patients received low dose corticosteroids (up to 200 mg of hydrocortisone / 24 h, 50 mg bolus every 6 h). Blood products were given to 51% of patients.
Patient characteristics on admission to the ICU.
SSS: sepsis severity score; SOFA: sequential organ failure assessment; PBC: packed blood cells; FFP: fresh frozen plasma; LMW: low-molecular weight.
The SSS was calculated based on data from 156 patients and the usefulness of the score for predicting mortality was evaluated based on the whole group. The mean SSS was 94.4 (95% CI 90.5–98.3), the mean 1st day SOFA was 10.4 (95% CI 9.7–11.1), the mean APACHE II was 21.5 (95% CI 20.1–22.8). The detailed group characteristics using the SSS variables are presented in Table 2. The mean SSS value in survivors was 80.5 points and was significantly lower (p < .001) than in non-survivors (103 points). The group was uniform in terms of geographical region, and all patients came from Europe. Factors present more frequently in the group of non-survivors were dysfunction of the cardiovascular system, unresponsive hypotension, and impaired organ perfusion, indicated by an elevation in the lactate level.
Sepsis severity score parameters by hospital outcome.
The discrimination of the score’s ability to distinguish between survivors and non-survivors was evaluated with the area under the ROC curve. Using the DeLong method, a pairwise comparison of the ROC curves was performed. There was no significant difference in predictive validity between the SSS (AUROC = 0.806; 95% CI, 0.734–0.866) and the 1st day SOFA (AUROC = 0.750; 95% CI, 0.673–0.817) with a difference in the AUROC of 0.056 (95% CI, –0.019–0.131, P = 0.142). There was also no difference in predictive validity between the SSS and APACHE II (AUROC = 0.789; 95% CI, 0.715–0.851) with the difference in the AUROC of 0.017 (95% CI, –0.055–0.09, P = 0.641). ROC curves for prediction of the likelihood of death in patients with sepsis and septic shock are presented in Figure 1 and the cut-off points are presented in Table 3. The Hosmer-Lemeshow H-L statistic was 9.59, with a P value greater than 0.05 (P = 0.29). A calibration curve was also created (Figure 2).
Receiver operating characteristic curves for prediction of the likelihood of death in patients with sepsis and septic shock. Sepsis severity score, APACHE II score, and 1st day SOFA score had areas under the ROC curve of 0.8060, 0.789, and 0.750, respectively.
SSS, APACHE II, and 1st day SOFA discrimination for hospital mortality.
The SSS calibration curve. The comparison of observed versus predicted mortality in the deciles of predicted mortality.
Discussion
To our knowledge, this study is the first external validation of the SSS for predicting hospital mortality in septic patients performed in a European ICU. Visual analysis of the calibration curve (Figure 2) might suggest an absence of accurate predictions in the 3rd and 4th risk deciles. However, the number of patients in these deciles was low, that is, 7 and 6 respectively. An analysis of the results in the 5th, 6th, 7th, and 9th risk deciles, with a much higher number of patients, showed more accurate predictions. These findings were difficult to interpret, because calibration curve analyses are not accompanied by any information of statistical significance; this is an intrinsic drawback of calibration curve analysis. 32 Therefore, we performed the Hosmer-Lemeshow test, which showed that in our group of patients the difference in predicted and observed mortality was not statistically significant across 10 risk groups. The obtained H-L test result indicated good calibration of the SSS model in the studied population. There are, however, serious limitations to this conclusion. First, the H-L test tends to confirm good calibration in a small sample size. Second, all validation tests are sensitive to sample size, which was confirmed using a random sampling procedure from a large Dutch ICU database. 33 Therefore, our result should be interpreted with caution and as an argument for further evaluation of the SSS model, not as a definitive confirmation of good calibration.
The discrimination of the SSS model, assessed by the area under the ROC curve, was above 0.8, which is considered excellent. The 1st day SOFA showed lower discrimination compared to the SSS model, but the difference was not significant. The SOFA score is considered to be a good tool for monitoring a patient’s clinical condition in sepsis; however, there is no clear consensus on how to use SOFA for predicting mortality. Admission SOFA, maximal SOFA, mean SOFA, delta SOFA, and other solutions have been studied, but no clear recommendations have been made yet. 34 Therefore, we believe that, if our results are reproducible, the SSS model could be applied to patients with sepsis and septic shock.
In our study, the APACHE II based model presented with discrimination, slightly lower than the SSS. The extensive use of the APACHE II scale by ICU researchers makes it possible to compare the severity of the clinical status of patients reported for large, multicentre studies with local data and single-center studies. However, due to differences in the case-mix, the APACHE II score has lost its ability to accurately predict mortality in the modern ICU setting.35 –38 As demonstrated for other general severity scores, a first-level recalibration could be the solution to this issue.39,40 Employing severity scores designed specifically for sepsis patients was suggested as another solution to predict outcome in a narrow group of patients.16,41 In this study, we evaluated a second solution using the SSS.
Predicting mortality risk in ICU patients has numerous applications. It is an essential part of quality benchmarking. Risk-adjusted mortality is the most commonly used ICU quality-of-care indicator utilized by quality improvement initiatives in many European countries such as Austria, the Netherlands, Scotland, Sweden, and the United Kingdom. Mortality risk assessment is also used in clinical trials for case-mix comparison and description. Predicting outcome is pragmatically used to allocate resources and assess individual ICU performance.42,43
The SSS was previously described by Osborne et al. as a severity model for predicting outcome in septic patients. 16 The model in this study was based on the Surviving Sepsis Campaign database, containing records from 23,438 ICU patients from Europe, North America, and South America. The model was developed based on 90% of the dataset (n = 21085, random sample). The final model was comprised of 34 categorical variables. The validation group included data from the remaining 10% of cases. The reported AUC value was 0.701 in the validation group and 0.736 in the development group and the calibration was determined to be “good” (H-L goodness-of-fit, P = .58).
Recently, Khwannimit et al. published results of a study comparing the performance of the SSS with the APACHE II-IV, Simplified Acute Physiology Score 2 (SAPS 2), and SAPS 3 scores for predicting hospital outcome in septic patients. It was a retrospective analysis of the performance of the SSS, using a dataset collected prospectively from an Asian ICU population between 2011 and 2015. The database contained the characteristics of 913 medical patients more than 15 years old with sepsis or septic shock. Sepsis and septic shock were defined following the criteria of the Third International Consensus Definitions for sepsis and septic shock (Sepsis-3). The reported hospital mortality was 43.9% and the mean SSS value was 82 points. Calibration of the model was evaluated with the Hosmer-Lemeshow goodness-of-fit H test and all analyzed scores (SSS, APACHE II-IV, SAPS 2, SAPS 3) failed to accurately predict mortality risk, with the P value for H-L < 0.001 for all tested scores. However, the SSS was able to correctly predict mortality in the subgroup of patients with severe sepsis (P value for H-L test = 0.1); Sepsis-2 criteria were employed to select this subgroup. 17 This result might suggest that the failure to predict mortality with the SSS in Khwannimit’s study could be linked to differences in the sepsis criteria. The Sepsis-2 criteria were used by Osborn in the first SSS development dataset and Khwannimit used Sepsis-3 criteria to externally validate the model. Differences in datasets are also visible, as the SSS lacks a point value for Asia as a geographic region. This precluded the use of some SSS variables in the Khwannimit’s study and could have influenced the analysis of calibration and discrimination. 28
The performance of the SSS was also evaluated in a cohort of patients presented to the Emergency Department who were presumed to have an infection. To predict mortality, Williams et al. used models based on the APACHE II and SAPS II scores and compared them with the SSS based model. 44 The discrimination of the SSS based model, reported by the authors, was good (AUC = 0.82), but the calibration of the SSS model was poor. There might be two reasons for the poor calibration result. Williams et al. enrolled 8871 emergency patients, but only 15% (1325 patients) were diagnosed with severe sepsis or septic shock. It should be noted that this discrepancy between the number of patients enrolled and the number of those diagnosed with severe sepsis or septic shock was significant in Williams’s study. For comparison, the SSS was first developed using a large cohort (n = 23,438) who were all diagnosed with severe sepsis or septic shock. A second disparity in Williams’s study was related to the applied endpoint. The SSS was originally developed to predict hospital mortality, while Williams used 30-day mortality as the endpoint. The differences in group characteristics and endpoints were probably responsible for the poor calibration of the SSS model in Williams et al. 44
The high hospital mortality observed in our study (60.3%) is in contrast to the mortality results reported in some sepsis studies. Recently, researchers involved in the ANDROMEDA-SHOCK trial reported hospital mortality of 34.9% in the septic shock group. 45 Mortality in the ADRENAL trial was similarly low; the 90-day survival rate in septic shock patients that received 200 mg of hydrocortisone was 62.1%. 46 In another study, Annane et al. 47 reported a 90-day mortality of septic shock patients who had received hydrocortisone and fludrocortisone of 43%. However, the inclusion and exclusion criteria used in these randomized clinical trials were different, and for this reason the mortality rates cannot be compared with a registry-based observation. For example, a nationwide registry database collected in Germany allowed for a more pragmatic comparison. 48 In this database, the hospital mortality of patients diagnosed with septic shock was 57% in 2014 and was comparable to the results obtained in our study. The use of the SSS allowed us to compare the severity of the clinical condition of patients in this study population with the cohort assessed for the development of the SSS model. In an international cohort used by Osborn et al., the mean SSS was 56.3 points. The mean SSS observed in our study was much higher and amounted to 94.4 points (95% CI 90.5–98.3), indicating a significantly worse clinical condition of patients at the time of ICU admission. The high percentage of patients admitted with septic shock (56%) also reflected the serious clinical condition of the patients. However, the data obtained do not give a clear explanation as to why such a high mortality was observed. Recently, Weigl et al., 49 after analyzing the national Polish ICU outcome database, reported a high observed ICU mortality of 43.5%. Grochla et al. 50 hypothesized an explanation that the absence of clear futile therapy legislation, the unpopularity of advanced directives, the insufficiency of rapid response team implementation, the fear of legal consequences in cases of refusing ICU admission, the extreme pressure on ICU admissions due to the poor development of intermediate care, and the lack of funding for patients with a lower TISS-28 score (< 19 pts.) are responsible for this high mortality rate. In our opinion, those factors were, to varying degrees, responsible for the high hospital mortality also observed in this study.
There are limitations of our study. This is a single-center observation, with a small sample size, in a hospital with a relatively low availability of ICU beds. The type of admission (usually the transfer of a patient from another hospital ward), ICU discharge criteria, case-mix, and the standard of care may be different than in better financed healthcare systems. The excellent discrimination obtained in our study was similar to that obtained in other SSS external validation studies.51,28 The calibration evaluation gave conflicting results; the H-L test result indicated a good calibration, while the visual analysis of the calibration curve suggested the opposite. Therefore, we cannot definitely conclude that the model has been adequately calibrated. Our observation is in line with the consensus that the performance of severity scores in predicting the outcome is poor outside of the cohort used for the development of the model.52,53 Our findings could also be affected by the susceptibility of the Hosmer-Lemeshow test to confirm good calibration for smaller sample sizes. 54 Therefore, our result should be interpreted with caution and as an argument for further evaluation of the SSS model, not as a definitive confirmation of good calibration.
Conclusion
The SSS demonstrated excellent discrimination. Calibration examination results were conflicting, with HL-test confirming good calibration and calibration curve analysis suggesting the opposite. The SSS requires further evaluation, before it can be safely recommended as an outcome of prediction model.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical approval
Ethical approval for this study was obtained from Bioethics Committee, Wrocław Medical University, Approval id: KB 23/2015
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors received funding support from Medical University Piastów Slaskich in Wroclaw (Grant id PBmn.149).
Informed consent
The need for informed consent was waived because of the retrospective, observational retrospective nature of the study.
