Abstract
Background:
Septic patients with heart failure (HF) have higher mortality and poorer prognosis than patients with either disease alone. Currently, no tool exists for predicting survival rate in such patients.
Objective:
This study aimed to develop an interpretable prediction model to predict survival rate for septic patients with HF.
Methods:
Severe septic patients with HF were recruited from the MIMIC-IV database (as training and internal validation cohorts) as well as from the MIMIC-III database (as external validation cohorts). Four models including Deep Learning Survival (DeepSurv) were constructed and evaluated. Furthermore, Shapley Additive Explanations (SHAP) method was employed to explain the DeepSurv model.
Results:
A total of 11,778 patients were included and 22 features were identified to construct the models. Among the 4 models, the DeepSurv model had the highest area under the curve (AUC) values with an AUC of 0.851 (internal) and 0.801 (external) and C-index of 0.8329 (internal) and 0.7816 (external). The mean cumulative/dynamic AUC values exceeded 0.85 in both internal and external validations. The Integrated Brier Score values were well below 0.25, at 0.068 and 0.093, respectively. Furthermore, the Decision Curve Analysis showed that the DeepSurv model achieved favorable net benefit. The SHAP method further confirmed the reliability of the DeepSurv model.
Conclusion:
Our DeepSurv model was the most comprehensive interpretable prediction model specifically developed and validated for septic critically ill patients with HF. It demonstrated good model performance in predicting the 28-day survival rate of such patients and will provide valuable decision support for clinicians.
Introduction
Sepsis is one of the most common diseases in the intensive care unit (ICU), with a mortality rate as high as 42%.1,2 Heart failure (HF), a common cardiovascular disease, often has a poor prognosis in patients admitted to the ICU, with a one-year mortality rate reaching 46.5%.3,4 It was indicated that HF was a common comorbidity in patients with sepsis, and septic patients with HF may face a poorer prognosis and higher mortality than patients with either disease alone.5,6 Therefore, a tool with good sensitivity and accuracy is urgently needed to predict the survival rates of septic patients with HF.
Although some scoring systems such as the Sequential Organ Failure Assessment (SOFA) and the Acute Physiology and Chronic Health Evaluation II (APACHE II) score already exist for assessing the condition of ICU patients, these systems have limited predictive capability and lack specificity and sensitivity in evaluating the prognosis of critically ill septic patients with HF.7,8 Additionally, while several new models were developed to predict the prognosis of patients with sepsis and HF, they were only designed for single diseases and often fail to fully account for the complex physiological states of patients with comorbidities.9,10 Therefore, it is necessary to develop a comprehensive survival prediction model specifically for septic patients with HF, particularly those in critical condition, to better guide clinical practice.
Based on above background, this study aims to use advanced machine learning (ML) and deep learning (DL) algorithms, combined with the open-source Medical Information Mart for Intensive Care III database version 1.4 (MIMIC-III v1.4) and MIMIC-IV v2.2, to construct and validate a predictive model for the survival rates of ICU septic patients with HF. Furthermore, the Shapley Additive Explanations (SHAP) method is also used to analyze the contribution of each prognostic factor to the prediction, providing a reasonable explanation for the predictive outcomes. Through these methods, we hope to provide more precise predictive tools for septic patients with HF.
Methods
Data sources
We extracted data from two retrospective, multilevel clinical databases, MIMIC-III and MIMIC-IV, both of which were open-access ICU databases jointly managed by the Massachusetts Institute of Technology (MIT) and Beth Israel Deaconess Medical Center (BIDMC).11,12 MIMIC-III contains data on over 40,000 critically ill patients from BIDMC, while MIMIC-IV, an update to MIMIC-III, includes data on more than 50,000 patients.13,14
Since both databases protect and anonymize personal details of patients, the requirement for obtaining patients’ informed consent was waived, and no ethical reviews were required. To gain access to the database, the authors passed the Human Research Participant Protection Examination and received certificates with numbers 56318337 and 56998638.
Study population and definition
Patients with sepsis were diagnosed according to the Third International Consensus definition of sepsis (Sepsis 3.0), which included: (1) existing evidence of suspected or confirmed infection; (2) SOFA score ≥ 2. 15 Patients with HF were identified using ICD-9th and ICD-10th codes among all eligible patients with sepsis. For patients with multiple hospital admissions, only data from their first admission were included. Besides, exclusion criteria were as follows: (1) patients aged less than 18 years; (2) patients with an ICU stay of less than 24 h.
Data collection
After identifying eligible subjects, we used their hadm_id and icustay_id parameters to extract information from the corresponding tables, including 58 variables such as demographic information, vital signs, severity scoring system scores, laboratory data, comorbidities, and drug use. The vital signs extracted were the average values during the first 24 h of the ICU stay, while the laboratory parameters were the parameters obtained for the first time after admission to the ICU.
Data preprocessing and feature selection
In this study, the missing data were < 30% and were imputed by multiple imputation (see supplementary material Table S1). The Least Absolute Shrinkage and Selection Operator (LASSO) analysis was used to identify potential survival-related variables in order to reduce overfitting. 16 Furthermore, to mitigate the impact of the absolute value range of different features on their importance and to accelerate the convergence of ML algorithms, we selected the formula x* = (x - min) * 0.99 / (max - min) for normalizing continuous variables. 17
Model development and validation
We developed 4 models to predict the survival of septic patients with HF: the Cox Proportional-Hazards model (CoxPH), the Deep Learning Survival model (DeepSurv), Random Survival Forests (RSF) and the Neural Multitask Logistic Regression model (NMTLR). Additionally, to validate their predictive performance, we compared these models with those based on three commonly used severity scoring systems: SOFA, the Simplified Acute Physiology Score II (SAPS II) and the Glasgow Coma Scale (GCS). To further enhance the stability and robustness of our models, we optimized their hyperparameters using the random search method.
The final data of MIMIC-IV were randomly divided into training and testing datasets at a ratio of 8:2. External validation was performed using MIMIC-III data. Model performance was evaluated using the concordance index (C-index), area under the receiver operating characteristics curve (AUC) and mean cumulative/dynamic AUC score, with AUC serving as the primary performance metric. In addition, the accuracy, precision, recall, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score were also calculated. Furthermore, calibration was assessed using Brier scores (BS) and calibration plots. BS measures the mean squared error between actual survival rates and estimated probabilities. A model with a BS of less than 0.25 is considered useful in practice. 18 To further evaluate the overall performance of the model across all available periods, we integrated these BS values to obtain the integrated Brier score (IBS). Finally, decision curve analysis (DCA) was conducted to evaluate the clinical application value and net benefit at different threshold probabilities of models. Based on these results, we selected the best model for the follow-up study.
Considering the imbalance between the death group and the surviving group in the training set, we ultimately chose random oversampling (ROS) to correct the imbalance. 19 Subsequently, we further assessed the performance of these models constructed on the balanced dataset and compared the results with those obtained from the unbalanced dataset to further validate the robustness of the models.
Permutation importance and Shapley Additive Explanations (SHAP) analyses were used to explore the interpretability of the predictive model in this study. Permutation importance was calculated by measuring the increase in prediction error after randomly shuffling each feature. 20 The latter utilizes the classic Shapley value from game theory to assign an importance score to each feature, ranking them based on their contribution to the final prediction, and also provides additional information regarding the prediction direction of these features. 21
Statistical analysis
Continuous variables were described using mean ± standard deviation or median and interquartile range (IQR), depending on whether they conformed to a normal distribution. Categorical variables were summarized through frequency counts and percentages. P values of less than 0.05 were considered statistically significant. Data preprocessing was conducted using R (version 4.3.2), and Python (3.7) was used to implement the models.
Results
Baseline characteristics
According to the exclusion criteria (Figure 1), we identified 7555 and 4223 patients from the MIMIC-IV and MIMIC-III databases, respectively. In the MIMIC-IV database, 1316 patients died and 6239 survived within 28 days. Table 1 summarized the comparison of baseline clinical characteristics within 28 days between non-survivors and survivors. Patients in both groups were elderly, with patients in the non-survivor group being older than those in the survivor group. Laboratory findings showed that the non-survivor group had more severe anemia, electrolyte imbalances, impaired coagulation functions, and renal impairment. Furthermore, compared to the survivor group, the non-survivor group had a higher proportion of patients with atrial fibrillation (AF) and acute kidney injury (AKI), a higher proportion of patients receiving continuous renal replacement therapy (CRRT) and mechanical ventilation (MV), and lower use rates of β-blockers and angiotensin-converting enzyme inhibitor (ACEI) or angiotensin receptor blockers (ARB).
Demographic and clinical characteristics at baseline.
Demographic and clinical characteristics at baseline.
Abbreviations: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; AKI: acute kidney injury; BMI, body mass index; BUN, blood urea nitrogen; DBP, diastolic blood pressure; GCS, Glasgow Coma Scale; HR, heart rate; H2RA, histamine H2 receptor antagonist; INR, international normalized ratio; MBP, mean blood pressure; MV, mechanical ventilation; PT, prothrombin time; PTT, partial thromboplastin time; RR, respiratory rate; SAPS II, simplified acute physiology score II; SOFA, sequential organ failure assessment score; SBP, systolic blood pressure; Spo2, peripheral capillary oxygen saturation; WBC, white blood cell.
Flowchart of patient selection. Abbreviations: HF, heart failure; ICU, intensive care unit.
In the LASSO method, the penalty on the β-coefficients was controlled by the tuning parameter λ (λ = 0.008433489; lambda.1se; Figure 2). Twenty-one features with non-zero coefficients were selected, including demographic information (age, ethnicity and weight), vital signs (heart rate (HR)), severity scoring system scores (SAPS-II and GCS), laboratory data (anion gap, blood urea nitrogen (BUN), chloride, potassium, partial thromboplastin time (PTT), and lactate), comorbidities (essential hypertension, AF, and AKI), drug use (β-blocker and ACEI/ARB), and others (CRRT, MV, hospital length of stay (LOS) and ICU LOS). Additionally, it was recently indicated that there is a significant correlation between the use of histamine H2 receptor antagonists (H2RA) and reduced mortality in critically ill patients with HF.22–24 In this regard, H2RA was also included and eventually 22 variables were selected for model construction.
Feature selection based on LASSO regression analysis. (A) Cross-validation plot for the penalty term. The dashed lines represent the lambda.min and lambda.1se. (B) Plots for the LASSO regression coefficients over different values of the penalty parameter. The vertical dashed lines correspond to the lambda.min and lambda.1se from the cross-validation. Abbreviations: LASSO, Least Absolute Shrinkage and Selection Operator.
We constructed 4 prediction models: CoxPH, RSF, DeepSurv, and NMTLR, along with 3 scoring system models based on SOFA, SAPS II, and GCS scores. In internal validation cohort, the DeepSurv model had a C-index of 0.8329, which was higher than the other seven models, including CoxPH (0.8246), RSF (0.8252), NMTLR (0.8016), SOFA (0.5652), SAPS II (0.6867) and GCS (0.6315). In external validation cohort, the C-index of the DeepSurv model was also higher at 0.7816, compared to CoxPH (0.7785), RSF (0.7554), NMTLR (0.7071) SOFA (0.6122), SAPS II (0.6701) and GCS (0.5461). The ROC curves of the 7 models for 28-day survival prediction were presented in Figure 3. Notably, the DeepSurv model outperformed the other models in both internal and external validation cohort, with AUCs of 0.851 and 0.801, respectively. This indicated that it had higher sensitivity and specificity in distinguishing patient survival status. Additionally, the time-dependent AUC values for the DeepSurv model in both internal and external validation cohort exceeded 0.85 (supplementary material Figure S1), indicating its robust capability in predicting both long-term and short-term survival outcomes. The confusion matrices and other relevant performance metrics for the survival predictions of the 4 models were shown in Supplementary Figure S2 and Supplementary Table S2, respectively, among which the DeepSurv model exhibited relatively small performance variation between the internal and external validation sets and had higher stability compared with other models.
The receiver operating curves (ROC) for CoxPH, DeepSurv, RSF, NMTLR, SOFA, SAPSII and GCS models. Abbreviations: CoxPH, Cox Proportional-Hazards model; DeepSurv, Deep Learning Survival model; RSF, Random Survival Forests model; NMTLR, Neural Multitask Logistic Regression model; SOFA, Sequential Organ Failure Assessment; SAPSII, Simplified Acute Physiology Score II; GCS, Glasgow Coma Scale.
To further evaluate the models, we also plotted prediction error curves representing the BS over time for 4 models and 3 scoring system models (see Supplementary Figure S3 and S4). The results showed that the IBS of the DeepSurv model in both internal and external validation cohorts was less than 0.25, indicating good discriminative ability. Supplementary Figure S5 depicted the calibration plots of the 4 models. Additionally, for clinical utility, the net benefit of the 4 models were all greater over the threshold probability range (supplementary material Figure S6). Notably, the DCA of DeepSurv was significantly higher than those of the SOFA, SAPS II and GCS in both the internal and external testing datasets.
After balancing the data using the ROS method, the DeepSurv model still performed excellently, with an AUC value of 0.838 for the internal validation set and 0.792 for the external validation set. When compared with the performance of the model trained on the unbalanced dataset (with AUC values of 0.851 and 0.801 respectively), the performance difference was relatively small, which further supported the robustness and reliability of the present model. Supplementary Table S3 provided detailed data on other relevant metrics such as the C-index, accuracy, and F1 score etc.
Through the assessment of feature importance, we identified several key features that were important for model accuracy for prognosis, including hospital LOS, GCS, ICU LOS, CRRT, SAPS II, and ACEI/ARB (supplementary material Figure S7). Replacing the data for these features resulted in an average reduction of more than 1% in the concordance index. The feature importance ranking based on permutation importance method for the 4 models was provided in supplementary material Table S4 and S5.
Next, the SHAP method was further used to interpret the optimal DeepSurv model (see Figure 4 and supplementary material Table S5). Although the 2 feature importance evaluation methods produced slightly different feature importance rankings, the most important features (such as hospital LOS and GCS) are consistently ranked, further strengthening the reliability of our models. Specifically, Figure 4A ranked the variables by comparing their contributions to survival predictions, highlighting the significance of β-blocker use, ACEI/ARB use, CRRT, AKI, and ethnicity. Figure 4B further details the associations between the 22 features and survival rates. The results showed that the hospital LOS, GCS, weight, potassium levels, and chloride levels were positively correlated with survival rates of septic patients with HF, implying that higher values of these characteristics correspond to higher survival rates for such patients. Additionally, the use of β-blockers and ACEI/ARB also exhibited a positive correlation with survival rates among septic patients with HF. However, indicators closely related to renal health, such as AKI and CRRT, had a negative impact on survival rates. Moreover, given that DeepSurv, as a deep learning extension of the CoxPH model, demonstrated similar performance to the CoxPH model, we also conducted SHAP analysis for the CoxPH model. The results were highly consistent with the DeepSurv model (supplementary material Figure S8), further validating the robustness of the present models.
Interpreting the results of DeepSurv model using SHAP explainer. Bar plots of mean absolute SHAP values: ranking of feature importance indicated by SHAP (A). The matrix plot depicts the importance of each covariate in the development of the final predictive model. SHAP summary plots for the 22 clinical features (B): the higher the SHAP value of a feature, the higher the probability of survival. Each line represents a feature, and the abscissa is the SHAP value. Red dots represent higher feature values, and blue dots represent lower feature values. Abbreviations: ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; AKI: acute kidney injury; BUN, blood urea nitrogen; CRRT, continuous renal replacement therapy; DeepSurv, Deep Learning Survival model; GCS, Glasgow Coma Scale; SAPS II, simplified acute physiology II; SHAP: SHapley Additive explanation, HR, heart rate; ICU: intensive care unit; LOS: Length of hospital stay.
To our knowledge, this is so far the most comprehensive survival prediction model developed and validated specifically for septic critically ill patients with HF. The results showed that each model constructed by these 4 algorithms performed satisfactorily in terms of discrimination and accuracy, with the DeepSurv model particularly standing out. Furthermore, all 4 models obtained relatively good net benefits in terms of clinical applicability. In summary, our study presented a predictive model developed specifically for septic patients with HF, who often faced a high mortality rate. It was expected to help clinicians identify high-risk patients early and implement appropriate treatment measures, thereby improving the 28-day post-discharge survival rate for these patients.
Hospitalized patients with sepsis typically require ICU admission, especially when their condition is complicated by various comorbidities such as HF. It was indicated that sepsis was one of the leading causes of death in patients with HF, accounting for a quarter of HF mortalities. 25 Additionally, the underlying cardiac dysfunction in HF could worsen the prognosis for sepsis patients, with studies showing that septic patients with HF faced a higher risk of death. 26 Furthermore, our study found that the 28-day mortality rate for septic patients with HF in the MIMIC-III database was 17.4%, while in the MIMIC-IV database it was as high as 24.8%, further indicating that this might be a patient group with a very high risk of death. However, the prognostic factors associated with septic patients with HF are currently unclear, so, it is necessary to construct models for this high-risk population. In this context, the present study utilized readily accessible clinical patient information to construct a simple and easy-to-use predictive model, which demonstrated excellent performance and provided relevant predictive indicators for assessing the survival rates of septic patients with HF.
In recent years, ML algorithms played a crucial role in early disease warning and prognosis prediction due to their ability to comprehensively identify complex feature networks and even make a real-time prediction based on time series, which cannot be completed by traditional regression analysis.27–29 DeepSurv, as the standout among ML algorithms, was widely applied in the development and practice of medical prediction models.30–32 Compared with other algorithms, DeepSurv comprehensively reveals nonlinear relationships within the data and is more flexible in handling complex clinical factors.33,34 Therefore, we constructed 4 predictive models including DeepSurv and subsequently compared model performance. The results showed that the DeepSurv model exhibited the best discrimination and calibration capabilities in both internal and external validations. Furthermore, the DeepSurv model demonstrated superior predictive performance compared to traditional scoring systems such as SOFA, SAPS II, and GCS. In contrast to previous similar studies,35,36 which mostly simplified predictions into binary classification problems, our present models integrated deep learning techniques and modeled the probability of an event as a function of time. This approach not only accurately captured the dynamic features of survival probability over time but also adapted flexibly to various complex situations and the models hence demonstrated outstanding predictive performance.
The SHAP analysis results revealed 22 important variables associated with the survival rates of septic patients with HF, most of which were widely recognized routine clinical variables. Notably, our results showed that β-blockers ranked third and ACEI/ARB ranked fifth as protective factors influencing the survival rates of septic patients with HF. This may be that β-blockers and ACEI/ARB not only effectively alleviate HF symptoms but also improve sepsis-related symptoms by reducing the activity of inflammatory cytokines.26,37,38 Additionally, it was indicated that β-blockers could further improve sepsis symptoms by counteracting metabolic dysregulation.26,39 Therefore, future research could further focus on the therapeutic effects of these two classes of medications in such patients, aiming to provide more effective treatment strategies for this patient population. Meanwhile, the ninth-ranked CRRT and tenth-ranked AKI are also noteworthy. Our results suggested the negative impact of renal impairment on survival rates of septic patients with HF. Studies showed that CRRT was associated with the risk of acute cardiac stunning in critically ill patients and might enhance inflammatory responses in patients.40,41 Moreover, AKI is closely related to poor outcomes in both sepsis and HF patients.42,43 Therefore, extra attention needs to be given to the condition of renal impairment in such patients, and appropriate treatment should be administered accordingly. Moreover, although ethnicity factors ranked relatively low in variable importance, they occupied three positions among the top 20 key variables. Previous studies showed significant differences in the incidence and mortality rates of sepsis and HF patients among different ethnicity groups, highlighting the importance of ethnicity.44–46 However, ethnicity factors are often overlooked in medical practice. Especially for septic patients with HF, the studies on the relationship between ethnicity and mortality are lacking. Thus, future research may need to further explore the impact of ethnicity factors on septic patients with HF to accurately identify high-risk ethnic groups and implement personalized treatment measures. In summary, these results not only further validated the reliability of our model but also provided potential research directions for the treatment of septic patients with HF.
An interesting point was that our previous research showed a significant association between H2RAs exposure and lower mortality in HF patients.22–24 Considering that septic patients with HF are a small subset of HF patients, we speculated that H2RAs exposure was also associated with reduced mortality in our study population. Therefore, we included H2RAs in the model construction process to evaluate its effect on septic patients with HF. However, the SHAP analysis results showed that H2RAs exposure did not exhibit variable importance in the mortality of septic patients with HF, which indicated that H2RAs exposure might not be related to the death outcomes of this subpopulation although the specific HF population sensitive to H2RAs exposure still needs further exploration.
Our study has several strengths. First, this is the first study to combine ML with time-to-event data to predict the survival rates of critically ill septic patients with HF. This innovation not only provides new analytical tools for clinical practice but also offers a novel perspective for understanding the prognosis of these complex patient populations. Second, we employed the SHAP method to enhance the interpretability of the DeepSurv model, thereby further validating the reliability of our predictions. Finally, all the feature variables used in the model are routinely recorded and widely employed in clinical practice, which underscores the good practical value of our model in real-world clinical decision-making.
However, this study has several limitations. First, as a retrospective study, it may introduce unavoidable selection bias and be influenced by missing data. Second, since the MIMIC database was a single-center database, the external generalizability of these outcomes was not robust. So, further studies are needed to validate our findings. Third, we only considered septic patients with HF in the ICU. The applicability of the model to patients in ordinary wards remains to be tested. Additionally, our model might only help physicians quickly identify critically ill patients but does not provide additional information about underlying life-threatening pathophysiological mechanisms. These limitations should be overcome in future related studies.
Conclusion
In summary, we developed and validated a DeepSurv model, which demonstrated good performance in predicting the 28-day survival rate of septic critically ill patients with HF. Meanwhile, interpretive analysis revealed the contributions of related predictive factors such as ACEI/ARB, β-blockers, AKI, CRRT, and ethnicity to the outcomes of septic patients with HF, further demonstrating the reliability of the model. Considering that these variables identified by the present model previously did not receive enough attention for septic critically ill patients with HF, they need to be closely monitored and integrated in future practice. Overall, the present model, when appropriately integrated into ICU workflows or electronic health record systems, would assist clinicians in screening high-risk septic patients with HF, enabling timely and appropriate treatment to improve patient survival rates.
Supplemental Material
sj-docx-1-thc-10.1177_09287329251346284 - Supplemental material for Predicting survival rates of critically ill septic patients with heart failure using interpretable machine learning models
Supplemental material, sj-docx-1-thc-10.1177_09287329251346284 for Predicting survival rates of critically ill septic patients with heart failure using interpretable machine learning models by Hai-Ying Yang, Meng-Han Jiang, Fang Yu, Li-Juan Yang, Xin Zhang, De-Min Li, Yu Guo, Jia-De Zhu, Sun-Jun Yin and Gong-Hao He in Technology and Health Care
Footnotes
Acknowledgements
The authors wish to acknowledge the MIMIC database for providing clinical data for this study.
ORCID iDs
Ethical considerations
The establishment of this database was approved by the Massachusetts Institute of Technology (Cambridge, MA, USA) and Beth Israel Deaconess Medical Center (Boston, MA, USA), and approval from the local ethics committee was not required, due to all patients’ data being anonymized before the data were obtained.
Author contributions
Haiying Yang, Menghan Jiang, Fang Yu, and Gonghao He contributed to the conception or design of the work. Fang Yu, Lijuan Yang, Sunjun Yin, and Xin Zhang contributed to the acquisition and analysis of data for the work. Demin Li, Yu Guo, and Jiade Zhu are responsible for the interpretation of data. Haiying Yang and Menghan Jiang drafted the manuscript. Haiying Yang and Gonghao He critically revised the manuscript. All authors reviewed and approved the final manuscript.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Grants from the National Science Foundation of China (Grant No. 81960664), the Applied Basic Research Program Yunnan Province of China (Grant No. 202501AS070163 and 202401AY070001-367) the Young and Middle-aged Academic and Technical Leaders Reserve Talent Project of Yunnan Province (Grant No. 202405AC350037). The funders had no part in the study design, conduct or data analysis and did not have any authority over these activities.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data availability statement
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
