Diagnostic performance of machine-learning algorithms for sepsis prediction: An updated meta-analysis

Abstract

BACKGROUND:

Early identification of sepsis has been shown to significantly improve patient prognosis.

OBJECTIVE:

Therefore, the aim of this meta-analysis is to systematically evaluate the diagnostic efficacy of machine-learning algorithms for sepsis prediction.

METHODS:

Systematic searches were conducted in PubMed, Embase and Cochrane databases, covering literature up to December 2023. The keywords included machine learning, sepsis and prediction. After screening, data were extracted and analysed from studies meeting the inclusion criteria. Key evaluation metrics included sensitivity, specificity and the area under the curve (AUC) for diagnostic accuracy.

RESULTS:

The meta-analysis included a total of 21 studies with a data sample size of 4,158,941. Overall, the pooled sensitivity was 0.82 (95% confidence interval [CI] $=$ 0.70–0.90; $P<$ 0.001; $I^{2}=$ 99.7%), the specificity was 0.91 (95% CI $=$ 0.86–0.94; $P<$ 0.001; $I^{2}=$ 99.9%), and the AUC was 0.94 (95% CI $=$ 0.91–0.96). The subgroup analysis revealed that in the emergency department setting (6 studies), the pooled sensitivity was 0.79 (95% CI $=$ 0.68–0.87; $P<$ 0.001; $I^{2}=$ 99.6%), the specificity was 0.94 (95% CI 0.90–0.97; $P<$ 0.001; $I^{2}=$ 99.9%), and the AUC was 0.94 (95% CI $=$ 0.92–0.96). In the Intensive Care Unit setting (11 studies), the sensitivity was 0.91 (95% CI $=$ 0.75–0.97; $P<$ 0.001; $I^{2}=$ 98.3%), the specificity was 0.85 (95% CI $=$ 0.75–0.92; $P<$ 0.001; $I^{2}=$ 99.9%), and the AUC was 0.93 (95% CI $=$ 0.91–0.95). Due to the limited number of studies in the in-hospital and mixed settings ( $n<$ 3), no pooled analysis was performed.

CONCLUSION:

Machine-learning algorithms have demonstrated excellent diagnostic accuracy in predicting the occurrence of sepsis, showing potential for clinical application.

Keywords

Sepsis machine learning prediction meta-analysis

1. Introduction

Sepsis, a severe infectious disease characterised by uncontrolled inflammation following infection, has shown an increasing incidence globally, posing a significant challenge to healthcare systems [1]. It is a major cause of mortality, particularly contributing to deaths in intensive care units (ICUs) [2]. Typically caused by bacteria, fungi or viruses, sepsis spreads through the bloodstream, leading to Systemic Inflammatory Response Syndrome [3]. Epidemiological studies indicate its prevalence across all age groups, with the elderly, immunocompromised individuals and those with hospital-acquired infections being high-risk populations. The inflammatory response triggered by sepsis not only harms the site of infection but may also result in Multiple Organ Dysfunction Syndrome, increasing the risk of patient mortality [4].

Early prediction of sepsis is crucial for timely and effective intervention, given the condition’s acute and rapidly evolving nature, allowing for a reduction in severity and improved patient survival rates [5]. Despite treatment guidelines providing recommendations for the timing of intervention, early identification of sepsis remains a challenging task due to its clinical complexity. Currently, there is a lack of widely accepted gold standards for early sepsis diagnosis, compounded by the high heterogeneity of underlying diseases in patients, making accurate early identification relatively challenging [6].

With the rapid advancement of computational capabilities and the widespread application of big data, machine-learning (ML) and artificial intelligence (AI) technologies have made significant breakthroughs in recent years. Compared with traditional clinical judgment, ML algorithms exhibit unique advantages, particularly in handling the dynamic and complex task of processing clinical information obtained from patients with sepsis [7]. Traditional physicians may face cognitive fatigue when confronted with large patient datasets, whereas ML algorithms efficiently process large-scale, high-dimensional data, uncovering potential patterns and correlations. One key feature of ML algorithms is their ability to automatically learn and optimise models, continuously updating predictive performance based on changing data. This is crucial for addressing the complex and dynamic nature of clinical information from patients with sepsis. Additionally, ML algorithms can uncover non-linear relationships in data, identifying latent features and patterns to enhance the accuracy of early sepsis prediction [8]. In the medical field, ML has already been successfully applied to areas such as tumour prediction and cardiovascular disease risk assessment. For sepsis prediction models, ML algorithms can integrate multiple dimensions of data, including clinical indicators, laboratory test results and medical imaging, to establish relatively comprehensive and accurate models [9].

In the past two years, numerous studies on ML prediction of sepsis have emerged. Researchers have successfully built models with high predictive performance by integrating patient medical imaging, electronic health records and laboratory test data. Although some previous systematic reviews have provided comprehensive summaries of relevant research, the diagnostic data lack timely updates and fail to reflect the recent surge in studies and models, resulting in a lack of a comprehensive understanding of current research progress. Therefore, the purpose of this study is to conduct a more detailed and quantitative meta-analysis, summarising the latest research and evaluating the diagnostic efficacy of ML algorithms in predicting sepsis.

2. Methods

This study followed the reporting items recommended by the Preferred Reporting Items for a Diagnostic Test Accuracy guideline [10] for reporting systematic reviews and meta-analyses of diagnostic test accuracy studies.

2.1 Search strategy and literature selection

PubMed, Embase and the Cochrane Central Register of Controlled Trials (CENTRAL) were selected for comprehensive searches in this study, covering literature from the inception of databases up to 11 December 2023. The search had no language restrictions and utilised a combination of controlled vocabulary terms (MeSH or Emtree) and free-text terms. The primary keywords included sepsis, ML and prediction. Additionally, manual searches were conducted to include relevant references from reviews or meta-analyses.

Two independent researchers conducted the search based on a predefined strategy. Electronic literature was managed using Endnote software, which automatically removed duplicate entries, and manual verification was performed. The researchers screened studies by reviewing titles and abstracts to identify potential studies that met the inclusion criteria. In the data collection process, essential data were extracted using a pre-established data collection form. In case of discrepancies between the two researchers’ results, resolution was achieved through consensus.

2.2 Inclusion and exclusion criteria

The inclusion and exclusion criteria in the present study were designed following the Population, Intervention, Comparison, Outcome, Study Design principle. Specifically, studies meeting the following conditions were included: (1) studies involving diseased or potentially diseased humans as the study participants; (2) studies using ML algorithms to model data for predicting the risk of sepsis or related conditions, such as severe sepsis or septic shock; (3) studies not mandating a specific diagnostic reference method; (4) studies reporting numerical values of diagnostic parameters, including false negatives, false positives, true negatives and true positives, or providing data to calculate a 2*2 table; and (5) studies designed as cohort studies or case-control studies.

Studies were excluded if they fell into the following categories: (1) redundant studies with identical data; (2) irrelevant reviews, letters, conference abstracts or non-peer-reviewed studies; and (3) studies with incomplete data or studies that did not report predefined outcomes.

2.3 Data extraction and risk of bias assessment

From each included study, two independent researchers extracted the following data: the first author and publication year of the study, study design and centre, geographical region, data year, data source, sample size, incidence, validation method, specific algorithm, number of included variables, duration between the prediction time point and the onset of sepsis, data types, predicted outcomes, outcome definitions, diagnostic parameters and area under the curve (AUC) data. If a study developed multiple algorithm models, priority was given to the model with the best performance (highest AUC value) and diagnostic parameters from the validation cohort. Two researchers independently assessed the methodological quality using the Quality Assessment of Diagnostic Accuracy Studies tool, which includes patient selection, index test, reference standard, flow and timing. Disagreements between the two researchers were resolved through discussion or consultation with a third researcher.

2.4 Statistical analysis

Data analysis for this study utilised the RevMan 5.4 software and Stata SE 15.0 software. Sensitivity and specificity calculations were based on 2*2 diagnostic tables, presented through forest plots and summary receiver operating characteristic curves. Stata SE 15.0 was used for merging diagnostic values, such as sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, diagnostic odds ratio and overall AUC. The merged values and their 95% confidence intervals (CIs) were reported. The heterogeneity of the included study results was assessed based on Cochran’s Q test and the $I^{2}$ statistic. Funnel plots and test values were used to assess the possibility of publication bias, with a $P$ -value of $>$ 0.05 indicating no obvious publication bias. Sensitivity analysis was employed to assess the stability of the overall results after excluding a single study. Clinical utility was evaluated based on Fagan’s nomogram, calculating post-test probabilities for the onset of sepsis given the pre-test probability.

Figure 1.

PRISMA flow diagram of study selection.

3. Results

3.1 Literature search

The process of research retrieval and selection is illustrated in Fig. 1. Initially, after searching three electronic databases, a total of 1,466 articles were obtained, with 569 from PubMed, 861 from Embase and 36 from CENTRAL. Following the removal of duplicate records using both software and manual methods, 983 independent electronic records were obtained. Subsequently, after reviewing the titles and abstracts, 938 articles unrelated to the research topic were excluded, leaving 45 articles for full-text reading. During the full-text reading process, a total of 24 articles were collectively excluded due to reasons such as having incomplete data, only prognostic outcomes or non-sepsis-related content and being abstracts/letters. Finally, 21 articles were included for the subsequent meta-analysis [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31].

3.2 Basic characteristics of included studies

The characteristics of the studies meeting the inclusion criteria are detailed in Table 1. Among the 21 studies, the total sample size was 4,158,941, with 6 studies focusing on the emergency department, 2 on inpatient settings, 11 on the ICU and 2 on mixed scenarios. With the exception of 1 study that did not report information, 20 studies employed a retrospective analysis approach, with the majority (61.9%) being single-centre studies and 7 (33.3%) being multi-centre studies. Of the 21 studies, 14 used data from the United States, 2 from China, 2 from South Korea, 1 from Sweden and 1 from Thailand. Only 1 did not report the data source. The study data spanned from 2001 to 2021, with 6 studies utilising data from the MIMIC database. The sample size ranged from 142 to 2,759,529, and the event occurrence rate for the study outcomes ranged from 0.41% to 79.5%. Various validation methods were employed, with 52.4% using random splitting, 19.0% using temporal validation and 19.0% using cross-validation, among others.

Table 2 provides an overview of all the constructed ML models. In the 21 studies meeting the criteria, different algorithms were used for modelling, with common ML algorithms including support vector machines, random forests and naïve Bayes. Nearly all studies (95.2%) used vital signs for modelling, 8 studies used demographic data, 13 studies used laboratory tests, 3 studies used nursing documentation, and 2 studies used baseline characteristics. The number of variables included ranged from 2 to 41. A total of 13 studies solely predicted sepsis, 3 studies predicted septic shock, and 4 studies predicted multiple events. There were differences in the definition of sepsis among the included studies, with 4 studies using International Classification of Diseases definitions, 9 studies using Sepsis-3 definitions and 3 studies using a mixed definition. The reported types of diagnostic accuracy parameters varied, with 3 studies not reporting the AUC; the reported AUC ranged from 0.65 to 0.994. A total of 8 studies reported the time before the onset of sepsis, ranging from 0 to 48 hours.

3.3 Risk of bias assessment

The methodological quality of the 21 included studies is presented in Fig. 2. Due to unclear descriptions, 1 study was at unclear risk of bias in the “Patient Selection” domain, and 7 studies were determined to be at high risk due to factors such as continuous inclusion. In comparison, most studies (90.5%) had a low risk of bias in the “Index Test” domain. Given the lack of a clear gold standard for sepsis and different studies using slightly different criteria, the risk of bias score for the “Reference Standard” domain was deemed unclear for all studies. Moreover, only 3 studies had some risk of bias in the “Flow and Timing” domain. It is noteworthy that, considering the relevance to the meta-analysis question, 1 study may have a relatively high sensitivity in the “Index Test” domain.

3.4 Meta-analysis

The findings from the meta-analysis regarding the early prediction of sepsis risk using ML are depicted in Fig. 3. The collective sensitivity across the 21 studies was 0.82 (95% CI $=$ 0.70–0.90; $P<$ 0.001; $I^{2}=$ 99.7%), and the specificity stood at 0.91 (95% CI $=$ 0.86-0.94; $P<$ 0.001; $I^{2}=$ 99.9%). The positive likelihood ratio reached 8.8 (95% CI $=$ 5.6–13.9), and the negative likelihood ratio was 0.20 (95% CI $=$

Table 1
Overview of the included studies

First author	Design	Center	Country	Data period	Dataset	Sample size	Prevalence (%)	Validation methods
ED
Brown, 2016 [11]	Retrospective	Multi-center	USA	2006/1–2008/12, 2009/4–2010/6	LDS Hospital and Intermountain Medical Center	93733–132748	0.41	Temporal validation
Delahanty, 2019 [12]	Retrospective	Multi-center	USA	2016/1–2017/10	Urban community hospitals (Tenet Healthcare)	2759529	2.02	Random splitting
Haug, 2016 [13]	Retrospective	Multi-center	USA	2006/1–2008/12	Two Salt Lake hospitals	280143	1.78	Cross-validation
Kijpaisalratana, 2022 [14]	Retrospective	Single-center	Thailand	2018/6–2020/5	A 1,500-bed, urban, university-affiliated hospital	135661	0.90	Random splitting
Kim, 2020 [15]	Retrospective	Single-center	South Korea	2008–2016	A tertiary academic hospital	49560	9.70	Random splitting
Mercurio, 2023 [16]	Retrospective	Single-center	USA	2017/1–2019/1	A tertiary care children’s hospital	35074	0.54	Cross-validation
In-hospital
Khojandi, 2018 [17]	Retrospective	Multi-center	USA	2008–2015	Health Facts dataset	261258	46.8	Random splitting
Thiel, 2010 [18]	Retrospective	Single-center	USA	2005–2007	A 1200-bed academic medical center	41459	4.50	Temporal validation
ICU
Chen, 2023 [19]	Retrospective	Single-center	China	2015/1–2021/4	Liver transplantation records	677	31.9	Temporal validation
Eskandari, 2023 [20]	Unknown	Unknown	Unknown	Unknown	The physio net database	40336	1.8	N/A
Henry, 2015 [21]	Retrospective	Single-center	USA	2001–2007	MIMIC-II	16234	14.1	Random splitting
Honoré, 2023 [22]	Retrospective	Single-center	Sweden	2016/2–2020/4	NICUs of Karolinska University Hospital	325	16.3	Random splitting
Kam, 2017 [23]	Retrospective	Single-center	USA	2001–2012	MIMIC-II	5803	6.2	Random splitting
Kim, 2023 [24]	Retrospective	Single-center	South Korea	2010/1–2021/12	A single tertiary center	28630	27.6	Temporal validation
Liu, 2022 [25]	Retrospective	Multi-center	USA	Unknown	MIMIC-II, PhysioNet Challenge 2019 Dataset	11840	34.3	Random splitting

Table 1, continued
First author	Design	Center	Country	Data period	Dataset	Sample size	Prevalence (%)	Validation methods
Liu, 2023 [26]	Retrospective	Single-center	USA	2001–2012, 2008–2009	MIMIC-III and MIMIC-IV	1672	19.51	Random splitting
Shashikumar, 2017 [27]	Retrospective	Single-center	USA	Unknown	Emory affiliated ICU	682	22	Cross-validation
Shimabukuro, 2017 [28]	Retrospective	Single-center	USA	2016/12–2017/2	UCSF	142	17.60	N/A
Wang, 2021 [29]	Retrospective	Single-center	China	2014–2016	ICU of the First Affiliated Hospital of Zhengzhou University	4449	79.50	Random splitting
Mixed situation
Barton, 2019 [30]	Retrospective	Multi-center	USA	2011/6–2016/3	UCSF and MIMIC-III	112952	3.48–5.43	Cross-validation
Mao, 2018 [31]	Retrospective	Multi-center	USA	2011–2016	UCSF and MIMIC-III	1140-239767	1.30-1.91	Random splitting

ED: emergency department; USA: the United States of America; ICU: intensive care unit; UCSF: University of California, San Francisco; MIMIC: Multiparameter Intelligent Monitoring in Intensive Care.

Table 2

Characteristics of machine learning-based prediction models for sepsis

First author	ML algorithm	Number of variables	Data types	Predictive outcome	Definition	Diagnostic parameters	AUC	Hours before onset
ED
Brown, 2016 [11]	NB	5	Vital signs and lab tests	Severe sepsis/septic shock	ICD-9	AUC, SEN, SPE, PPV, NPV, FPR	0.953	N/A
Delahanty, 2019 [12]	GB	12	Vital signs and lab tests	Sepsis	Sepsis-3	AUC, SEN, SPE, PPV	0.93-0.97	N/A
Haug, 2016 [13]	NB	6	Demographics, vital signs, lab tests, radiology, nursing documentation	Sepsis	ICD-9	AUC, SEN, PPV	0.9505-0.9659	N/A
Kijpaisalratana, 2022 [14]	RF	13	Vital signs, baseline characteristics, and chief complains	Sepsis	Sepsis-3, ICD-10-CM	AUC, SEN, SPE, PPV, NPV	0.931	N/A
Kim, 2020 [15]	SVM, GBM, RF, MARS and LASSO	24	Vital signs, baseline characteristics, lab tests, and chief complains	Septic shock	Sepsis-3	AUC, SEN, SPE, PPV, NPV	0.883-0.929	N/A
Mercurio, 2023 [16]	RF and CART	20	Sociodemographic information, vital signs, nursing documentation	Sepsis/severe sepsis/septic shock	ICD-10	AUC, SEN, SPE, PPV, F1 score	0.65-0.82	N/A
In-hospital
Khojandi, 2018 [17]	EM	9	Demographics, vital signs, lab tests	Sepsis	SIRS criteria	AUC, SEN, SPE, accuracy, F1 score, MCC	0.65	0
Thiel, 2010 [18]	DT	9	Vital signs and lab tests	Septic shock	ICD-9	PPV, NPV, MCR	N/A	N/A
ICU
Chen, 2023 [19]	RF	8	Vital signs and nursing documentation	Sepsis	Sepsis-3	AUC, SEN, SPE, accuracy, F1 score	0.755	N/A
Eskandari, 2023 [20]	KNN	41	Vital signs and lab tests	Sepsis	Unknown	SEN, SPE, accuracy	N/A	N/A
Henry, 2015 [21]	PHM	27	Vital signs and lab tests	Septic shock	ICD-9 and SIRS criteria	AUC, SEN, SPE	0.83	28.2

Table 2, continued
First author	ML algorithm	Number of variables	Data types	Predictive outcome	Definition	Diagnostic parameters	AUC	Hours before onset
Honoré, 2023 [22]	NB	Unknown	Vital signs	Sepsis	A neonatologist’s documented assessment	AUC, LR+, LR-	Unknown	20
Kam, 2017 [23]	NNM, LSTM	7	Demographics, vital signs, lab tests	Sepsis	SIRS criteria	AUC, SEN, SPE, accuracy	0.929	0–3
Kim, 2023 [24]	DL	21	Demographics, vital signs, lab tests	Sepsis/Septic shock	Sepsis-3	AUC, SEN, SPE, PPV, NPV, F1 score	0.7888, 0.8494	N/A
Liu, 2022 [25]	XGBoost	Unknown	Demographics, vital signs, lab tests, intervention treatment	Sepsis	Sepsis-3	AUC, SEN, SPE, F1 score	0.8257	N/A
Liu, 2023 [26]	GBDT	13	Demographics, vital signs, lab tests	Sepsis	Sepsis-3	AUC, SEN, SPE, PPV, NPV	0.994	N/A
Shashikumar, 2017 [27]	SVM	2-22	Vital signs	Sepsis	Sepsis-3	AUC	0.78	4
Shimabukuro, 2017 [28]	GLM	6	Vital signs	Severe sepsis	Surviving Sepsis Campaign	AUC, SEN, SPE	0.952	0
Wang, 2021 [29]	RF	20	Demographics, lab tests	Sepsis	Sepsis-3	AUC, accuracy, Precision, F1 score	0.91	N/A
Mixed situation
Barton, 2019 [30]	GBDT	6	Vital signs	Sepsis	Sepsis-3	AUC, SEN, SPE, DOR, LR+, LR-	0.88	0–48
Mao, 2018 [31]	GBDT	3–6	Vital signs	Sepsis/Severe sepsis/Septic shock	SIRS criteria and ICD-9	AUC, SEN, SPE	0.92	0–4

ML: machine learning; AUC: area under the curve; ED: emergency department; NB: naïve Bayes; ICD-9: International Classification of Disease-9; PPV: positive predictive value; NPV: negative predictive value; FPR: False positive rate; GB: gradient boosting; RF: random forest; SVM: support vector machine; GBM: gradient boosting machine; MARS: multivariate adaptive regression splines; LASSO: least absolute shrinkage and selection operator; CART: classification and regression trees; EM: ensemble methods; SIRS: systemic inflammatory response syndrome; DT: decision tree; KNN: K- Nearest Neighbor; PHM: proportional hazards model; NNM: neural network model; LSTM: long short term memory; GBDT: gradient boosting decision tree; GLM: generalized linear model; DOR: diagnostic odds ratio; LR+: positive likelihood ratio; LR-: negative likelihood ratio.

Figure 2.

Methodological evaluation of included literatures.

0.11–0.35). Moreover, the diagnostic odds ratio was 44 (95% CI $=$ 19–101), and the AUC was 0.94 (95% CI $=$ 0.91–0.96), collectively underscoring the robust predictive capability of ML for sepsis (Fig. 4). Specifically, within the 6 studies conducted in the emergency department, the amalgamated sensitivity was 0.79 (95% CI $=$ 0.68–0.87; $P<$ 0.001; $I^{2}=$ 99.6%), coupled with a specificity of 0.94 (95% CI $=$ 0.90–0.97; $P<$ 0.001; $I^{2}=$ 99.9%). The positive likelihood ratio was 13.6 (95% CI $=$ 8.4–22.2), and the negative likelihood ratio was 0.22 (95% CI $=$ 0.15–0.34), resulting in a diagnostic odds ratio of 61 (95% CI $=$ 43–86) and an AUC of 0.94 (95% CI $=$ 0.92–0.96). Among the 11 studies conducted in the ICU, the sensitivity attained was 0.91 (95% CI $=$ 0.75–0.97; $P<$ 0.001; $I^{2}=$ 98.3%), and the specificity was 0.85 (95% CI $=$ 0.75–0.92; $P<$ 0.001; $I^{2}=$ 99.9%). The positive likelihood ratio was 6.1 (95% CI $=$ 3.3–11.5), and the negative likelihood ratio was 0.11 (95% CI $=$ 0.04–0.34). The diagnostic odds ratio reached 55 (95% CI $=$ 10–298), and the AUC was 0.93 (95% CI $=$ 0.91–0.95). Given the limited number of studies available for inpatient and mixed scenarios ( $n<$ 3), a combined analysis was not conducted.

Figure 3.

Forest plot of machine learning-based models for predicting sepsis.

Figure 4.

SROC curve of machine learning-based models for predicting sepsis.

Figure 5.

Funnel plot of machine learning-based models for predicting sepsis.

3.5 Publication bias and sensitivity analysis

Publication bias analysis of the included studies, as shown in Fig. 5, demonstrated no apparent publication bias based on the asymmetry test of the funnel plot ( $P=$ 0.35). Sensitivity analysis of the meta-analysis revealed that the exclusion of any single study did not significantly impact the overall diagnostic performance results, indicating the stability of the study results. It is essential to note the high significance among studies.

3.6 Clinical utility

Machine-learning-based sepsis prediction models were used; when the pre-test probability was positive, the post-test probability increased from 50% to 90%, with a positive likelihood ratio of 9. When the pre-test probability was negative, the post-test probability decreased from 50% to 17%, with a negative likelihood ratio of 0.20 (Fig. 6).

Figure 6.

Fagan plot of machine learning-based models for predicting sepsis.

4. Discussion

This study conducted a comprehensive analysis of the latest research using a quantitative approach, specifically a meta-analysis, to evaluate the accuracy of ML algorithms in predicting the risk of sepsis. The main findings are as follows: (1) In the 21 studies included, ML algorithms demonstrated excellent diagnostic performance for predicting sepsis, not only in the ICU but also in the emergency department. (2) The methodological assessment revealed that the data quality of current studies is not high, with many studies being excluded due to incomplete data reporting. This suggests the need for more high-quality studies following guidelines in the future. (3) Before clinical application, research on the algorithm’s impact on patient prognosis and clinical decision-making is necessary. This meta-analysis affirms the diagnostic potential of ML algorithms, providing direction for future research and potentially aiding clinical decision-making. Machine-learning algorithms can seamlessly integrate with clinical decision support systems, providing clinicians with actionable insights at the point of care. By embedding ML-based prediction models into electronic health record systems, clinicians can receive real-time alerts and recommendations based on individual patient data, facilitating early recognition of sepsis and guiding appropriate treatment strategies. The study’s meta-analysis underscores the potential of ML-based decision support systems to augment clinical decision-making and enhance patient safety.

The characteristics of sepsis, including its prolonged duration, rapid changes and extensive course data, pose significant challenges to clinical decision-making. Machine-learning algorithms show notable advantages in addressing these challenges. By extracting patterns of disease progression from rich data at different time points, ML provides clinicians with more comprehensive and in-depth information, potentially compensating for the limitations of traditional medical judgment when handling extensive course data [32]. The analysis revealed that most studies primarily utilised vital signs and laboratory results in the modelling process, indicating the crucial role of these physiological indicators in predicting sepsis [33]. Future research could further explore the weight and impact of these specific data in ML models to better understand their relationship with the progression of sepsis. The potential widespread application of ML, from emergency rooms to wearable devices in home settings, positions it as a powerful tool for assisting clinical decision-making [33, 34, 35]. Real-time monitoring of patients’ physiological data enables ML to provide timely risk assessments and predictions, facilitating more prompt therapeutic interventions and reducing treatment delays. The advantage of ML lies in its ability to detect trends in disease progression early, which is crucial for early sepsis prediction. Early identification of sepsis allows for targeted therapeutic interventions, improving patient outcomes before the onset of systemic inflammatory storms and reducing adverse events [34, 36]. This study conducted a comprehensive analysis of the latest research using a quantitative approach, specifically a meta-analysis, to evaluate the accuracy of ML algorithms in predicting the risk of sepsis. In addition, recent studies demonstrated that ML techniques could inform the development and validation of ML algorithms for predicting sepsis risk in neonatal populations, potentially enhancing early detection and management strategies for sepsis in newborns [37] and leveraging diverse data sources to enhance diagnostic accuracy and enable early intervention for patients with sepsis combined with underlying cardiac conditions [38]; ML-based software suites for respiratory disease diagnosis may offer valuable lessons for developing similar tools for sepsis prediction, facilitating rapid risk assessment and treatment initiation in patients presenting with respiratory symptoms suggestive of sepsis [39]. Mustafic et al. focused on the diagnosis of severe aortic stenosis using an implemented expert system, which may incorporate ML algorithms to analyse clinical data and imaging findings. Incorporating insights from these articles highlights the versatility of ML techniques in various domains of healthcare [40]. By drawing parallels between different applications of ML in medical diagnosis and prediction, the study can underscore the overarching importance of leveraging advanced analytics to improve patient outcomes and enhance clinical decision-making across diverse clinical scenarios.

However, challenges and limitations exist in the future application of ML algorithms for clinical sepsis prediction or diagnosis. Currently, studies applying ML algorithms to sepsis prediction exhibit methodological issues. Variability in standards and methods across studies, coupled with a lack of a unified gold standard for sepsis diagnosis, may lead to unreliable research results, affecting the genuine performance assessment of algorithms. Therefore, some heterogeneity is observed in the meta-analysis, indicating differences between studies. This heterogeneity may stem from various factors, including the heterogeneity of study participants, different ML algorithms used, differences in sample size and variations in study design. Different clinical environments and disease backgrounds may also contribute to existing heterogeneity. To better understand this heterogeneity, future research can explore the relationships between these factors and attempt subgroup analyses to reveal potential reasons. In the process of applying ML algorithms to the medical field, regulation and standardisation are prominent issues. The lack of uniform standards and regulatory mechanisms may lead to the application of algorithms in clinical practice without sufficient validation [41]. This raises concerns about patient safety and the risks associated with clinical application. Additionally, the interpretability and explainability of algorithms pose challenges, especially in the medical field where clinicians need to understand the decision-making process of algorithms to accept and trust their results.

This study represents a recent and comprehensive meta-analysis of ML algorithms for sepsis prediction. Despite existing systematic reviews [42, 43], the rapid growth of research in the field, especially in the last two years, necessitates timely updates and synthesis of the latest research findings. The study’s meta-analysis not only provides a comprehensive overview of the current landscape of ML-based sepsis prediction but also identifies key areas for future research and clinical practice. By highlighting methodological gaps, such as variability in study design and validation methods, the meta-analysis informs the design of future studies aimed at improving the accuracy and generalisability of ML algorithms. Additionally, by emphasising the clinical implications of ML-based prediction models, the study encourages the integration of these tools into routine clinical practice, paving the way for more proactive and personalised sepsis management strategies. However, certain limitations should be acknowledged. The exclusion of grey literature and non-peer-reviewed literature, along with the relatively strict inclusion criteria, may result in a limited number of covered publications, especially considering that ML studies might be presented at various academic conferences. Therefore, the study results may not comprehensively reflect all relevant research in the field, introducing some selection bias. Additionally, the lack of unified standards for evaluating ML algorithm performance poses challenges. Focusing solely on a single parameter (e.g. AUC) may not comprehensively assess the algorithms’ merits. The selection of the model with the highest AUC during the combined analysis may lead to oversimplification, potentially neglecting the contributions of other performance indicators. Consequently, determining the optimal algorithm for real-world application remains uncertain. The study exhibits relatively high heterogeneity in the meta-analysis, potentially influenced by the limited number of studies with independent external validation. Considering the potential impact of heterogeneity on result consistency, careful interpretation and control of this heterogeneity are required. Increasing the number of studies with independent external validation could improve the methodological quality of the meta-analysis and the robustness of the results.

5. Conclusion

In conclusion, this meta-analysis demonstrates that ML algorithms exhibit excellent diagnostic accuracy in predicting the occurrence of sepsis. They outperform traditional risk scores in numerous studies, showing potential value in future clinical applications. Machine learning provides new insights and methods for the diagnosis and treatment of sepsis. Nevertheless, further high-quality studies are needed to validate the results of this study, propelling the clinical application of ML algorithms in this field.

Footnotes

Conflict of interest

None to report.

References

Rudd

Johnson

Agesa

, et al. Global, regional, and national sepsis incidence and mortality, 1990-2017: analysis for the Global Burden of Disease Study. Lancet. 2020; 395(10219): 200-211. doi: 10.1016/S0140-6736(19)32989-7.

Fleischmann-Struzek

Mellhammar

Rose

, et al. Incidence and mortality of hospital- and ICU-treated sepsis: results from an updated and expanded systematic review and meta-analysis. Intensive Care Med. 2020; 46(8): 1552-1562. doi: 10.1007/s00134-020-06151-x.

Bosmann

Ward

. The inflammatory response in sepsis. Trends Immunol. 2013; 34(3): 129-136. doi: 10.1016/j.it.2012.09.004.

Jacobi

. The pathophysiology of sepsis – 2021 update: Part 2, organ dysfunction and assessment. Am J Health Syst Pharm. 2022; 79(6): 424-436. doi: 10.1093/ajhp/zxab393.

Rhodes

Evans

Alhazzani

, et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Intensive Care Med. 2017; 43(3): 304-377. doi: 10.1007/s00134-017-4683-6.

Yadav

. Progress in Diagnosis and Treatment of Neonatal Sepsis: A Review Article. JNMA J Nepal Med Assoc. 2022; 60(247): 318-324. Published 2022 Mar 11. doi: 10.31729/jnma.7324.

Cockrell

. Sepsis reconsidered: Identifying novel metrics for behavioral landscape characterization with a high-performance computing implementation of an agent-based model. J Theor Biol. 2017; 430: 157-168. doi: 10.1016/j.jtbi.2017.07.016.

Ratzinger

Haslacher

Perkmann

, et al. Machine learning for fast identification of bacteraemia in SIRS patients treated on standard care wards: a cohort study. Sci Rep. 2018; 8(1): 12233. Published 2018 Aug 15. doi: 10.1038/s41598-018-30236-9.

Stemmer

Shadmi

Bregman-Amitai

, et al. Using machine learning algorithms to review computed tomography scans and assess risk for cardiovascular disease: Retrospective analysis from the National Lung Screening Trial (NLST). PLoS One. 2020; 15(8): e0236021. Published 2020 Aug 3. doi: 10.1371/journal.pone.0236021.

10.

McInnes

MDF

Moher

Thombs

, et al. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: The PRISMA-DTA Statement. JAMA. 2018; 319(4): 388-396. doi: 10.1001/jama.2017.19163.

11.

Brown

Jones

Kuttler

Keddington

Allen

Haug

. Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department. BMC Emerg Med. 2016; 16(1): 31. Published 2016 Aug 22. doi: 10.1186/s12873-016-0095-0.

12.

Delahanty

Alvarez

Flynn

Sherwin

Jones

. Development and Evaluation of a Machine Learning Model for the Early Identification of Patients at Risk for Sepsis. Ann Emerg Med. 2019; 73(4): 334-344. doi: 10.1016/j.annemergmed.2018.11.036.

13.

Haug

Ferraro

. Using a semi-automated modeling environment to construct a Bayesian, sepsis diagnostic system. BCB ’16. 2016. doi: 10.1145/2975167.2985841.

14.

Kijpaisalratana

Sanglertsinlapachai

Techaratsami

Musikatavorn

Saoraya

. Machine learning algorithms for early sepsis detection in the emergency department: A retrospective study. Int J Med Inform. 2022; 160: 104689. doi: 10.1016/j.ijmedinf.2022.104689.

15.

Kim

Chang

Kim

Jang

Park

Kim

. Machine learning for prediction of septic shock at initial triage in emergency department. J Crit Care. 2020; 55: 163-170. doi: 10.1016/j.jcrc.2019.09.024.

16.

Mercurio

Pou

Duffy

Eickhoff

. Risk Factors for Pediatric Sepsis in the Emergency Department: A Machine Learning Pilot Study. Pediatr Emerg Care. 2023; 39(2): e48-e56. doi: 10.1097/PEC.0000000000002893.

17.

Khojandi

Tansakul

Koszalinski

Paiva

. Prediction of Sepsis and In-Hospital Mortality Using Electronic Health Records. Methods Inf Med. 2018; 57(4): 185-193. doi: 10.3414/ME18-01-0014.

18.

Thiel

Rosini

Shannon

Doherty

Micek

Kollef

. Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010; 5(1): 19-25. doi: 10.1002/jhm.530.

19.

Chen

Yang

, et al. Development and validation of a practical machine learning model to predict sepsis after liver transplantation. Ann Med. 2023; 55(1): 624-633. doi: 10.1080/07853890.2023.2179104.

20.

Eskandari

Moridani

Mohammadi

. Detection of sepsis using biomarkers based on machine learning. Bratisl Lek Listy. 2023; 124(3): 239-250. doi: 10.4149/BLL_2023_037.

21.

Henry

Hager

Pronovost

Saria

. A targeted real-time early warning score (TREWScore) for septic shock. Sci Transl Med. 2015; 7(299): 299ra122. doi: 10.1126/scitranslmed.aab3719.

22.

Honoré

Forsberg

Adolphson

Chatterjee

Jost

Herlenius

. Vital sign-based detection of sepsis in neonates using machine learning. Acta Paediatr. 2023; 112(4): 686-696. doi: 10.1111/apa.16660.

23.

Kam

Kim

. Learning representations for the early detection of sepsis with deep neural networks. Comput Biol Med. 2017; 89: 248-255. doi: 10.1016/j.compbiomed.2017.08.015.

24.

Kim

Tae

Yeo

, et al. Development and Validation of Deep-Learning-Based Sepsis and Septic Shock Early Prediction System (DeepSEPS) Using Real-World ICU Data. J Clin Med. 2023; 12(22): 7156. Published 2023 Nov 17. doi: 10.3390/jcm12227156.

25.

Liu

Wang

Liu

Sun

. Dynamic Sepsis Prediction for Intensive Care Unit Patients Using XGBoost-Based Model With Novel Time-Dependent Features. IEEE J Biomed Health Inform. 2022; 26(8): 4258-4269. doi: 10.1109/JBHI.2022.3171673.

26.

Liu

Yao

Liu

Shou

. Construction and validation of machine learning models for sepsis prediction in patients with acute pancreatitis. BMC Surg. 2023; 23(1): 267. Published 2023 Sep 1. doi: 10.1186/s12893-023-02151-y.

27.

Shashikumar

Clifford

Nemati

. Multiscale network representation of physiological time series for early prediction of sepsis. Physiol Meas. 2017; 38(12): 2235-2248. Published 2017 Nov 30. doi: 10.1088/1361-6579/aa9772.

28.

Shimabukuro

Barton

Feldman

Mataraso

Das

. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res. 2017; 4(1): e000234. Published 2017 Nov 9. doi: 10.1136/bmjresp-2017-000234.

29.

Wang

Sun

, et al. A Machine Learning Model for Accurate Prediction of Sepsis in ICU Patients. Front Public Health. 2021; 9: 754348. Published 2021 Oct 15. doi: 10.3389/fpubh.2021.754348.

30.

Barton

Chettipally

Zhou

, et al. Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs. Comput Biol Med. 2019; 109: 79-84. doi: 10.1016/j.compbiomed.2019.04.027.

31.

Mao

Jay

Hoffman

, et al. Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open. 2018; 8(1): e017833. Published 2018 Jan 26. doi: 10.1136/bmjopen-2017-017833.

32.

Oliveira

Luís

Sargento

. Machine Learning for the Dynamic Positioning of UAVs for Extended Connectivity. Sensors (Basel). 2021; 21(13): 4618. Published 2021 Jul 5. doi: 10.3390/s21134618.

33.

Sullivan

BA.

. Nagraj

. Berry

. et al. ‘Clinical and Vital Sign Changes Associated with Late-onset Sepsis in Very Low Birth Weight Infants at 3 NICUs’ 1. Technology and Health Care. 2021: 553-561. doi: 10.3233/NPM-200578.

34.

Wessels

Bußoff

Adam

. et al. ‘Comorbidity Scores and Machine Learning Methods Can Improve Risk Assessment in Radical Cystectomy for Bladder Cancer’. 1. Technology and Health Care. 2022: 155-163. doi: 10.1155/2022/5032435.

35.

Hazratifard

Gebali

Mamun

. Using Machine Learning for Dynamic Authentication in Telehealth: A Tutorial. Sensors (Basel). 2022; 22(19): 7655. Published 2022 Oct 9. doi: 10.3390/s22197655.

36.

Burrell

McLaws

Fullick

Sullivan

Sindhusake

. SEPSIS KILLS: early intervention saves lives. Med J Aust. 2016; 204(2): 73. doi: 10.5694/mja15.00657.

37.

Abd Rahman

Ibrahim

Hasikin

Abd Razak

. Critical Device Reliability Assessment in Healthcare Services. J Healthc Eng. 2023; 2023: 3136511. doi: 10.1155/2023/3136511.

38.

Begic

Gurbeta Pokvic

Begic

, et al. From Heart Murmur to Echocardiography - Congenital Heart Defects Diagnostics Using Machine-Learning Algorithms. Psychiatr Danub Dec 2021; 33(Suppl 13): 236-1246.

39.

Badnjevic

Cifrek

Koruga

. Integrated software suite for diagnosis of respiratory diseases. 2013; 564-568.

40.

Mustafic

Gurbeta

Badnjevic-Ceng

. Diagnosis of Severe Aortic Stenosis Using Implemented Expert System. 2020.

41.

Mooney

Pejaver

. Big Data in Public Health: Terminology, Machine Learning, and Privacy. Annu Rev Public Health. 2018; 39: 95-10112. doi: 10.1146/annurev-publhealth-040617-014208.

42.

Moor

Rieck

Horn

Jutzeler

Borgwardt

. Early Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review. Front Med (Lausanne). 2021; 8: 607952. Published 2021 May 28. doi: 10.3389/fmed.2021.607952.

43.

Islam

Prithula

Kumar

, et al. Machine Learning-Based Early Prediction of Sepsis Using Electronic Health Records: A Systematic Review. J Clin Med. 2023; 12(17): 5658. Published 2023 Aug 30. doi: 10.3390/jcm12175658.