Machine learning-based antibiotic resistance prediction models: An updated systematic review and meta-analysis

Abstract

BACKGROUND:

The widespread use of antibiotics has led to a gradual adaptation of bacteria to these drugs, diminishing the effectiveness of treatments.

OBJECTIVE:

To comprehensively assess the research progress of antibiotic resistance prediction models based on machine learning (ML) algorithms, providing the latest quantitative analysis and methodological evaluation.

METHODS:

Relevant literature was systematically retrieved from databases, including PubMed, Embase and the Cochrane Library, from inception up to December 2023. Studies meeting predefined criteria were selected for inclusion. The prediction model risk of bias assessment tool was employed for methodological quality assessment, and a random-effects model was utilised for meta-analysis.

RESULTS:

The systematic review included a total of 22 studies with a combined sample size of 43,628; 10 studies were ultimately included in the meta-analysis. Commonly used ML algorithms included random forest, decision trees and neural networks. Frequently utilised predictive variables encompassed demographics, drug use history and underlying diseases. The overall sensitivity was 0.57 (95% CI: 0.42–0.70; $p<$ 0.001; $I^{2}=$ 99.7%), the specificity was 0.95 (95% CI: 0.79–0.99; $p<$ 0.001; I² = 99.9%), the positive likelihood ratio was 10.7 (95% CI: 2.9–39.5), the negative likelihood ratio was 0.46 (95% CI: 0.34–0.61), the diagnostic odds ratio was 23 (95% CI: 7–81) and the area under the receiver operating characteristic curve was 0.78 (95% CI: 0.74–0.81; $p<$ 0.001), indicating a good discriminative ability of ML models for antibiotic resistance. However, methodological assessment and funnel plots suggested a high risk of bias and publication bias in the included studies.

CONCLUSION:

This meta-analysis provides a current and comprehensive evaluation of ML models for predicting antibiotic resistance, emphasising their potential application in clinical practice. Nevertheless, stringent research design and reporting are warranted to enhance the quality and credibility of future studies. Future research should focus on methodological innovation and incorporate more high-quality studies to further advance this field.

Keywords

Antibiotics machine learning meta-analysis prediction

1. Introduction

Antibiotic resistance is a formidable challenge in today’s global medical landscape. The widespread use of antibiotics has led to a gradual adaptation of bacteria to these drugs, diminishing the effectiveness of treatments. The epidemiological trends of antibiotic resistance have garnered widespread attention, posing serious challenges to public health and clinical practice [1]. Over the past decades, the misuse and irrational use of antibiotics have accelerated the development of resistance, rendering the treatment of infections more complex and challenging. Traditionally, the clinical detection of antibiotic resistance relies heavily on bacterial culture and sensitivity testing, a process that typically takes 2–5 days. Such delayed diagnostic procedures not only postpone the initiation of treatment but also increase the difficulty of obtaining effective treatment in the early stages of infection [2]. Currently, physicians are constrained to empirical antibiotic therapy, and with the escalating bacterial resistance, the efficacy of such empirical treatments has markedly decreased, leading to unpredictable treatment outcomes for patients [3]. Against this backdrop, there is an urgent need for the early prediction of antibiotic resistance. The establishment of prediction models can assist healthcare professionals in obtaining early insights into the antibiotic sensitivity of patient infections, thereby providing more targeted recommendations for treatment planning. Timely intervention not only improves patient prognosis but also helps mitigate the progression of antibiotic resistance by avoiding the overuse of ineffective antibiotics [4].

With the rapid advancement of technology, the application of artificial intelligence and machine learning (ML) algorithms in the medical field has become a research hotspot [5, 6]. Compared with traditional clinical judgment, ML algorithms offer significant advantages, particularly in predicting antibiotic resistance. This advantage stems from the efficient processing of large-scale data and the sensitivity to complex relationships [7]. Traditional clinical judgment is often based on experience and professional knowledge, but when faced with vast amounts of patient information, particularly including extensive data such as genetic sequencing, the judgment of clinicians may be challenged. These data are not only vast and complex but also involve interactions among multiple variables, exceeding the limits of human processing. With their powerful computational and learning capabilities, ML algorithms can discover potential patterns and trends in this huge amount of data, thereby providing more accurate predictions. One of the main characteristics of ML algorithms is their adaptability, as they can learn from data and adapt to new information, making them better suited to handle emerging data and knowledge in the medical field [8]. Additionally, ML can perform nonlinear modelling, identifying complex relationships crucial for the multifactorial and multilayered nature of antibiotic resistance. In medical applications, ML algorithms have been applied successfully in disease diagnosis, genomics, drug development and other fields [9, 10]. In the context of predicting antibiotic resistance, these algorithms can construct predictive models by analysing patients’ genetic information, clinical manifestations, medical records and other multi-source data, providing more accurate treatment recommendations for clinicians.

Despite the flourishing trend in research on ML predictions of antibiotic resistance in recent years, there remain some gaps in the research domain. Notably, although some systematic reviews have summarised and synthesised related studies, these reviews may be outdated in the rapidly evolving research field. A plethora of new studies and models has emerged in the last two years, offering fresh perspectives on our understanding of ML predictions for antibiotic resistance [11, 12]. Therefore, we conducted this updated meta-analysis, consolidating the latest research findings, to provide insights for the future clinical use of ML algorithms in predicting antibiotic resistance.

2. Methods

This meta-analysis adheres to the preferred reporting items for systematic reviews and meta-analyses guidelines [13].

2.1 Search strategy and literature selection

The search period of this study extends from the establishment of the database to 30 December 2023. Three electronic databases – PubMed, Embase and the Cochrane Central Register of Controlled Trials (CENTRAL) – were selected for comprehensive searches with no language restrictions. A combination of controlled vocabulary terms (MeSH or Emtree) and free-text terms was employed. Key terms mainly included antimicrobial resistance, ML and prediction. The full search strategy for each database is described in the supplementary material (Table S1). Additionally, manual screening of relevant references in reviews or meta-analyses within this field was conducted. The search process was performed independently by two researchers. Initially, duplicate records were removed using reference management software, followed by manual exclusion. Subsequently, articles were screened based on titles and abstracts to exclude studies unrelated to the topic. Finally, full-text reading was conducted to determine the ultimately included literature. In the case of disagreement between the two researchers, a third researcher facilitated resolution.

2.2 Inclusion and exclusion criteria

The PICOS principles were applied to define inclusion and exclusion criteria. Studies meeting the following conditions were included: 1) populations requiring antibiotics without obtaining susceptibility test results before prediction; 2) prediction of antibiotic resistance using ML algorithms; 3) no mandatory requirement for a control diagnostic method, and traditional risk scoring models could be used; 4) systematic reviews requiring diagnostic evaluation parameters, such as C-statistic, sensitivity and specificity, and meta-analyses requiring studies to calculate false negative, false positive, true negative and true positive data; and 5) observational study design. The exclusion criteria included the following: 1) studies with irrelevant outcomes; 2) studies not using ML algorithms; and 3) studies of irrelevant types, such as research letters, conference abstracts or reviews that did not report diagnostic evaluation parameters or did not focus on ML models.

2.3 Data extraction and bias risk assessment

Based on a standard data extraction table, two independent researchers extracted data, including study author, publication year, study design, region, data time span, data source, sample size, event occurrence rate, validation methods, specific algorithms used, number of input variables, types of predictive variables, observed outcomes, infection sites, diagnostic parameters and area under the receiver operating characteristic curve (AUC) value range. In cases where multiple algorithms were used in the same study, the model with the best performance (i.e. the highest AUC value) was prioritised in the meta-analysis. The prediction model risk of bias assessment tool (PROBAST) was employed for the methodological assessment of prediction model studies [14]. The PROBAST framework categorises potential bias into four domains: study participants, predictors, outcomes and analysis, with the assessment of the prediction model’s applicability covering the first three domains. Each domain that is assessed as ‘low risk’ is required to categorise the overall risk as ‘low risk’. If one domain is assessed as ‘high risk’, the overall risk is categorised as ‘high risk’. If one domain is assessed as ‘unclear’ while the other domains are assessed as ‘low risk’, the overall categorisation is ‘unclear’. Disagreements between the two researchers were resolved by a third senior researcher.

2.4 Statistical analysis

Data synthesis was performed using Stata SE 15.0 software. Sensitivity and specificity were calculated through 2 $\times$ 2 tables, followed by the presentation of the comprehensive diagnostic performance through forest plots and the summary receiver operating characteristic curve. Specific parameters included sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, diagnostic odds ratio and AUC value with its corresponding 95% CI. Heterogeneity was assessed using the Cochran Q test and $I^{2}$ statistic test. Funnel plots and Deeks’ test were utilised to evaluate the potential publication bias, with a $p$ -value of $<$ 0.05 indicating possible publication bias. Sensitivity analysis was conducted to assess the impact of individual studies on overall results, and Fagan plots were used to evaluate the clinical utility of prediction models.

3. Results

3.1 Literature search

The search and selection process for this study are illustrated in Fig. 1. A total of 3,210 records were obtained after searching three electronic databases, including 636 from PubMed, 2,565 from Embase and nine from CENTRAL. After removing duplicate literature, 2,893 independent electronic records were obtained. Based on the review of titles and abstracts, 2,848 irrelevant literature items were preliminarily excluded, leaving 45 articles for full-text reading and further screening. Due to reasons such as irrelevant outcomes ( $n=$ 15), non-ML algorithms ( $n=$ 3) and research letters/reviews/abstracts ( $n=$ 5), a total of 23 articles were excluded. Finally, 22 articles [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36] were included in the systematic review, with data from 10 articles being combined for the meta-analysis.

Figure 1.

PRISMA flow diagram of study selection. PRISMA: Preferred reporting items for systematic reviews and meta-analyses.

3.2 Characteristics of included studies

Table 1 presents the basic characteristics of the 22 studies included in the systematic review. The total sample size across these studies was 43,628. Most studies were conducted in the United States ( $n=$ 6), Greece ( $n=$ 3) and Spain ( $n=$ 3), although three studies did not report the region. The majority of studies (54.5%) were single-centre studies, 36.4% were multicentre studies and the status of 9.1% of the studies was unknown. The data collection period was 1996–2019, with some studies using data from public databases. The sample size ranged from 245 to 9,352 (two unknown), and the event occurrence rate in the study outcomes ranged from 5% to 71.0%. Model validation methods included cross-validation (54.5%), random splitting (22.7%), temporal validation (4.5%) and external validation (13.6%).

Table 1
Main characteristics of the included studies

Author, year	Region	Design	Time period	Data source	Sample size	Event rate (%)	Validation types
Feretzakis, 2020 [15]	Greece	Single-center	2017/1–2018/12	A public tertiary hospital	345	51.3	Cross-validation
Feretzakis, 2020-2 [16]	Greece	Single-center	2nd Semester 2018	A public tertiary hospital	5590	NA	Cross-validation
Feretzakis, 2021 [17]	Greece	Single-center	2019/1–2019/12	A public tertiary hospital	6086	71.0	Cross-validation
Garcia-Vidal, 2021 [18]	Spain	Single-center	2018/01–2017/12	A 700-bed university institution	349	37.8	Temporal validation
Goodman, 2016 [19]	USA	Single-center	2008/10–2015/03	Johns Hopkins Hospital clinical microbiology laboratory database	1288	15.1	Cross-validation
Goodman, 2019 [20]	USA	Single-center	2008/10–2015/03	Johns Hopkins Hospital clinical microbiology laboratory database	1288	15.1	Cross-validation
Goodman, 2019-2 [21]	USA	Single-center	2016/7–2017/7	Johns Hopkins Hospital MICU or SOT	2878	7.5	Cross-validation
Lee, 2021 [22]	China	Multi-center	2015/1–2019/12	A territory-wide database of three publicly funded acute hospitals in Hong Kong	5625	23.7	Cross-validation
Martiínez-Agüero, 2019 [23]	Spain	Single-center	2004–2016	The ICU of University Hospital of Fuenlabrada	2630 (maximum)	36.8–63.5	Cross-validation
Moran, 2020 [24]	UK	Multi-center	2010/1–2016/10	Three hospitals in Birmingham and Solihull	9352	NA	Random splitting
Noman, 2023 [25]	65 countries	Multi-center	NA	GenBank at NCBI	1200	44.7	Random splitting
Oonsivilai, 2018 [26]	Cambodia	Single-center	2013/2–2016/1	The Angkor Hospital for Children	245	27.8	Cross-validation
Ren, 2022 [27]	NA	Multi-center	NA	Giessen data and the public data	1509 (maximum)	5–34	External validation
Shang, 2000 [28]	USA	Multi-center	1996/3–1997/3	Fve medical facilities in the Pittsburgh area	472	26.1	Cross-validation

Table 1, continued
Author, year	Region	Design	Time period	Data source	Sample size	Event rate (%)	Validation types
Sick-Samuels, 2020 [29]	USA	Single-center	2009/6–2015/6	A freestanding 315-bed tertiary care pediatric hospital in Pittsburgh	689	31.5	Cross-validation
Sousa, 2019 [30]	Spain	Single-center	2015/1–2016/12	University Hospital Complex of Vigo	448	29.5	External validation
Valizadeh Aslani, 2020 [31]	NA	Multi-center	NA	Published data and PATRIC database	NA	NA	Cross-validation
Vazquez-Guillamet, 2017 [32]	USA	Single-center	2008/1–2015/4	A 1300-bed academic referral center	1618 (maximum)	6.6–28.6	NA
Visona, 2023 [33]	Switzerland	Multi-center	2016–2018	DRIAMS dataset	803	NA	Random splitting
Weis, 2022 [34]	Switzerland	Multi-center	2016–2018	DRIAMS dataset	803	NA	External validation
Yasir, 2022 [35]	NA	NA	NA	NCBI	410	NA	Random splitting
Yelin, 2019 [36]	Israel	NA	2007/7–2017/6	NA	NA	NA	Random splitting

NA: non-applicable; USA: the United States of America; MICU: medical intensive care unit; SOT: solid organ transplant; UK: the United Kingdom; NCBI: national center for biotechnology information; PATRIC: PAThosystems Resource Integration Center; DRIAMS: Database of Resistance Information on Antimicrobials and MALDI-TOF Mass Spectra.

Table 2

Characteristics of machine learning-based prediction models for sepsis

Author, year	Algorithm	No. of variables	Predictor types	Outcome of interest	Infection site	Performance metrics	AUC range
Feretzakis, 2020	SVC, SMO, KNN, J48, RF, RIPPER, MLP	3	Demographics, microbiological data	Resistance in ICU	Mixed	TP, FP, Pre, Rec, F1-score, MCC, AUC, PRC area	0.568–0.726
Feretzakis, 2020-2	LR, RF, KNN, J48, MLP	3	Demographics, microbiological data	Resistance in ICU	Mixed	TP, FP, Pre, Rec, F1-score, MCC, AUC, PRC area	0.721–0.758
Feretzakis, 2021	RF, JRip, MLP, Class. Regr., REPTree	3	Demographics, microbiological data	Resistance in ICU	Mixed	TP, FP, Pre, Rec, F1-score, MCC, AUC, PRC area	0.857-0.918
Garcia-Vidal, 2021	LR, RF, GBM, XGBoost	10	Demographics, underlying disease, drug use history, clinical variables, microbiological data	Resistance	Mixed	AUC, F1-score, Sen, Spe, PPV, NPV	0.782–0.790
Goodman, 2016	DT	5	Demographics, underlying disease, drug use history, clinical variables	ESBL	Blood	C-statistic, PPV, NPV	0.78
Goodman, 2019	DT, LR	5, 14	Demographics, underlying disease, drug use history, clinical variables	ESBL	Blood	C-statistic, AUC, PPV, PNV, Sen, Spe,	0.77, 0.89
Goodman, 2019-2	DT	3	Underlying disease, drug use history	CRO	Perirectal	C-statistic, PPV, NPV, Sen, Spe	0.57, 0.58
Lee, 2021	LR, DNN	15, 136	Demographics, underlying disease, drug use history, clinical variables	ESBL	Blood	AUC, PPV, NPV, Acc, F1-score, Sen, Spe	0.761
Martiínez-Agüero, 2019	LR, KNN, DT, RF, MLP	20–30	Demographics, clinical variables	AMG, CAR, CF4, PAP, POL, QUI	Blood	Acc, F1-score, Sen, Spe	NA
Moran, 2020	LR, GBDT	5, 10	Underlying disease, drug use history, clinical variables	AMO/CLA, PT	Urinary/ blood	AUC	0.70

Table 2, continued
Author, year	Algorithm	No. of variables	Predictor types	Outcome of interest	Infection site	Performance metrics	AUC range
Noman, 2023	LR, BioWeka, RF	NA	Whole genome sequence of Pseudomonas aeruginosa	Resistance	NA	Sen, Spe, Acc, Pre	NA
Oonsivilai, 2018	LR, DT, SVM, KNN	35	Demographics, underlying disease, drug use history, clinical variables	Resistance	Blood	AUC	0.74–0.85
Ren, 2022	LR, SVM, RF, CNN	NA	Whole genome sequence	Resistance	NA	AUC, Pre, Recall	0.69–0.96
Shang, 2000	LR, DNN	38	Demographics, underlying disease, drug use history, clinical variables, microbiological data	MRSA	Mixed	AUC	0.869, 0.928
Sick-Samuels, 2020	DT	6	Demographics, underlying disease, drug use history	Resistance	Blood	AUC, Sen, Spe	0.70
Sousa, 2019	DT	5	Demographics, underlying disease, drug use history, clinical variables	ESBL	Blood	C-statistic, PPV, NPV, Sen, Spe	0.76
Valizadeh Aslani, 2020	LR, RR, SVR, RF, AdaBoost, XGBoost	NA	Whole genome sequence	Resistance	NA	Acc	NA
Vazquez-Guillamet, 2017	LR, DT	4–6	Underlying disease, drug use history, clinical variables	PT, CE, ME	Blood	AUC	0.83, 0.63, 0.68
Visona, 2023	ResMLP	NA	MALDI-TOF mass spectra	Resistance	NA	AUC, Acc	0.47–0.87
Weis, 2022	LightGBM, MLP	NA	MALDI-TOF mass spectra	Resistance	NA	AUC	0.74–0.80
Yasir, 2022	10 ML classifiers	NA	Whole genome sequence of Pseudomonas aeruginosa	Resistance	NA	F1-score, Acc, Pre, Spe	NA
Yelin, 2019	LR, GBDT	10	Demographics, underlying disease, drug use history	Resistance	Urinary	AUC	0.7–0.83

NA: non-applicable; SVC: support vector clustering; SMO: sequential minimal optimization; KNN: k-NearestNeighbor; J48: RF: random forest; RIPPER: repeated incremental pruning to produce error reduction; MLP: multi-layer perceptron; TP: true positive; FP: false positive; MCC: a correlation coefficient calculated from all four values of the confusion matrix; AUC: area under the curve; PRC: The Precision-Recall Plot; LR: logistic regression; GBM: gradient boost machine; PPV: positive predictive value; NPV: negative predictive value; DT: decision tree; ESBL: extended spectrum $\beta$ -lactamase; CRO: carbapenem-resistant organism; DNN: deep neural network; AMG: aminoglycosides resistance; CAR: carbapenem resistance; CF4: 4-generation cephalosporins; PAP: broad-spectrum antibiotic resistance; POL: polymyxin resistance; QUI: quinolones resistance; GBDT: gradient-boosting decision tree; AMO/CLA: amoxicillin-clavulanic acid; PT: piperacillin-resistance; MRSA: methicillin-resistant Staphylococcus aureus.

Table 2 provides detailed information on the construction of the ML prediction models. For the 22 studies that met the criteria, various ML algorithms were used, including random forest (40.9%), decision tree (50%) and neural network (36.4%). Thirteen studies used demographics, 12 used underlying disease, 12 used drug use history, 10 used clinical variables, five used microbiological data and six used whole genome sequence or mass spectrometry data. The number of input variables in the models varied from 3 to 136. Thirteen studies defined the outcome as antibiotic resistance (mixed), whereas the remaining studies specified individual antibiotics. Regarding infection sites, the majority of studies (36.4%) focused on blood, six included mixed sites, one focused on urine, one on perirectal and six did not specify. There were variations in evaluation parameters, with four studies not reporting AUC; the AUC range was 0.47–0.928.

Table 3

Methodological evaluation of included studies by PROBAST

Author, year	ROB				Applicability			Overall
	Participants	Predictors	Outcome	Analysis	Participants	Predictors	Outcome	ROB	Applicability
Feretzakis, 2020	?	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Feretzakis, 2020-2	?	$+$	$+$	?	$+$	$+$	$+$	?	$+$
Feretzakis, 2021	?	$+$	$+$	?	$+$	$+$	$+$	?	$+$
Garcia-Vidal, 2021	–	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Goodman, 2016	?	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Goodman, 2019	?	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Goodman, 2019-2	$+$	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Lee, 2021	–	–	–	–	$+$	$+$	$+$	–	$+$
Martiínez-Agüero, 2019	–	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Moran, 2020	–	–	–	?	$+$	$+$	$+$	–	$+$
Noman, 2023	?	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Oonsivilai, 2018	–	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Ren, 2022	?	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Shang, 2000	?	–	–	–	$+$	$+$	$+$	–	$+$
Sick-Samuels, 2020	–	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Sousa, 2019	$+$	$+$	$+$	$+$	$+$	$+$	$+$	$+$	$+$
ValizadehAslani, 2020	?	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Vazquez-Guillamet, 2017	?	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Visona, 2023	$+$	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Weis, 2022	$+$	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Yasir, 2022	?	$+$	$+$	–	$+$	$+$	$+$	–	$+$
Yelin, 2019	–	$+$	$+$	–	$+$	?	$+$	–	?

PROBAST $=$ Prediction model Risk of Bias ASsessment Tool; ROB $=$ risk of bias. $+$ indicates low ROB/low concern regarding applicability; – indicates high ROB/high concern regarding applicability; and ? indicates unclear ROB/unclear concern regarding applicability.

3.3 Methodological assessment

Table 3 presents the bias risk assessment and applicability evaluation of the 22 included studies under the PROBAST framework. At the participants level, 11 studies were defined as ‘unclear’ due to unclear study design descriptions, and seven studies were defined as ‘high risk’. At the analysis level, the lack of use of a calibration plot for model evaluation in most studies resulted in high risk, leading to an overall higher risk assessment for the studies. In the applicability assessment, only one study raised concerns at the predictor level.

Figure 2.

Forest plot of machine learning-based models for the prediction of antimicrobial resistance.

3.4 Meta-analysis

The combined results of the 10 studies with available data are shown in Fig. 2. Overall, the sensitivity was 0.57 (95% CI: 0.42–0.70; $p<$ 0.001; $I^{2}=$ 99.7%), the specificity was 0.95 (95% CI: 0.79–0.99; $p<$ 0.001; $I^{2}=$ 99.9%), the positive likelihood ratio was 10.7 (95% CI: 2.9–39.5), the negative likelihood ratio was 0.46 (95% CI: 0.34–0.61), the diagnostic odds ratio was 23 (95% CI: 7–81) and the AUC was 0.78 (95% CI: 0.74–0.81; $p<$ 0.001), indicating good discriminative ability of ML algorithm models for antibiotic resistance (Fig. 3).

Figure 3.

SROC curve of ML-based models for the prediction of antimicrobial resistance. SROC: summary receiver-operating characteristic; ML: machine learning.

Figure 4.

Funnel plot of ML-based models for the prediction of antimicrobial resistance. ML: machine learning.

3.5 Publication bias and sensitivity analysis

The results of the publication bias analysis are shown in Fig. 4, where the Deeks’ funnel plot asymmetry test indicated a certain publication bias in the included studies ( $p=$ 0.02). A sensitivity analysis revealed that the exclusion of individual studies did not significantly impact the overall diagnostic results, indicating the stability of the study results, although the significance among studies did not disappear.

3.6 Clinical utility

After using ML algorithm-based antibiotic resistance prediction models, the post-test probability increased from 50% to 91% when the pre-test was positive, with a positive likelihood ratio of 11. Post-test probability for positive likelihood ratio was 91% (95% CI: 88%–93%); conversely, when the pre-test was negative, the post-test probability decreased from 50% to 31%, with a negative likelihood ratio of 0.46. Post-test probability for negative likelihood ratio was 31% (95% CI: 28%–34%) (Fig. 5).

Figure 5.

Fagan plot of ML-based models for the prediction of antimicrobial resistance. ML: machine learning.

4. Discussion

This study conducted a systematic review and meta-analysis to analyse the latest relevant research, evaluating the performance of ML algorithms in predicting antibiotic resistance. The main findings are as follows: 1) machine learning algorithm models exhibit good discriminative ability for antibiotic resistance, with weaker sensitivity but strong specificity; and 2) methodological assessment using the PROBAST criteria indicates that the included studies have a higher risk of bias due to factors such as lack of calibration and low event per variable, emphasising the need for high-quality studies to validate the results. In summary, this study affirms the potential application of ML algorithms in predicting antibiotic resistance, providing valuable decision support for clinicians.

In the context of predicting antibiotic resistance, the prolonged time required for traditional blood culture and susceptibility testing has become a significant obstacle to early intervention in infections by clinicians. Blood culture takes several days to confirm the bacterial species, and susceptibility testing requires additional time to assess antibiotic sensitivity, resulting in delayed treatment plans [37]. In this context, ML has shown significant advantages in predicting antibiotic resistance in recent years. One of the enormous advantages of ML is its ability to handle and analyse large-scale data, a task that is challenging for humans to accomplish in a short time. By utilising extensive patient data, clinical information and biochemical indicators, ML models can more comprehensively establish a patient’s infection status, providing more accurate predictions for early infection intervention [38]. Another advantage of ML in predicting antibiotic resistance lies in its ability to process microbiological data, such as sequencing data, which are crucial for predicting antibiotic resistance. Machine learning models can extract patterns and rules from these complex microbiological data, providing clinicians with more comprehensive infection information [39]. In contrast, traditional methods often struggle to handle such vast and complex datasets, whereas ML, with its powerful computing capabilities and algorithmic advantages, can better exploit this information, enhancing the predictive accuracy of antibiotic resistance. Moreover, ML possesses inherent adaptability, continuously learning and adjusting models to adapt to new microbiological changes and antibiotic resistance mechanisms. This flexibility makes ML more adaptive to addressing new challenges in infectious diseases, providing robust support for individualised treatment plans [40]. Overall, the ability of ML to predict antibiotic resistance is primarily derived from its processing capabilities for large-scale data and microbiological information, as well as its flexibility in model adaptability. This offers clinicians the possibility of earlier intervention in infections and more precise formulation of treatment plans, with the potential to improve the effectiveness of antibiotic use and reduce the risk of resistance development in the future.

Despite the potential advantages shown by ML in predicting antibiotic resistance, its practical application still faces a series of challenges and difficulties. Potential issues include the following. 1) Clinical complexity: clinical decision-making is an extremely complex process involving multiple factors, including individual differences, preferences, economic status and accessibility to medical services. Machine learning models may struggle to consider these complexities comprehensively, as some factors may be challenging to incorporate into the model or difficult to extract from big data sets. 2) Decision interpretability: machine learning models are often presented as black boxes, making it difficult to interpret their decision-making processes. In clinical decision-making, doctors and patients typically need to understand and trust the predictive results of the model. Poor interpretability may affect the acceptance of model recommendations by healthcare professionals, making it a crucial challenge to improve model interpretability. 3) Data quality and standardisation: machine learning models require high-quality and standardised input data. Healthcare data are often dispersed across different systems and formats, with missing values and errors, which can affect model performance; standardising and integrating this data is a time-consuming task. 4) Patient prognosis outcomes: despite some studies suggesting the impact of ML algorithms on patient prognosis [41, 42], the prognosis effect of these algorithms on predicting antibiotic resistance is currently unknown. Further research is needed to verify the feasibility, accuracy and actual clinical effects of these models before applying them in clinical settings. 5) Ethical and regulatory issues: applying ML algorithms to clinical decision-making involves ethical and regulatory issues. Issues such as privacy and security of patient data, as well as the review and regulation of ML algorithms, require the establishment of a robust legal and ethical framework. 6) Model generalisation ability: machine learning models are typically trained on specific datasets during development. The model’s ability to generalise to other clinical environments is a critical issue, as clinical data may vary significantly due to regional and population differences. 7) Acceptance by doctors and patients: the acceptance of ML models by doctors and patients is a key factor in the successful application of these models. If healthcare professionals and patients lack trust or have low acceptance of model recommendations, the actual effectiveness of the model may be limited. Therefore, although ML shows potential benefits in predicting antibiotic resistance, these challenges need to be addressed in practical clinical applications. Continuous research and innovation are necessary to gradually overcome these obstacles and achieve the sustainable and effective application of ML in antibiotic treatment decisions.

This study demonstrates significant novelty, primarily as the most recent and updated systematic review and meta-analysis. Although there were previous relevant studies, the rapidly increasing number of studies in the past 2 years, particularly in predicting antibiotic resistance, has rendered previous reviews unable to comprehensively reflect the current progress in the field [11, 12]. The use of ML algorithms for predicting antibiotic resistance has several advantages over traditional methods, such as faster and more accurate results, better handling of large-scale and complex data and more flexibility and adaptability to new information and scenarios. Machine learning algorithms can provide clinicians with timely and reliable predictions of antibiotic resistance, which can help them choose the most appropriate antibiotics and improve patient outcomes. However, the use of ML algorithms also poses some challenges and limitations, such as data quality and availability, model interpretability and explainability, ethical and regulatory issues and acceptance by doctors and patients. Therefore, future research should address these challenges and limitations and explore the best practices and standards for applying ML algorithms in clinical settings.

The advantages of our study as an updated meta-analysis are that we provide the most comprehensive and up-to-date evidence on the performance of ML algorithms for predicting antibiotic resistance and that we use rigorous methods and tools to assess the quality and heterogeneity of the included studies. Our study can help clinicians, researchers and policymakers to understand the current state and progress of ML applications in antibiotic resistance prediction and to identify the gaps and directions for future research.

However, we must carefully consider the limitations of the study to provide a more accurate interpretation of the results and guidance for future research. First, there is insufficient data in the studies included in this meta-analysis, with only a few studies providing adequate data for the combined analysis. This limitation restricts our assessment to only certain ML algorithms in predicting antibiotic resistance, failing to comprehensively reflect the current status of the entire field. Additionally, the results of the publication bias test suggest the possibility of selection bias, indicating that published studies are more likely to provide sufficient data, while unpublished studies may be excluded due to insufficient data, affecting the completeness of the results. Second, there is high heterogeneity in the meta-analysis of this study. The presence of heterogeneity may originate from various aspects, including differences in study design, sample characteristics and the diversity of the ML algorithms themselves. This suggests the need for a cautious interpretation of the results of combined effects in the meta-analysis, as heterogeneity may reflect the diversity of data sources and also indicate that existing data analysis methods may not be very applicable to the complexity of ML algorithms and predictive models. A meta-regression analysis to investigate the potential sources of heterogeneity in our meta-analysis, such as region, study design, data source, sample size, event rate, validation type, algorithm, number of variables, predictor types, outcome of interest and infection site is an important direction for future research, as it can help us understand how the performance of ML models for predicting antibiotic resistance may vary depending on the characteristics of the studies, the data and the models. Third, we must recognise the higher risk of bias in the included studies. In the methodological assessment, we used the recently developed PROBAST framework but still found a certain risk of bias in the studies. The main risk arises from insufficient descriptions of study cohorts and the lack of calibration, among other factors. Finally, a network meta-analysis is needed to compare the performance of different ML algorithms for predicting antibiotic resistance, such as logistic regression, random forest, gradient boosting, decision tree, support vector machine, k-nearest neighbour, multi-layer perceptron and deep neural network. This is an important direction for future research, as it can help us rank the algorithms based on their sensitivity and specificity and identify the most suitable algorithm for the prediction task. This indicates that in future research, stricter adherence to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis guidelines [43] is needed to ensure the quality and credibility of the studies.

5. Conclusion

In summary, this study indicates that the use of ML algorithms in predicting antibiotic resistance demonstrates robust discriminative capabilities, coupled with excellent specificity but relatively weaker sensitivity. Undoubtedly, this provides a new decision-making tool for the future application of antibiotics. However, given the high heterogeneity observed, further high-quality research in this field is essential to advance the clinical application of ML algorithms.

Availability of data and materials

All data generated or analyzed during this study are included in the article.

Author contributions

Conception and design of the work: Lv GD; Data collection: Wang YT; Analysis and interpretation of the data: Lv GD, Wang YT; Statistical analysis: Lv GD, Wang YT; Drafting the manuscript: Lv GD. All authors critically revised the manuscript and approved the final version.

Funding

The authors did not receive financial support for the research, authorship, or publication of this manuscript.

Supplementary data

The supplementary files are available to download from http://dx.doi.org/10.3233/THC-240119.

Footnotes

Acknowledgments

None to report.

Conflict of interest

None of the authors have any personal, financial, commercial, or academic conflicts of interest.

References

Ferri

Ranucci

Romagnoli

, et al. Antimicrobial resistance: A global emerging threat to public health systems. Crit Rev Food Sci Nutr. 2017; 57(13): 2857-2876. doi: 10.1080/10408398.2015.1077192.

Vasala

Hytönen

Laitinen

. Modern Tools for Rapid Diagnostics of Antimicrobial Resistance. Front Cell Infect Microbiol. 2020; 10: 308. doi: 10.3389/fcimb.2020.00308.

Holmes

Moore

Sundsfjord

, et al. Understanding the mechanisms and drivers of antimicrobial resistance. Lancet. 2016 Jan 9; 387(10014): 176-87. doi: 10.1016/S0140-6736(15)00473-0.

Kim

Maguire

Tsang

, et al. Machine Learning for Antimicrobial Resistance Prediction: Current Practice, Limitations, and Clinical Perspective. Clin Microbiol Rev. 2022 Sep 21; 35(3): e0017921. doi: 10.1128/cmr.00179-21.

Lycholip

Puronaitė

Skorniakov

Navickas

Tarutytė

Trinkūas

Burneikaitė

Kazėnaitė

Jankauskienė

. Assessment of the disease severity in patients hospitalized for COVID-19 based on the National Early Warning Score (NEWS) using statistical and machine learning methods: An electronic health records database analysis. Technol Health Care. 2023; 31(6): 2513-2524. doi: 10.3233/THC-235016.

Choudhury

. Predicting cancer using supervised machine learning: Mesothelioma. Technol Health Care. 2021; 29(1): 45-58. doi: 10.3233/THC-202237.

Ngiam

Khor

. Big data and machine learning algorithms for health-care delivery. Lancet Oncol. 2019; 20(5): e262-e273. doi: 10.1016/S1470-2045(19)30149-4.

Eraslan

Avsec

Gagneur

, et al. Deep learning: new computational modelling techniques for genomics. Nat Rev Genet. 2019; 20(7): 389-403. doi: 10.1038/s41576-019-0122-6.

Azizi

Culp

Freyberg

, et al. Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nat Biomed Eng. 2023 Jun; 7(6): 756-779. doi: 10.1038/s41551-023-01049-7.

10.

Schreiber

Singh

. Machine learning for profile prediction in genomics. Curr Opin Chem Biol. 2021; 65: 35-41. doi: 10.1016/j.cbpa.2021.04.008.

11.

Wang

Xiao

Yang

Wang

Yuan

. Clinical prediction models for multidrug-resistant organism colonisation or infection in critically ill patients: a systematic review protocol. BMJ Open. 2022 Sep 29; 12(9): e064566. doi: 10.1136/bmjopen-2022-064566.

12.

Tang

Luo

Tang

Song

Chen

. Machine learning in predicting antimicrobial resistance: a systematic review and meta-analysis. Int J Antimicrob Agents. 2022 Nov-Dec; 60(5-6): 106684. doi: 10.1016/j.ijantimicag.2022.106684.

13.

Liberati

Altman

Tetzlaff

, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009 Jul 21; 6(7): e1000100. doi: 10.1371/journal.pmed.1000100.

14.

Wolff

Moons

KGM

Riley

, et al. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med. 2019 Jan 1; 170(1): 51-58. doi: 10.7326/M18-1376.

15.

Feretzakis

Loupelis

Sakagianni

, et al. Using Machine Learning Techniques to Aid Empirical Antibiotic Therapy Decisions in the Intensive Care Unit of a General Hospital in Greece. Antibiotics (Basel). 2020 Jan 31; 9(2): 50. doi: 10.3390/antibiotics9020050.

16.

Feretzakis

Loupelis

Sakagianni

, et al. Using Machine Learning Algorithms to Predict Antimicrobial Resistance and Assist Empirical Treatment. Stud Health Technol Inform. 2020 Jun 26; 272: 75-78. doi: 10.3233/SHTI200497.

17.

Feretzakis

Sakagianni

Loupelis

, et al. Using Machine Learning to Predict Antimicrobial Resistance of Acinetobacter Baumannii, Klebsiella Pneumoniae and Pseudomonas Aeruginosa Strains. Stud Health Technol Inform. 2021 May 27; 281: 43-47. doi: 10.3233/SHTI210117.

18.

Garcia-Vidal

Puerta-Alcalde

Cardozo

, et al. Machine Learning to Assess the Risk of Multidrug-Resistant Gram-Negative Bacilli Infections in Febrile Neutropenic Hematological Patients. Infect Dis Ther. 2021 Jun; 10(2): 971-983. doi: 10.1007/s40121-021-00438-2.

19.

Goodman

Lessler

Cosgrove

, et al. A Clinical Decision Tree to Predict Whether a Bacteremic Patient Is Infected With an Extended-Spectrum β-Lactamase-Producing Organism. Clin Infect Dis. 2016 Oct 1; 63(7): 896-903. doi: 10.1093/cid/ciw425.

20.

Goodman

Lessler

Harris

, et al. A methodological comparison of risk scores versus decision trees for predicting drug-resistant infections: A case study using extended-spectrum beta-lactamase (ESBL) bacteremia. Infect Control Hosp Epidemiol. 2019 Apr; 40(4): 400-407. doi: 10.1017/ice.2019.17.

21.

Goodman

Simner

Klein

, et al. Predicting probability of perirectal colonization with carbapenem-resistant Enterobacteriaceae (CRE) and other carbapenem-resistant organisms (CROs) at hospital unit admission. Infect Control Hosp Epidemiol. 2019 May; 40(5): 541-550. doi: 10.1017/ice.2019.42.

22.

Lee

ALH

CCK

Lee

ALS

, et al. Deep learning model for prediction of extended-spectrum beta-lactamase (ESBL) production in community-onset Enterobacteriaceae bacteraemia from a high ESBL prevalence multi-centre cohort. Eur J Clin Microbiol Infect Dis. 2021 May; 40(5): 1049-1061. doi: 10.1007/s10096-020-04120-2.

23.

Martínez-Agüero

Mora-Jiménez

Lérida-García

, et al. Machine Learning Techniques to Identify Antimicrobial Resistance in the Intensive Care Unit. Entropy (Basel). 2019; 21(6): 603. doi: 10.3390/e21060603.

24.

Moran

Robinson

Green

, et al. Towards personalized guidelines: using machine-learning algorithms to guide antimicrobial selection. J Antimicrob Chemother. 2020; 75(9): 2677-2680. doi: 10.1093/jac/dkaa222.

25.

Noman

Zeeshan

Arshad

, et al. Machine Learning Techniques for Antimicrobial Resistance Prediction of Pseudomonas Aeruginosa from Whole Genome Sequence Data. Comput Intell Neurosci. 2023 Mar 1; 2023: 5236168. doi: 10.1155/2023/5236168.

26.

Oonsivilai

Luangasanatip

, et al. Using machine learning to guide targeted and locally-tailored empiric antibiotic prescribing in a children’s hospital in Cambodia. Wellcome Open Res. 2018 Oct 10; 3: 131. doi: 10.12688/wellcomeopenres.14847.1.

27.

Ren

Chakraborty

Doijad

, et al. Prediction of antimicrobial resistance based on whole-genome sequencing and machine learning. Bioinformatics. 2022 Jan 3; 38(2): 325-334. doi: 10.1093/bioinformatics/btab681.

28.

Shang

Lin

Goetz

. Diagnosis of MRSA with neural networks and logistic regression approach. Health Care Manag Sci. 2000; 3(4): 287-297. doi: 10.1023/a1019018129822.

29.

Sick-Samuels

Goodman

Rapsinski

, et al. A Decision Tree Using Patient Characteristics to Predict Resistance to Commonly Used Broad-Spectrum Antibiotics in Children With Gram-Negative Bloodstream Infections. J Pediatric Infect Dis Soc. 2020 Apr 30; 9(2): 142-149. doi: 10.1093/jpids/piy137.

30.

Sousa

Pérez-Rodríguez

Suarez

, et al. Validation of a clinical decision tree to predict if a patient has a bacteraemia due to a β-lactamase producing organism. Infect Dis (Lond). 2019 Jan; 51(1): 32-37. doi: 10.1080/23744235.2018.1508883.

31.

ValizadehAslani

Zhao

Sokhansanj

, et al. Amino Acid k-mer Feature Extraction for Quantitative Antimicrobial Resistance (AMR) Prediction by Machine Learning and Model Interpretation for Biological Insights. Biology (Basel). 2020 Oct 28; 9(11): 365. doi: 10.3390/biology9110365.

32.

Vazquez-Guillamet

Vazquez

Micek

, et al. Predicting Resistance to Piperacillin-Tazobactam, Cefepime and Meropenem in Septic Patients With Bloodstream Infection Due to Gram-Negative Bacteria. Clin Infect Dis. 2017 Oct 30; 65(10): 1607-1614. doi: 10.1093/cid/cix612.

33.

Visonà

Duroux

Miranda

, et al. Multimodal learning in clinical proteomics: enhancing antimicrobial resistance prediction models with chemical information. Bioinformatics. 2023 Dec 1; 39(12): btad717. doi: 10.1093/bioinformatics/btad717.

34.

Weis

Cuénod

Rieck

, et al. Direct antimicrobial resistance prediction from clinical MALDI-TOF mass spectra using machine learning. Nat Med. 2022 Jan; 28(1): 164-174. doi: 10.1038/s41591-021-01619-9.

35.

Yasir

Karim

Malik

, et al. Application of Decision-Tree-Based Machine Learning Algorithms for Prediction of Antimicrobial Resistance. Antibiotics (Basel). 2022 Nov 10; 11(11): 1593. doi: 10.3390/antibiotics11111593.

36.

Yelin

Snitser

Novich

Katz

Tal

Parizade

Chodick

Koren

Shalev

Kishony

. Personal clinical history predicts antibiotic resistance of urinary tract infections. Nat Med. 2019 Jul; 25(7): 1143-1152. doi: 10.1038/s41591-019-0503-6.

37.

Fabre

Carroll

Cosgrove

. Blood Culture Utilization in the Hospital Setting: a Call for Diagnostic Stewardship. J Clin Microbiol. 2022 Mar 16; 60(3): e0100521. doi: 10.1128/JCM.01005-21.

38.

Farhat

Athar

Ahmad

, et al. Antimicrobial resistance and machine learning: past, present, and future. Front Microbiol. 2023 May 26; 14: 1179312. doi: 10.3389/fmicb.2023.1179312.

39.

Khaledi

Weimann

Schniederjans

, et al. Predicting antimicrobial resistance in Pseudomonas aeruginosa with machine learning-enabled molecular diagnostics. EMBO Mol Med. 2020 Mar 6; 12(3): e10264. doi: 10.15252/emmm.201910264.

40.

Huang

Xue

, et al. Identification of potent antimicrobial peptides via a machine-learning pipeline that mines the entire space of peptide sequences. Nat Biomed Eng. 2023 Jun; 7(6): 797-810. doi: 10.1038/s41551-022-00991-2.

41.

Wijnberge

Geerts

Hol

, et al. Effect of a Machine Learning-Derived Early Warning System for Intraoperative Hypotension vs Standard Care on Depth and Duration of Intraoperative Hypotension During Elective Noncardiac Surgery: The HYPE Randomized Clinical Trial. JAMA. 2020 Mar 17; 323(11): 1052-1060. doi: 10.1001/jama.2020.0592.

42.

Strömblad

Baxter-King

Meisami

, et al. Effect of a Predictive Model on Planned Surgical Duration Accuracy, Patient Wait Time, and Use of Presurgical Resources: A Randomized Clinical Trial. JAMA Surg. 2021 Apr 1; 156(4): 315-321. doi: 10.1001/jamasurg.2020.6361.

43.

Collins

Reitsma

Altman

, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015 Jan 7; 350: g7594. doi: 10.1136/bmj.g7594.

Machine learning-based antibiotic resistance prediction models: An updated systematic review and meta-analysis

Abstract

BACKGROUND:

OBJECTIVE:

METHODS:

RESULTS:

CONCLUSION:

Keywords

1. Introduction

2. Methods

2.1 Search strategy and literature selection

2.2 Inclusion and exclusion criteria

2.3 Data extraction and bias risk assessment

2.4 Statistical analysis

3. Results

3.1 Literature search

Table 1 Main characteristics of the included studies

3.6 Clinical utility

5. Conclusion

Availability of data and materials

Author contributions

Funding

Supplementary data

Footnotes

Acknowledgments

Conflict of interest

References

Table 1
Main characteristics of the included studies