Decoding sepsis: A technical blueprint for an algorithm-driven system architecture

Introduction

Sepsis is a life-threatening condition that occurs when the body has an extreme response to infection.^1,2 It is a time-critical medical emergency that can lead to organ failure and possible death. One in five patients with sepsis deteriorates within 72 h after admission.^2,3 Sepsis is one of the most frequent and costly adverse health events during healthcare delivery, estimated to make up 2.65% of a national healthcare budget and approximately 0.33% of gross national product. ⁴ The emergency department (ED) is usually the first place of medical assistance for septic patients. Thus, emergency clinicians play a crucial role in the early diagnosis and treatment.

Sepsis remains difficult to diagnose as it can cause organ dysfunction that is life-threatening without overt signs of underlying infection.^1,5 Several healthcare organisations have developed and implemented quality and safety programs to improve the detection and management of sepsis. In New South Wales, Australia, the Clinical Excellence Commission's SEPSIS-KILLS program, ⁶ a paper-based clinical pathway, is the standard practice, which assists clinicians in recognising and responding to patients at risk of developing sepsis. Clinicians who recognise patients with sepsis can manually turn on the Sepsis Alert (a Systemic Inflammatory Response Syndrome (SIRS)-based program), the existing tool in the electronic medical record (eMR), to indicate the initiation of sepsis pathway. Patients presenting to EDs can sometimes have long waiting times before treatment can be commenced. During this period, the limited information collected about patients during the clinical triage process can lead to patients with sepsis or at high risk of developing sepsis being missed, particularly in busy overcrowded EDs. Therefore, due to the busy nature of EDs, the risk of sepsis may still be missed in the waiting room.

The Sepsis-3 definition ¹ recognises that scoring with limited available information such as quick Sequential Organ Failure Assessment (qSOFA) could be trailed in ED settings instead of relying on SIRS-based algorithms ¹ and pathology results e.g. lactate. ⁷ But past evaluations of the qSOFA screening algorithm have highlighted the poor performance of the simplistic algorithm. ⁸

Most machine learning (ML)-based sepsis detection algorithms have been developed on Intensive Care Unit (ICU) datasets, inpatients versus the ED.^5,9–12 Our sepsis algorithm is referred to as the Sepsis risk AI algorithm For Emergency department WAITing room^13,14 (SAFE-WAIT). Our ML operations (ML Ops) architecture integrates eMR triage assessments, vital signs, patient demographics to calculate sepsis risk in the ED waiting room, where limited patient information is available. This Australian-first clinical AI tool^8,15 leverages vital observations collected at triage, such as blood pressure, pulse, respiratory rate (RR), temperature, and oxygen saturation, combined with demographic data to predict the risk of developing sepsis. The calculated sepsis risk for each patient is displayed as a percentage and is flagged as Low risk (green), Moderate risk (yellow), and High risk (red) to assist clinicians in their decision-making process. We provided a read-only proprietary visualisation dashboard for clinical decision support that we called ‘SAFE-WAIT tool’.^13,14 Our ML Ops architecture is designed to support model validity, scalability and security throughout the system. This paper outlines our pilot of near real-time sepsis risk-stratification solution architecture in one hospital (in Western Sydney) suitable for ED waiting rooms.

Method

The system architecture (Figure 1) was designed for near real-time ingestion of Health Level Seven (HL7) messages, processing and visualisation of patient data for sepsis risk scoring in the ED waiting room. It leveraged an event-driven and serverless design combining both cloud components and on-premises infrastructure.

OPEN IN VIEWER

Figure 1.

System architecture, in a secure Virtual Private Cloud (VPC) environment for near real-time sepsis detection, data storage, processing and retrieval via AWS Cloud, with the user interface in SAFE-WAIT tool. AWS: Amazon Web Services; SAFE-WAIT: Sepsis risk AI algorithm For Emergency department WAITing room.

Sepsis risk AI algorithm For Emergency department WAITing room model development

Data source and collection

The study was conducted using a large dataset of ED patients (N = 556,652) drawn from four hospitals within the Western Sydney Local Health District (WSLHD), which serves a multicultural population of over one million residents. Each unique presentation to the ED during the study period (January 2017 to December 2019) was treated as a separate encounter. From the initial cohort, those under the age of 16 (N = 72,035) were excluded (Figure 2). Triage category 1 patients were excluded under the assumption that they were already receiving critical care. Where available, historical medical and medication records were also included to enrich the clinical context.

OPEN IN VIEWER

Figure 2.

The flowchart illustrates the stepwise inclusion and exclusion criteria applied to emergency department encounters to formalise the dataset.

Among the eligible ED encounters, 104,935 were identified as having a suspected infection, defined by either a culture order within 24 h (N = 99,422) or a rapid viral test order within 24 h (N = 15,584). A subset of 22,628 encounters met criteria for suspected sepsis, based on classification into the Adult Sepsis Pathway Red Zone (N = 6660) or Yellow Zone (N = 20,360). Of these, 28,320 encounters had a SOFA score ≥2, and 1575 were diagnosed with sepsis based on either an explicit diagnosis (N = 1300) or infection plus ICU admission or death within 24 h (N = 302). The final number of confirmed sepsis cases used for model development was 29,895, representing a prevalence of 6.17% in the dataset of ED encounters.

Data labelling

We calculated new fields to allow for labelling of the data based on the Sepsis-3 definition selected by our subject matter experts. The following criteria were utilised for data labelling, and we defined the following fields:

Adverse Outcome

Time to death less than or equal to 24 h, OR

Time to ICU admission less than or equal to 24 h.

Sepsis in Discharge Note

Identified by the presence of the following pattern in the discharge diagnosis: ‘sepsis’ or ‘septic’.

Viral Infection in Discharge Note

Detected by the presence of the following patterns in the discharge diagnosis: ‘infect’, ‘abscess’, ‘pneumonia’, ‘viral’, or ‘virus’.

SOFA Total (Sequential Organ Failure Assessment)

Calculated as the sum of individual organ scores including respiratory, platelet, liver, cardiac, brain, and kidney.

Suspected Infection

Positive culture result within 24 h, OR

Rapid tests indicating infections (A, B, or RSV) within 24 h.

Diagnosed Sepsis

If Suspected Infection is not true, or if Suspected Infection is true and SOFA Total is less than 2, and Sepsis is noted in the discharge summary, OR

If Suspected Infection is not true, or if Suspected Infection is true and SOFA Total is less than 2, and Adverse Outcome is true, and Viral Infection is noted in the discharge summary.

Confirmed Sepsis

Suspected Infection is true, and SOFA Total is greater than or equal to 2; OR

Diagnosed Sepsis is true.

The definition of Suspected Infection, Diagnosed Infection and Confirmed Sepsis provided a criterion for what was defined as a ground truth sepsis case for a given patient in the ED.

Data preprocessing

A comprehensive cleaning and transformation pipeline was implemented to prepare the raw data for modelling. Initially, rows with entirely missing values were excluded. For each patient encounter, vital signs (e.g. Glasgow Coma Scale (GCS), RR, systolic blood pressure (SBP), diastolic blood pressure (DBP), mean arterial pressure (MAP), tympanic temperature (TEMP), oxygen saturation (O2SATS), heart rate (PULSE), see Table 1) were often recorded multiple times. Each vital sign was recorded up to three times representing different time points. If the primary value (e.g. GCS_1) was missing, we checked if GCS_2 or GCS_3 was available and used one of those instead.

OPEN IN VIEWER

Table 1.

Model features.

Model features	Fields used for labelling
Systolic blood pressure (SBP)*	Presenting problem
Diastolic blood pressure (DBP)*	Medical history
Heart rate (PULSE)	Discharge notes
Tympanic temperature (TEMP)	Rapid tests (Influenza A, B, or RSV)
Respiratory rate (RR)	Positive culture results
Oxygen saturation (O2SATS)*	Time to death
Glasgow Coma Scale (GCS)/ AVPU*	Time to ICU admission
Age	Time of year/season
Sex	Hospital
Triage category	SOFA (respiratory, platelet, liver, cardiac, brain, and kidney)

Asterisk (*) indicates features used in the SOFA score.

SOFA: Sequential Organ Failure Assessment; ICU: Intensive Care Unit.

When primary measurements remained unavailable, clinically informed substitutions were applied, for example, MAP was calculated from SBP and DBP, and tympanic temperature was estimated from oral readings.

Text-based discharge diagnoses were parsed using regular expressions to identify mentions of sepsis or infection for sepsis case labelling purposes.

Although no outlier detection methods were applied, thresholds were applied to clinical features to mitigate the influence of extreme and impossible values e.g. age < 150.

Model training

To evaluate model performance, the dataset was randomly partitioned into training, validation, and test sets. A reproducible shuffle was applied using a fixed random seed, after which 70% of the data was allocated for training. The remaining 30% was evenly divided, with 15% used for validation and 15% for testing. Patient-level separation was applied, to assign all encounters from each patient to the corresponding split.

Features and model selection

Within a clinical AI model definition, we used domain knowledge to select limited features only available at initial triage, as detailed in Table 1. The chosen data-driven features were informed by the contextual implementation within the ED waiting room where features such as invasive blood tests were not readily available post patient triage.[Supplementary Material – Human factors study]. Therefore, in near real-time detection, we relied on our model to learn from the pattern using SOFA scores and only input vital signs in Table 1 as a new observation.

To interpret feature importance, SHapley Additive exPlanations (SHAP) values ¹⁶ were computed, revealing the relative contribution of each feature to the model's predictions. The most influential features included O2SATS and age, both with SHAP values of 0.046, followed closely by TEMP (0.042), GCS (0.031), and SBP (0.029). Other contributing features were PULSE DBP, RR, and categorical variables such as gender and triage category.

In our study and given the nature of the problem, we selected Extreme Gradient Boosting (XGBoost). A comparison of its performance against baseline models can be found in the results section.

Each selected model underwent an exhaustive search over a predefined parameter space to identify the optimal configuration. Techniques such as grid search were employed to efficiently explore the hyper-parameter space and identify combinations that yield the best performance metrics on validation data.

Result

Model development – baseline comparison

To justify the adoption of XGBoost over simpler methods, we compared its performance with two alternative models: SageMaker Linear Learner and a baseline Logistic Regression model with age group thresholds (Tables 2 and 3).

OPEN IN VIEWER

Table 2.

Performance comparison of three machine learning models for sepsis prediction.

Model name	Probability threshold	Accuracy	Sensitivity	Specificity	Precision	F1-score
SageMaker XGBoost	0.11	0.84	0.77	0.84	0.84	0.8
SageMaker Linear Learner	0.11	0.83	0.72	0.84	0.84	0.77
Logistic Regression	[0.01, 0.04, 0.14]	0.62	0.8	0.61	0.61	0.69

XGBoost: Extreme Gradient Boosting.

OPEN IN VIEWER

Table 3.

Performance of three machine learning models for sepsis prediction across different age groups.

Model name	Age group	Probability threshold	Accuracy	Sensitivity	Specificity	Precision	F1-score
XGBoost	16–35	0.014	0.76	0.80	0.76	0.76	0.78
XGBoost	36–65	0.045	0.76	0.81	0.75	0.75	0.78
XGBoost	66–115	0.145	0.68	0.80	0.65	0.65	0.72
Linear Learner	16–35	0.014	0.43	0.88	0.42	0.42	0.57
Linear Learner	36–65	0.045	0.64	0.82	0.63	0.63	0.71
Linear Learner	66–115	0.145	0.67	0.75	0.65	0.65	0.70
Logistic Regression	16–35	0.014	0.60	0.78	0.59	0.59	0.67
Logistic Regression	36–65	0.045	0.66	0.80	0.66	0.66	0.72
Logistic Regression	66–115	0.145	0.60	0.81	0.55	0.55	0.66

XGBoost: Extreme Gradient Boosting.

While the logistic model achieved reasonable sensitivity (0.80) across all patients, its specificity (0.61) and precision (0.61) were notably lower than those of XGBoost (specificity 0.84, precision 0.84). Linear Learner performed comparably to XGBoost in some global metrics. The XGBoost model achieved the highest overall accuracy (0.84) and F1-score (0.80), with balanced sensitivity and specificity. Logistic regression, while achieving high sensitivity (0.80), showed lower specificity and precision, indicating more false positives. All models were evaluated at their respective probability thresholds. These results suggest that XGBoost not only matches but exceeds the performance of traditional models, justifying its use for this task.

Model development – age group model performance

Subgroup analysis using age-specific probability thresholds revealed that XGBoost maintained a favourable balance between sensitivity and specificity across all age groups, achieving the highest accuracy and F1-score in younger populations. Linear Learner exhibited a marked drop in specificity for younger patients, falling to 0.42 despite achieving a high sensitivity of 0.88, and showed lower overall accuracy in the 16–35 age group. Logistic Regression demonstrated relatively consistent sensitivity across groups but persistently low specificity, particularly in the oldest cohort, and reduced precision, indicating a higher rate of false positives. Collectively, these results indicate that XGBoost offers the most favourable trade-off between sensitivity and specificity, delivers high precision, and maintains consistent performance across subgroups, outperforming simpler baseline models in both aggregate and stratified analyses.

ROC analysis (Figure 3) was performed across three age groups to evaluate the discriminative performance of the XGBoost binary classifier. For the 16–35 age group, the model achieved an AUC of 0.864 with an optimal threshold of 0.0224, indicating great sensitivity at low probability cutoffs. In the 36–65 cohort, the model performed similarly well with an AUC of 0.867 and an optimal threshold of 0.0556. For the 66–115 age group, the model maintained strong performance with an AUC of 0.806 and an optimal threshold of 0.1902. These results demonstrate the model's robustness and generalizability across diverse age groups.

OPEN IN VIEWER

Figure 3.

Receiver operating characteristic (ROC) analysis curves for XGBoost classifiers across age groups (16–35, 36–65, 66–115), showing strong performance in all cohorts (AUCs: 0.864, 0.867, 0.806). XGBoost: Extreme Gradient Boosting.

Architecture tests

As the sepsis detection algorithm went live in the ED, a total of (N = 1,802,370) HL7 messages were processed between August 2021 and June 2024, with 99.7% successfully processed and 0.3% failing (Table 1). Success rates were consistently high across years. The largest volume of messages occurred in 2023, with (N = 724,884) processed, of which (N = 5685) failed (99.2% success rate). The failures here are due to downtime and code set mismatches from updates or missing data in the HL7 messages.

A summary of key metrics of latency and invocation is provided in Table 4, where the sepsis-data-transformation function processes incoming HL7 messages for scoring. It performed an average execution time of 3.3 s over a 30-day period. While the minimum duration was 0.75 s, peak execution times reached 10.5 s. This variability is likely due to factors such as fluctuating payload sizes, network latencies, and the time required for data retrieval from S3 before transformation processes.

OPEN IN VIEWER

Table 4.

Latency and invocation metrics.

Metric	1 Day			30 Days
	Min	Max	Avg	Min	Max	Avg
Duration (seconds)	0.73	0.98	0.76	0.75	10.55	3.30
Invocations	1069	6965	2267	64,365	70,414	67,525

Discussion

The system architecture demonstrates robust performance for near real-time sepsis detection with key benefits. Fault tolerance and high availability are ensured through multi-AZ RDS deployments, enabling disaster recovery and regular backups with options for cross-region replication. The serverless design ensures seamless scalability, enabling the system to process large volumes of HL7 messages without performance degradation. Lambda and S3 dynamically handle fluctuating demands, offering both efficiency and cost savings. The event-driven design ensures compute resources are used only when triggered, eliminating costs associated with idle infrastructure. However, cost analysis reveals opportunities to explore alternatives for data storage such as on-premises databases or DynamoDB.

The architecture is well suited to be implemented for additional facilities in the future by leveraging serverless components, with minimal re-configuration of processing and scoring of HL7 messages. System design embeds modularity and flexibility at its core, allowing the migration of its ‘brain’ – the cloud infrastructure and core components e.g. XGBoost model to accommodate institutional needs. This means that the architecture can transition between different cloud providers or hybrid setups based on cost, compliance, or operational requirements. This enables integration of additional features or hospital systems without requiring significant changes to the existing infrastructure.

Cost analysis shows that for the current workflow, RDS accounts for 73% of total system expenses, highlighting a key opportunity for total cost of ownership reduction. ²⁰ To address this, the system could explore hybrid infrastructure alternatives such as Amazon Aurora Serverless or self-managed PostgreSQL database servers as it can save costs when DB is idle and automatically scale to demand. Additional savings can be achieved by using RDS Reserved Instances.

The average message processing latency was 3.3 s, with a maximum of 10.5 s over a 30-day period. This level of latency did not impact immediate clinical flow. However, the more significant limitation was the small proportion of failed messages. In 2023, the system processed a large volume of messages (N = 724,884), with a 99.22% success rate prior to 2024 with its’ 99.99% success rate (see Table 5). The (N = 5685) failed messages were primarily due to downtime or/and code set mismatches from updates or missing data in the HL7 messages. Assuming each failed message corresponds to a unique patient encounter, and given a sepsis detection rate of 6.17%, approximately 351 potential sepsis cases may have been missed due to these failures in 2023. It is important to note that not every HL7 message corresponds to a unique patient; a single patient encounter may generate multiple HL7 messages. Therefore, the number of failed messages represents an upper bound, and the actual number of patients potentially affected is likely much lower.

OPEN IN VIEWER

Table 5.

Processed HL7 messages: Success vs failure.

Year	2021	2022	2023	2024	Total
Total messages	183,672	535,232	730,569	418,980	1,866,453
Fail	188	98	5685	12	5983
Success	183,484	535,134	724,884	418,968	1,866,453
Success (%)	99.90	99.98	99.22	99.99	99.68%

This further emphasises that architectural systems need to be robust in detecting downtime or unprocessed messages that failed to go through. Following the improvements in 2024, the message success rate increased to 99.99%, substantially reducing the potential risk of missed detections (N = 12,<1patient) and further supporting reliable clinical decision support.

S3 event notifications are not always instantaneous and can be delayed or missed which may lead to latency in downstream workflows. When a high volume of S3 events occurs in rapid succession, Lambda can become overwhelmed. This was observed when our system demonstrated a higher latency or missed messages as previously discussed. Introducing a Simple Queue Service queue between S3 and Lambda can help with throttling, buffering, and retry logic. If a Lambda function fails after being triggered, the event is recorded but not reprocessed by default. To address this, future versions of the system should incorporate a retry mechanism, for example using a Dead Letter Queue to capture and retry failed events. ²¹

Our SAFE-WAIT tool utilises ML model development that is targeted for our cohort to address the limited features available in the ED waiting room and risk-stratification using age groups. The use of largely proprietary sepsis alerting systems has resulted in mixed results, largely negative except a recent study by Cull and colleagues. ⁹ In an uncontrolled before-and-after study, Cull et al. (2023) observed a 44% reduction in the odds of sepsis-related mortality after implementation of the Epic Sepsis Model alert (threshold ≥ 5). ⁹ Most EPIC sepsis alert models use the developer recommended^10,12 threshold ≥ 6, and recent external validation studies show sub-optimal diagnosis alerting assistance missing 67% patients with sepsis with AUC = 0.63, ¹⁰ no better than current clinical practice and with reports of potential alert fatigue. ¹⁰ This shows the vital need for prospective multi-site validation when scaling up the solution. Current research including our study indicates that targeted validation ²² should be required for any sepsis alerting system prior to deployment. The use of silent trials as part of prospective multi-site validation is a sensible bridge prior to wide-spread deployment and should be evaluated during implementation. ¹⁹

Limitations

The SAFE-WAIT tool was evaluated only at a single site ED due to the nature of the pilot. The on-premises proprietary dashboard for clinical decision support was not integrated into the eMR system but existed as a separate web-app visualisation tool, outside the usual clinical workflow.

SageMaker is used for inference, but there is no monitoring for data drift implemented but in future model monitoring should be considered to detect various data drifts like input features, data quality, inference etc. to maintain model performance including when high volume issues remained unprocessed.

Latency of downstream information processing might impact urgent clinical decision-making and the evaluation of potential sepsis cases, and the number of cases might potentially be underestimated by SAFE-WAIT tool, as discussed.

Conclusion

Our study has demonstrated a piloted system architecture that is robust, scalable, serverless and secure for near real-time sepsis detection and management. By leveraging an event-driven, serverless design, the system ensures seamless data processing, high availability and efficient scalability potential across multiple hospitals. However, cost and latency considerations remain a challenge; the architecture's modular design provides flexibility for future optimisation.

We have demonstrated that it is possible to develop a model that is robust within current clinical workflows and considers types of data availability within the ED when identifying cases of sepsis in the waiting room, integrating this model into our architecture for near real-time insights. Human factors insights are critical to identify model usability and the process of comprehensive evaluation should have a silent phase leading to an implementation pilot to optimise human–computer interaction and behavioural insights for integration and adoption. This technical blueprint for an Australian-first clinical AI ED deployment not only addresses current clinical needs but also lays a foundation for enabling broader deployment in adoption of targeted validation and more advanced scalable features in the future.

Supplemental Material

Supplemental Material

Supplemental Material