Comparison of Sepsis Screening Tools' Ability to Detect Sepsis Accurately

Abstract

Background:

Sepsis is the primary cause of perioperative mortality among general surgery patients and is the leading cause of death in non-cardiac intensive care units. To address this issue, the Surviving Sepsis Campaign Guidelines advocate for sepsis screening. However, there is little information in the current medical literature to suggest which sepsis screening tool is optimal. The purpose of this study was to compare a sepsis screening tool that we have validated and published previously, the Sepsis Screening Score (SSS), with a commercially available sepsis screening tool, the St. John's Sepsis Agent (SJSA) developed by Cerner (Kansas City, MO).

Methods:

This prospective observational study compares the accuracy of the SSS with that of the SJSA in the same patient population. The SSS was performed on each patient in our surgical intermediate care unit (SIMU) twice daily. The SJSA monitored these same patients continuously via the electronic medical record (EMR). Epidemiologic data related to sepsis were collected prospectively, and the performance characteristics of the two tests were compared using the two-sample test of proportions.

Results:

A total of 348 patients were included in the study, and 47 (13.5%) of these patients developed sepsis. The SJSA was determined to have a sensitivity of 44.7%, a specificity of 84.7%, a positive predictive value (PPV) of 31.3%, and a negative predictive value (NPV) of 90.7%, while the SSS was determined to have a sensitivity of 74.5%, a specificity of 86.4%, a PPV of 46.1%, and an NPV of 95.6%. The differences in sensitivity (p < 0.001), PPV (p < 0.001), and NPV (p = 0.011) were found to be statistically significant.

Conclusion:

Despite the fact that SJSA had constant surveillance over patients' EMRs, it still detected fewer septic patients than the SSS, which was performed twice per day. The difference in sensitivities and NPVs between the two tests is of particular importance, because this indicates that the SSS is more effective in identifying patients with sepsis. This study establishes a basis for the utilization of the SSS instead of the SJSA.

Despite extensive research and quality improvement efforts, sepsis remains the primary cause of perioperative mortality among general surgery patients and the leading cause of death in non-cardiac intensive care units [1]. The development of sepsis in the surgical patient has been linked to increased hospital length of stay (LOS), increased incidence of organ failure, increased likelihood of discharge to a long-term acute care facility or skilled nursing facility, and increased mortality [2]. Additionally, with an annual aggregate health care cost of over $20 billion in the United States, it is the most expensive condition billed to Medicare [3].

The significant morbidity, mortality, and cost of sepsis could be reduced with proper identification of sepsis in its early stages. This is because sepsis progresses along a continuum from sepsis, defined as whole-body inflammation stemming from an infection, to severe sepsis, defined as sepsis with organ dysfunction or tissue hypoperfusion, to septic shock, which is severe sepsis associated with hypotension that is non-responsive to the administration of intravenous fluids [4]. If a patient progresses from sepsis to septic shock, the mortality rate increases to greater than 40% [5]. Many of the interventions to treat sepsis, such as the administration of broad spectrum antibiotics, are time-sensitive and are dependent upon the early recognition of sepsis in order to be maximally effective. Because early intervention relies upon early and accurate identification of sepsis, we believe that sepsis screening is an integral part of a comprehensive sepsis treatment program.

One obstacle in the early identification of sepsis is the wide variability in the clinical presentation of a patient with sepsis. This can be particularly challenging in the surgical patient when common clinical scenarios such as oliguria and altered mental status are often attributed to clinical entities other than sepsis. Whereas patients with sepsis may present with grossly elevated temperature, they may also present with body temperatures well below normal levels, especially in the elderly population. To address this issue, the Surviving Sepsis Campaign Guidelines were updated in 2012 to advocate for routine screening of patients [4], so that sepsis can be identified in its early, more treatable stages. However, there is still little information in the medical literature to suggest which sepsis screening tool is optimal.

We have constructed and evaluated a sepsis screening tool previously, the Sepsis Screening Score (SSS), which utilizes concise parameters of the systemic inflammatory response system (SIRS) criteria to identify patients with sepsis (Table 1) [6]. The SSS quantifies the parameters into an overall numerical value, providing a means to triage those identified with sepsis and allowing for a high degree of sensitivity and specificity. If a patient scores greater than or equal to four, physicians are advised to assess the patient for the presence of an infection and then begin an evidence-based protocol promptly to treat aggressively for sepsis. The SSS demonstrated impressive performance characteristics in a population of general surgical patients, yielding a sensitivity of 96.5%, specificity of 96.7%, a positive predictive value (PPV) of 80.2%, and a negative predictive value (NPV) of 99.5% [6]. Furthermore, we have previously utilized this screening tool to decrease sepsis-related mortality in a surgical intensive care unit (SICU) by one third [6].

Table 1.

The Sepsis Screening Score

	0	1	2	3	4
Heart rate	70–109		55–69110–139	40–54140–179	≤39≥180
Temp (°C)	36–38.4	34–35.938.5–38.9	32–33.9	30–31.939–40.9	≤29.9≥41
Temp (°F)	96.8–101.1	93.1–96–6101.2–1–2.0	89.6–93.0	86–89.5102.1–105.6	≤85.9≥105.7
Respiratory rate	12–24	10–1125–34	6–9	35–49	≤5≥50
Latest WBC count	3–14.9	15–19.9	1–2920–39.9		<1≥40
Acute change in mental status	No	Yes
SIRS Score (total points)

The Sepsis Screening Score quantifies the parameters into an overall numerical value, providing a means to triage those identified with sepsis and allowing for a high degree of sensitivity and specificity. If a patient scores greater than or equal to four, physicians should begin evidence-based protocol to treat for sepsis aggressively.

WBC = white blood cell; SIRS = systemic inflammatory response system.

In addition to the SSS, there are other screening tools available to health care practitioners. One such tool is the St. John's Sepsis Agent (SJSA) Version 13 developed by Cerner (Kansas City, MO). The SJSA is an automated screening tool embedded in the EMR that evaluates 16 parameters to identify patients with sepsis [7]. These parameters include temperature, heart rate, respiratory rate, glucose level, capillary blood glucose, white blood cell count, number of bands, lactate levels, systolic blood pressure, mean arterial blood pressure, creatinine levels, total bilirubin levels, platelet count, partial thromboplastin time, urinalysis results, and blood culture results [7]. The algorithm constantly monitors patients' temperature, heart rate, respiratory rate, blood glucose, and white blood cell count. If a patient has demonstrated two of the SIRS criteria, the SJSA will then attempt to identify whether any end-organ dysfunction has occurred. It does this by reviewing the patient's record to determine if any of the following conditions are met: Systolic blood pressure below 90 mm Hg or mean arterial pressure below 65 mm Hg in the prior 30 h; creatinine increase ≥0.5 mg/dL from baseline; total bilirubin levels 2.0–10 mg/dL in the prior 30 h; or lactate over 2.0 mmol/L in the prior 12 h. If three SIRS criteria have been met but there is no evidence of end-organ dysfunction, the SJSA will fire a SIRS alert. If two SIRS criteria have been met but there is no evidence of end-organ dysfunction, the SJSA will not produce an alert at all and will continue to monitor the patient. Importantly, the SJSA will not fire a sepsis alert in patients who have met SIRS criteria unless the SJSA also determines that the patient has developed end-organ dysfunction [7].

If the SJSA determines that a patient has sepsis, it produces a notification that the patient's bedside nurse will see upon logging into the patient's EMR and the nurse is then responsible for alerting the clinical team. Although SJSA is utilized in many hospitals throughout the country, its performance characteristics have not been evaluated in any patient population. The purpose of this study was to compare the SSS and SJSA to determine which tool is optimal in a surgical intermediate care unit (SIMU) population.

Patients and Methods

This was an Institutional Review Board-approved prospective observational study conducted in the SIMU at Memorial Hermann Hospital, a tertiary referral hospital in Houston, Texas. Inclusion criteria were patients ≥16 years old; exclusion criteria were patients classified as “comfort care only” or those awaiting hospice.

The SSS and the SJSA were compared within the same patient population. Patients were screened using the SSS twice daily, at 5:00 am and at 5:00 pm, and screened continuously by SJSA via data from the EMR. Epidemiologic and compliance data related to sepsis were collected prospectively from the EMR by utilizing data feeds through the Health Level 7 protocol (HL7), which allowed for data to be obtained pertaining to electronic charting tools, vital signs monitors, and hospital laboratory equipment.

We used the sepsis definition as outlined by the American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference (ACCP/SCCM) as the gold standard to determine whether a patient developed sepsis [8]. The source of infection for all sepsis cases was recorded along with microbiologic culture data obtained as part of standard care. The time stamp of confirmation of sepsis was recorded using the time a patient first reached sepsis as defined by ACCP/SCCM Consensus Conference Committee. At the conclusion of the study, the sensitivity, specificity, NPV, and positive predictive value PPV of the SSS and the SJSA were calculated and compared.

Statistical analysis

Descriptive statistics were used to describe the percentages of sensitivities, specificities, PPVs, and NPVs. Data were analyzed using the two-sample test of proportions. The level of significance was set at p < 0.05 (two-sided). Statistical analysis was done with Stata software version 13 for Windows (StataCorp, College Station, TX).

Results

A total of 348 patients were included in this study. Over the 16-wk study period, 1,702 SSS screens were performed on 348 patients (60.1% male, average age = 52.8 [range, 16–100]) from the SIMU. Of the 348 patients included in the study, 47 (13.5%) were determined to be septic according to the ACCP/SCCM definition of sepsis [9]. Table 2 displays the demographics of the study population, as well as the different sources of infection for the patients with sepsis. Table 3 shows the screening results of the SSS and Table 4 shows the screening results of the SJSA. Of the 47 patients with sepsis, 35 of were identified by the SSS, whereas only 21 of them were identified by the SJSA. As shown in Table 5, there were 23 patients with sepsis identified by SSS who were not identified by SJSA, but there were only nine patients with sepsis identified by SJSA who were not identified by SSS. There were three patients with sepsis in the study who were not detected by either screening tool. Table 6 compares the performance characteristics of the SSS with those of the SJSA. The differences in sensitivities, PPVs, and NPVs were found to be statistically significant.

Table 2.

Descriptive Statistics: Side-by-Side Comparison

Variable	Sepsis	Not septic
n	47	301
Age	52.8 ± 19.5	52.8 ± 19.4
Male gender	28	181
Race
Caucasian	24	153
African American	6	66
Other	17	82
Source of infection
Pulmonary	12
Intra-abdominal	11
Skin/soft tissue	9
Urinary tract	8
Blood stream	6
Surgical implant	1

Table 3.

Sensitivity Analysis: Sepsis Screening Score

	Positive	Negative	Total
Sepsis	35	12	47
No sepsis	41	260	301
Total	76	272	348
Prevalence of sepsis	13.5%
Sensitivity	74.5%	PPV	46.1%
Specificity	86.4%	NPV	95.6%

PPV = positive predictive value; NPV = negative predictive value.

Table 4.

Sensitivity Analysis: St. John's Sepsis Agent

	Positive	Negative	Total
Sepsis	21	26	47
No sepsis	46	255	301
Total	67	281	348
Prevalence of sepsis	13.5%
Sensitivity	44.7%	PPV	31.3%
Specificity	84.7%	NPV	90.7%

PPV = positive predictive value; NPV = negative predictive value.

Table 5.

Screening Results of Patients with Sepsis

Screening results	n = 47	Frequency
+SSS/+SJSA	12	25.5%
+SSS/−SJSA	23	48.9%
−SSS/+SJSA	9	19.1%
−SSS/−SJSA	3	6.4%

SSS = Sepsis Screening Score; SJSA = St. John's Sepsis Agent.

Table 6.

Comparison of the Performance Characteristics

	Sepsis screening score	St. John's sepsis agent	p
Sensitivity	74.5%	44.7%	<0.001
Specificity	86.4%	84.7%	0.535
PPV	46.1%	31.3%	<0.001
NPV	95.6%	90.7%	0.011

PPV = positive predictive value; NPV = negative predictive value.

Discussion

Prompt recognition and implementation of early evidence-based therapies for the treatment of sepsis has proven to be one of the most significant factors in determination of a patient's outcome. Early identification of patients with sepsis allows for early intervention, decreasing the morbidity, mortality, and financial toll that this condition has taken on our health care system. One of the current barriers to the early implementation of evidence-based care is failure of bedside clinicians to recognize sepsis. This difficulty was perhaps best demonstrated in a recent study by Assuncao et al. [9] that concluded that only 26.5% of physicians can recognize non-severe sepsis, highlighting the importance of uniform screening tools to identify patients with sepsis.

By definition, a diagnosis of sepsis requires only the presence of two SIRS criteria in association with a severe infection [8]. By the time that sepsis has progressed to the point at which end-organ dysfunction is apparent, it is classified as severe sepsis [4], which is much less amenable to therapy than early sepsis [4,5,9,10]. One of the potential criticisms of the SJSA algorithm is that it will not fire a sepsis alert until a patient has demonstrated evidence of end organ dysfunction (i.e., the patient is already in severe sepsis) [7]. This strategy leaves patients with sepsis vulnerable to being undiagnosed until it is too late. Furthermore, this design model is flawed in that the parameters that the SJSA uses to determine the presence of end-organ dysfunction may be secondary to a medical condition other than sepsis, not immediately available within the EMR, or may be erroneously recorded. In contrast, the SSS does not require evidence of end-organ dysfunction to identify patients with sepsis, and instead utilizes concise parameters of the SIRS criteria to limit false-positives. Additionally, the parameters that the SSS is based upon can be recorded directly from the bedside, eliminating the need for multiple laboratory values and providing a means for immediate diagnosis.

An ideal screening tool for sepsis must have a high sensitivity in order to identify patients with sepsis before they progress along the all-too-familiar continuum to septic shock and potentially death. Additionally, an ideal screening tool should be designed in a manner that will maximize the NPV, so that clinicians can be confident in negative screens. The results of this study demonstrate that the SSS has a statistically significant higher sensitivity and NPV than the SJSA.

The requirement of strong evidence of end-organ dysfunction suggests that the SJSA algorithm was designed to concede sensitivity and NPV in an effort to maximize specificity and PPV. While this design strategy may be suitable for confirmatory testing, it is inappropriate and potentially dangerous for a screening tool to be engineered in this manner. Furthermore, even with the lower sensitivity and NPV, the findings of this study demonstrate that the SJSA and the SSS have similar specificities and that the SJSA actually has a statistically significant lower PPV than the SSS.

Importantly, the findings of this study suggest that the use of the SJSA can result in missed diagnoses for more than half of all sepsis cases. In contrast, the SSS demonstrated an ability to identify correctly almost 75% of septic cases. Additionally, nearly half of the patients with sepsis were detected by the SSS alone, whereas less than a fifth of the patients with sepsis were detected by the SJSA alone. Of the nine cases that were detected by the SJSA alone, the majority were ultimately discharged home with no complications. However, there were three patients detected by the SJSA alone who did have undesirable outcomes. One of these patients was most likely missed by the SSS because of the effects of metoprolol on the SSS parameters, resulting in a false-negative screen. The other two patients would have been detected by the SSS with continuous monitoring. It is important to note that the SJSA had constant surveillance over patients via continuous feeds from the EMR, while the SSS was conducted on patients only twice daily. However, continuous monitoring by the SSS would not have detected the three cases that were missed by both screening tools. These three patients were on high-dose antipyretics that affected the SSS parameters and likely prevented detection. The findings of this study suggest that physicians and other health care workers maintain a high index of suspicion for sepsis when using the SSS and take into account each patient's unique diagnosis and medication list. However, the results of this study do not lead us to advocate for the use of the SJSA to complement the SSS, as this combination is not superior to the use of the SSS alone.

Unlike the SJSA, the SSS is based upon parameters that are easily measurable from the bedside, which allows for rapid sepsis diagnosis and subsequent treatment. In the 12 cases in which there was an agreement between the SJSA and the SSS, the SSS detected septic cases 6.78 h earlier, on average, than the SJSA. Considering the fact that every hour that antibiotic administration is delayed in patients with sepsis-related hypotension results in an approximate 7% increase in mortality [10], these findings suggest that the SSS may be able to reduce the mortality rates of sepsis in patient care units where the SJSA is currently utilized. Furthermore, because the disability rate of septic shock is so great, the SSS may also be able to reduce the number of patients with sepsis being discharged to long-term acute care facilities or skilled nursing facilities.

One limitation in this study is that the SJSA was based on data from the EMR, whereas the SSS was based on data from direct observation. Data from the EMR may be more susceptible to error than that from direct observation. Therefore, it is possible that the SSS may not perform as well if it was automated within the EMR. Nonetheless, if the SSS was automated within the EMR, it would have much more surveillance time over patients, providing more opportunity to detect septic patients. Future studies should analyze the performance characteristics of an automated SSS to determine whether the integrity of the SSS will be compromised or enhanced by implementation of an electronic version.

Conclusions

The findings of this study suggest that the SSS can detect sepsis more accurately than the SJSA. The difference in sensitivities between the two tests is of particular importance, because this indicates that the SSS is more likely to diagnose sepsis effectively in the critical early stages. This study establishes a basis for the utilization of the SSS instead of the SJSA.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

References

Moore

, Moore

, Jones

, et al. Sepsis in general surgery: A deadly complication. Am J Surg, 2009; 198:868–874.

Hall

, Williams

, DeFrances

, Golosinskiy

. Inpatient care for septicemia or sepsis: A challenge for patients and hospitals. NCHS Data Brief, 2011; June(62):1–8.

Torio

, Andrews

. National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2011: Statistical Brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Health Care Policy and Research, 2006. www.ncbi.nlm.nih.gov/books/NBK169005/ (Last accessed March 12, 2015).

Dellinger

, Levy

, Rhodes

, et al. Surviving Sepsis Campaign: International guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med, 2013; 39:165–228.

Dombrovskiy

, Martin

, Sunderram

, Paz

. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: A trend analysis from 1993 to 2003. Crit Care Med, 2007; 35:1244–1250.

Moore

, Jones

, Kreiner

, et al. Validation of a screening tool for the early identification of sepsis. J Trauma, 2009; 66:1539–1546.

St. John's Sepsis Agent. https://store.cerner.com/items/1527 (Last accessed April 8, 2015).

Bone

, Balk

, Cerra

, et al. American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference: Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. Chest, 1992; 101:1644–1655.

Assuncao

, Machado

, Akamine

, et al. Do they know what is sepsis time? Septic Patients Survey Enrolling Staffs study. Crit Care, 2007; 11(Suppl 2):P45.

10.

Kumar

, Roberts

, Wood

, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med, 2006; 34:1589–1596.