Validity of International Classification of Diseases,Ninth Revision,Clinical Modification (ICD-9-CM) Screening for Sepsis in Surgical Mortalities

Abstract

Background:

Sepsis is among the leading causes of death in the United States. The Agency for Healthcare Research and Quality uses International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) billing code screening for the identification of sepsis. We investigated the incidence of sepsis in mortality at our academic medical center through ICD-9-CM screening of billing codes corresponding to sepsis and compared this approach for accuracy using physician chart review as the gold-standard.

Methods:

Two hundred forty-three surgical mortalities between January 2012 and January 2013 were reviewed by a Performance Improvement team. All mortalities were screened and evaluated for sepsis using physician chart review and ICD-9-CM codes for sepsis (995.91), severe sepsis (995.92), and septic shock (785.52).

Results:

Unexpected mortalities were associated with higher rates of sepsis and expected mortalities than anticipated (p<0.0001). A total of 40.6% of patients with sepsis suffered from more than one infection; the most common infectious sources were intra-abdominal (43.5%), blood stream (40.3%), and pulmonary (38.7%) infections. Screening by ICD-9-CM identified sepsis in 23.0% of mortalities, and physician review identified sepsis in 25.5% of mortalities. The sensitivity and specificity of ICD-9-CM screening were 82.3% and 78.3%, respectively. The positive and negative predictive values were 91.1% and 62.1%, respectively.

Conclusion:

Sepsis is a common concurrent condition in surgical patients who die unexpectedly. Screening by ICD-9-CM for sepsis is accurate in identifying patients with sepsis but misses the identification of all patients with sepsis. The diagnostic accuracy of ICD-9-CM screening for sepsis is currently not adequate for public reporting or benchmarking, and is useful only as a guide for institutional quality improvement.

Sepsis has repeatedly been identified as a major determinant of in-hospital mortality and increased health care cost [1,2]. In a landmark epidemiologic survey of sepsis in the United States, Angus et al. modeled that there were 750,000 cases of sepsis annually, associated with a 28.6% mortality rate and $16.7 billion in annual health care cost [1]. Among surgical patients, there has been reported a 1.7% incidence of sepsis after elective surgery and 4.5% after non-elective surgery, resulting in increased hospital stays of nearly 11 d and increased hospital costs of $32,900 per sepsis episode [3,4].

Given the impact of sepsis on outcomes, post-operative sepsis has been included as a patient safety indicator (PSI) by the Agency for Healthcare Quality and Research (AHRQ) [5]. Patient safety indicators query International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) administrative data to benchmark patient safety events at institutions. In a previous validation study at our urban academic medical center, we demonstrated that a set of 10 AHRQ patient safety indicators were associated with a 17% false-positive rate, primarily due to coding errors and documentation errors [6]. As a result, we validated the ICD-9-CM identification method for sepsis through the use of chart review as the gold standard. In this single-center review, we characterize the incidence of sepsis in surgical mortalities and validate the system of ICD-9-CM screening for sepsis.

Patients and Methods

We reviewed all adult surgical inpatient mortalities between January 2012 and January 2013. Surgical inpatients included all patients older than 18 y who were admitted to a surgical service, discharged from a surgical service, or whom had an operative procedure performed by a surgical service. Mortality was defined as a death during the hospitalization. The study was approved by the Institutional Review Board.

All surgical mortality patient records were reviewed by a performance improvement team consisting of a performance improvement registered nurse, coding specialist, and documentation improvement nurse. Mortalities were categorized as: An anticipated death within 24 h of admission, an expected death upon admission due to severity of the presenting conditions, or an unexpected death. Categorizations of deaths were performed subjectively and independently by the members of the performance improvement team based on factors present on admission.

Surgical mortality patient records were screened for sepsis using the ICD-9-CM codes for sepsis (995.91), severe sepsis (995.92), and septic shock (785.52). The records identified with sepsis based on ICD-9-CM codes and the records of patients who had unexpected deaths were reviewed further by a physician to confirm the presence or absence of sepsis.

Physician review was utilized as the gold standard. For physician reviews, sepsis was defined in accordance with the American Association for Chest Physicians/Society for Critical Care Medicine (ACCP/SCCM) guidelines as the presence of systemic inflammatory response syndrome (SIRS) with an infectious cause [7]. Infectious sources among the mortalities were evaluated and categorized as blood stream infections, pulmonary infections, intra-abdominal infections, genitourinary infections, or soft tissue infections. Blood stream infections were defined as the presence of one blood culture positive for any non-coagulase-negative staphylococcal organism or two simultaneously positive blood cultures for coagulase-negative staphylococci. Pulmonary infections included those confirmed through cultures from bronchial brushings and those presumed to be present through documented deterioration in pulmonary status or chest radiographs. Intra-abdominal infections included organ/space infections, abscesses, and gastrointestinal perforations or necrosis. Genitourinary infections were defined as the presence of positive urine cultures for bacterial or fungal organisms. Soft tissue infections included abscesses and deep wounds with positive cultures for organisms, necrotizing fasciitis, and extremities with wet requiring debridement. Some patients had multiple sources of sepsis and, in such cases, all sources of sepsis were recorded.

Statistical analysis was performed using JMP 9.0 (SAS Institute, Cary, NC). Chi-square and Fisher exact tests were used to evaluate categorical variables. The validity of using ICD-9-CM codes to reflect the incidence of sepsis was evaluated using diagnostic testing.

Results

During the 1-y study period, there were 243 surgical mortalities. A total of 104 deaths were anticipated within 24 h of admission, 83 were expected deaths due to severity of the presenting conditions, and 56 deaths were unexpected. The majority of anticipated deaths within 24 h succumbed to severe multisystem trauma or non-survivable intracranial hemorrhages within a few hours of admission. Most expected deaths were seriously or critically ill patients transferred from other medical centers, or patients presenting to our center with severe head or multisystem trauma.

Physician review identified that 62 of the 243 (25.5%) mortalities experienced sepsis during their hospitalization. When comparing the incidence of sepsis among mortality categories, unexpected mortalities had a significantly higher rate of sepsis than all mortalities, anticipated mortalities, and expected mortalities (p<0.0001). Anticipated deaths within 24 h had a significantly lower sepsis rate than all mortalities, expected mortalities and unexpected mortalities (p<0.0001; Fig. 1).

FIG. 1.

Rate of sepsis by identification method and mortality categorization.

Among the 62 patients with sepsis at time of death, there were 98 total infectious sources identified. The most common infectious sources were intra-abdominal (43.5%), blood stream (40.3%). and pulmonary (38.7%) infections. Twenty-five of the 62 patients (40.3%) had concurrent infections. A total of 90.9% of all genitourinary infections, 72.7% of soft tissue infections, and 68.0% of blood stream infections were associated with concurrent infections (Table 1). The most common concurrent infection pairs were blood stream and intra-adominal infections (8/62), blood stream and pulmonary infections (8/62), and pulmonary and genitourinary infections (7/62).

Table 1.

Sources of Sepsis

Source	Total incidence (n=62 patients)	Frequency of being associated with a concurrent infection
Blood stream	25 (40.3%)	17 (68.0%)
Pulmonary	24 (38.7%)	14 (58.3%)
Intra-abdominal	27 (43.5%)	10 (37.0%)
Genitourinary	11 (17.7%)	10 (90.9%)
Soft tissue	11 (17.7%)	8 (72.7%)
Total	98

Compared with the 62 (25.5%) patients with sepsis identified by physician review, sepsis screening using ICD-9-CM administrative data revealed that 56 of the 243 (23.0%) mortalities experienced sepsis during their hospitalization. The positive predictive value of ICD-9-CM screening for sepsis was 91.1% (95% confidence interval [CI]; 80.4%-97.0%) and the negative predictive value was 62.1% (95% CI; 42.3%-79.3%). The sensitivity of ICD-9-CM screening was 82.3% (95% CI; 70.5%-90.8%) and the specificity was 78.3% (95% CI; 56.3%-92.5%) (Table 2).

Table 2.

Validity of ICD-9-CM Screening for Sepsis

	ICD-9-CM Screening
Positive predictive value	91.1% (80.4% to 97.0%)
Negative predictive value	62.1% (42.3% to 79.3%)
Sensitivity	82.3% (70.5% to 90.8%)
Specificity	78.3% (56.3% to 92.5%)

ICDM-9-CM=International Classification of Diseases, Ninth Revision, Clinical Modification.

Discussion

More than one-quarter of all mortalities at our center were associated with sepsis. The rate of sepsis among unexpected mortalities was two-fold greater than the overall mortality of rate sepsis, suggesting that sepsis may be a major determinant of late, potentially preventable, mortality. In the setting of trauma, the impact of sepsis on late mortality has been well described and our results support the notion that effective prevention and treatment of sepsis could prevent late inpatient mortality [8,9]. Infection is believed to progress to sepsis and multiple organ dysfunction syndrome through cascading series of inflammatory mediator and cytokine release. The role that concurrent infections may play in the progression to severe sepsis is less clear. Concurrent infections may amplify these cascading inflammatory processes and worsen sepsis. Conversely, the systemic inflammatory state of sepsis and organ dysfunction may weaken the resiliency of local innate immune function and render the host more susceptible to new sources of infection [10,11].

We found that more than 40% of patients who died of sepsis had multiple sources of infection. More than 90% of all genitourinary infections were associated with a concurrent infection and more than one-half of all blood stream and pulmonary infections were associated with concurrent infections. The high rate of concurrent infections with blood stream, pulmonary, and genitourinary infections emphasizes the wider implications that targeted single-infection initiatives may have. Such initiatives include timely urinary catheter removal or avoidance, central venous catheter bundles, and ventilator bundles [12 –14]. Whereas our study design precludes the ability to establish definitively such causal relationships, the high incidence of concurrent infections in surgical mortalities suggests a relationship.

At our academic medical center, ICD-9-CM screening for sepsis underreported the true incidence of sepsis (23.0% vs. 25.5%). Although there was a robust positive predictive value (91.1%), the negative predictive value of 62.1% indicates that the absence of sepsis as identified by ICD-9-CM screening is an unreliable marker. Furthermore, ICD-9-CM screening of sepsis was associated with only moderate sensitivity (82.3%) and specificity (78.3%), reflecting its limitation as a screening test. Whereas this may positively impact public reporting and reimbursement in the short-term, such inaccuracies in screening hamper the ability to perform root cause analyses that are imperative to improve process measures designed to decrease sepsis. Beyond its application for quality improvement, the validity of ICD-9-CM screening for sepsis is important due to its usage in institutional benchmarking and public reporting through the federal AHRQ patient safety indicators (5).

The moderate sensitivity and specificity of ICD-9-CM screening in this study mirrors other validation studies of the AHRQ patient safety indicators at our institution. We reported previously that the positive predictive value of 10 surgically-related AHRQ PSIs was limited to 83%. Coding errors contributed to 30% of the PSI inaccuracies, followed by 19% documentation errors, and 16% insufficient patient safety indicator criteria in the chart (6). In reviewing charts for this study, we identified similarly that coding errors were due primarily to insufficient physician documentation of infections in the chart. For example, intra-abdominal infection in a patient with perforated bowel was often not listed as an infection source in clinical documentation. Conversely, sepsis often was used interchangeably with systemic inflammatory response syndrome, contributing to false-positive identification of sepsis.

Although sepsis is among the most common complications encountered after major surgery, consensus methods to quantify its incidence and impact on mortality have not been developed [15]. Most studies comparing administrative data collection with clinician review have found that clinician review is superior. However, clinician review is resource-intensive and not a feasible alternative to benchmark surgical safety at high-volume centers or regions. The National Surgical Quality Improvement Program (NSQIP) of the American College of Surgeons offers a potentially beneficial alterative. Originally developed by the Veteran's Health Administration, NSQIP uses multiple data sources and has been reported to have a high level of accuracy [16,17]. The NSQIP system uses performance improvement-trained nurses to abstract charts to collect administrative and clinical data, thereby increasing the likelihood that poorly documented clinical subtleties are recognized for patient safety benchmarking. More clinically accurate data collection by trained quality improvement chart-abstractors is an important advantage of NSQIP over the ICD-9-CM screening system employed by the AHRQ PSI.

The increased and standardized use of ICD-9-CM administrative data for safety and quality assessment is a recent development and is reflected in our study period and relatively small sample size. A limitation in our methodology is that the chart reviewers were not blinded and the physician reviews were not validated independently, thus inviting the potential for bias. Additionally, the retrospective nature of the study does not allow inference regarding causal relationships between sepsis and mortality.

In conclusion, sepsis is a common concurrent condition in surgical patients who die unexpectedly. Sepsis affects patient outcomes adversely and may impact hospital safety benchmarking, public reporting, and reimbursement. Screening for sepsis using ICD-9-CM billing codes is a fair screening test that can help guide institutional interventions to decrease the incidence of sepsis. Whereas all screening tests are prone to error, to provide public benchmarking adequately and to measure quality improvement accurately, screening tests should strive for sensitivity and specificity of more than 90%. Given that the current sensitivity and specificity of ICD-9-CM screening are less than 85%, it should preclude the use of ICD-9-CM screening for sepsis in public benchmarking and determination of quality. Other methods such as the National Surgical Quality Improvement Project (NSQIP) may provide more valid data through eliminating the inherent issues of coding and documentation errors in the AHRQ PSI system, thereby enabling improved quality improvement interventions and benchmarking. Future prospective comparisons of these methods may help clarify the optimal approach to measuring surgical patient quality and safety.

Footnotes

Author Disclosure Statement

No conflicting financial interests exist.

References

Angus

, Linde-Zwirble

, Lidicker

, et al. Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care. Crit Care Med, 2001; 29:1303–1310.

Martin

, Mannino

, Eaton

, Moss

. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med, 2003; 348:1546–1554.

Eber

, Laxminarayan

, Perencevich

, Malani

. Clinical and economic outcomes attributable to health care-associated sepsis and pneumonia. Arch Intern Med, 2010; 170:347–353.

Vogel

, Dombrovskiy

, Carson

, et al. Postoperative sepsis in the United States. Ann Surg, 2010; 252:1065–1071.

Agency for Healthcare Research and Quality. AHRQ Quality Indicators: Guide to Patient Safety Indicators, 2003. www.qualityindicators.ahrq.gov/downloads/modules/psi/v31/psi_guide_v31.pdf. (Accessed April 12, 2014).

Ramanathan

, Leavell

, Stockslager

, et al. Validity of Agency for Healthcare Research and Quality patient safety indicators at an academic medical center. Am Surg, 2013; 79:578–582.

Bone

, Balk

, Cerra

, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest, 1992; 101:1644–1655.

Sauaia

, Moore

, et al. Epidemiology of trauma deaths: A reassessment. J Trauma, 1995; 38:185–193.

Demetriades

, Kimbrell

, Salim

, et al. Trauma deaths in a mature urban trauma system: Is “trimodal” distribution a valid concept?. J Am Coll Surg, 2005; 201:343–348.

10.

Barie

, Hydo

, Pieracci

, et al. Multiple organ dysfunction syndrome in critical surgical illness. Surg Infect (Larchmt), 2009; 10:369–377.

11.

Fry

. Sepsis, systemic inflammatory response, and multiple organ dysfunction: The mystery continues. Am Surg, 2012; 78:1–8.

12.

Clarke

, Tong

, Pan

, Easley

, Norrick

, Ko

, et al. Reduction in catheter-associated urinary tract infections by bundling interventions. Int J Qual Health Care, 2013; 25:43–49.

13.

Croce

, Brasel

, Coimbra

, et al. National Trauma Institute prospective evaluation of the ventilator bundle in trauma patients: Does it really work?. J Trauma Acute Care Surg, 2013; 74:354–360.

14.

Duane

, Brown

, Borchers

, et al. A central venous line protocol decreases bloodstream infections and length of stay in a trauma intensive care unit population. Am Surg, 2009; 75:1166–1170.

15.

Gaieski

, Edwards

, Kallan

, Carr

. Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med, 2013 May; 41:1167–1174.

16.

Neuman

, Michelassi

, Turner

, Bass

. Surrounded by quality metrics: What do surgeons think of ACS-NSQIP?. Surgery, 2009; 145:27–33.

17.

Ingraham

, Richards

, Hall

, Ko

. Quality improvement in surgery: The American College of Surgeons National Surgical Quality Improvement Program approach. Adv Surg, 2010; 44:251–267.