Can We Define Surgical Site Infection Accurately in Colorectal Surgery?

Abstract

Background:

Increasingly, surgical site infection (SSI) is being tied to quality of care. The incidence of SSI after colorectal surgery differs widely. We hypothesize that it is difficult to define SSI reliably and reproducibly when adhering to the U.S. Centers for Disease Control and Prevention (CDC) definitions.

Methods:

Elective intra-abdominal colorectal procedures via a clean-contaminated incision performed at a single institution between January 1 and May 1, 2011 were queried. Three attending surgeons examined all patients' records retrospectively for documentation of SSI. These data were compared with the institutional National Surgeon Quality Improvement Program (NSQIP) data with regard to deep and superficial incisional SSI.

Results:

Seventy-one cases met the inclusion criteria. There were six SSIs identified by NSQIP, representing 8.4% of cases. Review of the three attending surgeons demonstrated a significantly higher incidence of SSI, at 27%, 38%, and 23% (p=0.002). The percent of overall agreement between all reviewers was 82.16 with a kappa of 0.64, indicating only modest inter-rater agreement. Lack of attending surgeon documentation and subjective differences in chart interpretation accounted for most discrepancies between the surgeon and NSQIP SSI capture rates.

Conclusions:

This study highlights the difficulty in defining SSI in colon and rectal surgery, which oftentimes is subjective and difficult to discern from the medical record. According to these preliminary data from our institution, there is poor reliability between clinical reviewers in defining SSI on the basis of the CDC criteria, which has serious implications. The interpretation of clinical trials may be jeopardized if we cannot define SSI accurately. Furthermore, according to current CDC definitions and infection tracking strategies, these data suggest that the institutional incidence of SSI may not be a reliable measure by which to compare institutions. Better methods for defining SSI should be implemented if these data are made publicly available and tied to performance measures.

Surgical site infection (SSI) is the most common hospital-acquired infection in the surgical population, contributing to 3.7 million excess hospital days and more than $1.6 billion in hospital and patient costs annually [1,2]. The inpatient costs for a patient with an SSI is approximately twice the cost for a patient without an SSI [3]. The presence of an SSI increases the cost of a colectomy by $17,324 [4]. The incidence of SSI ranges from 5%–30% depending on the degree of contamination and the operative site. These infections are associated with substantial morbidity and mortality rates, prolonged hospitalization, increased intensive care unit admissions, and more frequent readmissions [5].

Increasingly, SSI is being tied to quality and pay for performance parameters. As of January 2012, the Centers for Medicare and Medicaid Services (CMS) requires hospitals to report deep and organ space SSI after colon surgery through the National Healthcare Safety Network (NHSN), which is an internet-based surveillance system that integrates patient and healthcare personnel safety surveillance systems managed by the Division of Healthcare Quality Promotion at the U.S. Centers for Disease Control and Prevention (CDC). Additionally, the American College of Surgeons (ACS) has been working with CMS to initiate a pilot program for voluntary public reporting of ACS National Surgical Quality Improvement Program (NSQIP) outcome data on the Hospital Compare website. A major focus of this NSQIP CMS pilot is 30-day morbidity and mortality rates following colectomy. By January 2013, the rate of SSI after colectomy for each institution was to be available on the website.

Colorectal surgery has been associated consistently with the highest risk of SSI, predominately because of the heavy bacterial inoculum in the distal colon. However, review of the literature demonstrates striking differences in incidence, depending on the method used to define and track incisions. By examining the national ACS NSQIP Participant Use File, we reported a 9.3% incidence of deep and superficial incisional SSI in patients undergoing elective colorectal surgery during 2009 [6]. This value is similar to that in other studies examining nationwide databases such as NSQIP [7]. However, we were one of the first institutions to report an alarmingly high incidence of SSI (26%) after elective colon and rectal surgery [3]. Following this publication, numerous others have reported similar findings when prospective incision surveillance strategies are implemented, such as those utilized in randomized controlled trials. Anthony et al. [8] randomized patients to an evidence-based bundle for preventing SSI. The rates of superficial and deep SSI were 36% in the control arm vs. 19% in the experimental arm. Krieger et al. [9] investigated the value of silver nylon in preventing SSI after colorectal surgery and demonstrated rates of 33% and 13% in the control and experimental arms, respectively. Similarly, a trial of ertapenem vs. cefotetan prophylaxis in elective colorectal surgery demonstrated a 17.1% incidence of SSI vs. 26.2% in the experimental and control groups, respectively [10]. Obviously, there will be variability among studies depending on the case mix and patient demographics. However, these studies demonstrate a persistent trend to a much higher incidence of SSI when prospective collection methods are used to track SSI.

Given the data demonstrating a national incidence of deep and superficial incisional SSI after colon and rectal surgery of only 9%, we were disappointed at what seemed to be a divergence of our institutional SSI rate from the national ACS NSQIP data. Prior studies from the University of Virginia (UVA) reported an SSI rate three-fold higher [3]. Our personal experience led us to believe that the incidence of SSI after colon surgery at UVA remained high, as we seemed to be opening infected surgical incisions on a regular basis. However, review of our current institutional NSQIP data revealed an SSI rate of only 8.4%. Therefore, we examined in more detail the incidence of SSI after elective colon and rectal surgery at our institution. We hypothesized that it is difficult to define SSI reliably and reproducibly when adhering to the CDC definitions.

Patients and Methods

A retrospective observational single-institution study was conducted. Data were analyzed on all adult patients (≥18 years) undergoing elective intra-abdominal colon and rectal procedures with a clean-contaminated site performed at UVA between January 1 and May 1, 2011. The Institutional Review Board at the UVA Health System approved this study.

NSQIP data collection

The UVA is a participant in the ACS NSQIP, which has been described. (For a complete description of the NSQIP methodology, please refer to the user guide found on the NSQIP website [www.acsnsqip.org]). More than one hundred variables are entered in the database by clinically trained staff. At our institution, two surgical clinical nurse reviewers (SCRs) track all major general surgery patients rather than merely a sampling of patients. These SCRs are highly trained (one with a PhD in nursing and the other an acute care nurse practitioner). The SCRs capture these data using a variety of methods, including medical chart abstraction. Required data variables are entered via Web-based data collection to the www.acsnsqip.org site. To ensure the data collected are of the highest quality, the ACS NSQIP has developed a host of training mechanisms for the nurse–reviewer and conducts an inter-rater reliability audit of participating sites. The combined results of the audits completed to date have demonstrated an overall disagreement rate of approximately 2.5% for all assessed program variables. Sites that have a higher than 5% disagreement rate are required to undergo an additional audit according to the recommendations of the ACS NSQIP.

Clinician review

Three attending surgeons, including two colorectal surgeons (TLH and CMF) and a specialist in surgical infectious disease (RGS), examined all patients' records for retrospectively documentation of SSI. Each surgeon reviewed daily progress notes, discharge summaries, and clinic notes individually for the 30 d after surgery. All patients were followed for at least 30 d. Surgeons were provided with CDC definitions of superficial and deep incisional SSIs as a reference. Surgeons were blinded to the NSQIP data and to the other reviewers' findings. The inter-rater reliability in defining deep and superficial incisional SSI among the three surgeons and NSQIP were compared using kappa analysis.

Primary outcome

The ACS NSQIP and the surgeon reviewers utilized the CDC's NNIS system [1], which classifies SSIs into incisional or organ/space infections. Incisional SSIs are further divided into those involving only skin and subcutaneous tissues (superficial) and those involving deeper softer tissues of the incision (deep). We did not include organ/space SSIs. According to the CDC, “NNIS definitions of SSIs have been applied consistently by surveillance and surgical personnel in many settings and currently are a de facto national standard.” The definitions are provided below:

A superficial incisional SSI is an infection that occurs within 30 days after the operation and involves only skin or subcutaneous tissue of the incision and at least one of the following:

• Purulent drainage, with or without laboratory confirmation, from the superficial incision;

• Organisms isolated from an aseptically obtained culture of fluid or tissue from the superficial incision;

• At least one of the following signs or symptoms of infection: Pain or tenderness, localized swelling, redness, or heat AND superficial incision deliberately opened by the surgeon, unless incision is culture-negative;

• Diagnosis of superficial incisional SSI by the surgeon or attending physician;

• Do not report the following conditions as SSI:

○ Stitch abscess (minimal inflammation and discharge confined to the points of suture penetration);

○ Infected burn wound;

○ Incisional SSI that extends into the fascial and muscle layers (see deep incisional SSI);

Deep infection occurs within 30 d, appears to be related to the operation, and involves deep soft tissues (e.g., fascial and muscle layers) of the incision and at least one of the following:

• Purulent drainage from the deep incision but not from the organ/space component of the surgical site;

• A deep incision spontaneously dehisces or is deliberately opened by a surgeon when the patient has at least one of the following signs or symptoms: fever (>38°C), localized pain, or tenderness, unless site is culture-negative;

• An abscess or other evidence of infection involving the deep incision is found on direct examination, during reoperation, or by histopathologic or radiologic examination;

• Diagnosis of a deep incisional SSI by a surgeon or attending physician.

Kappa statistic

Kappa is a statistical measure of inter-rater agreement that is more robust than a simple percent agreement calculation because it takes into account the agreement that may occur by chance. Cohen's kappa, which measures the agreement between two raters, was used to measure the inter-rater agreement between each pair of individual evaluators. Fleiss' Kappa, which is used when there are more than two raters, was employed to evaluate the inter-rater agreement among all four evaluators, including the three surgeons and NSQIP. Values range from −1.0 to 1.0, with −1.0 indicating perfect disagreement below chance, 0.00 indicating agreement equal to chance, and 1.0 indicating perfect agreement above chance. A kappa value of 0.70 or above indicates adequate inter-rater agreement.

Results

Seventy-one cases met the inclusion criteria. Demographic data are demonstrated in Table 1. The majority of incisions were classified as clean-contaminated. However, two were contaminated as the result of intraoperative bowel spillage, and three additional incisions were considered dirty. Two of these patients had Crohn disease with an inflammatory phlegmon, and one patient had a diverticular abscess. There were six deep and superficial incisional SSIs identified by NSQIP, representing 8.4% of cases. Review by the three attending surgeons demonstrated a significantly higher incidence SSI, at 27%, 38%, and 23% (p=0.002) (Fig. 1).

FIG. 1.

Deep and superficial incisional surgical site infection rates as determined by three attending surgeons and nurse reviewers based on review of the same patient records.

Table 1.

Demographic Data of Patients Undergoing Elective Colorectal Surgery

Total number of patients	71
Age±standard deviation (SD)(years)	51.5±5.8
Procedure (CPT)(%)
Hartman's reversal (44626)	5 (7.0)
Partial colectomy (44140)	20 (28.1)
Low anterior resection (44145, 44146)	17 (23.9)
Total proctocolectomy (44158, 44155)	7 (9.9)
Laparoscopic partial colectomy (44204)	15 (21.1)
Abdominoperineal resection (45110, 45126)	7 (9.9)
Diagnosis (%)
Malignancy	47 (66.1)
Inflammatory bowel disease	11 (15.5)
Diverticular disease	4 (5.6)
Other	9 (12.7)
Body mass index±SD	29.0±6.6
Diabetes mellitus (%)	15 (21.1)
Steroid use (%)	8 (11.2)
Incision classification (%)
Clean-contaminated	66 (93.0)
Contaminated	2 (2.8)
Dirty	3 (4.2)

The percent of overall agreement between all reviewers was 82.16% with a kappa of 0.64, indicating only modest inter-rater agreement. Kappa statistics between each pair of reviewers are demonstrated in Table 1. The three surgeons demonstrated greater inter-rater agreement among themselves compared with the NSQIP reviewers. However, even between the surgeon reviewers, there was substantial variability.

In review of the charts, lack of attending documentation and subjective differences in chart interpretation accounted for most discrepancies between the surgeon and NSQIP SSI capture rates. Surgeons were more likely to consider an incision that was opened on purpose in the post-operative period to be an SSI, whereas NSQIP reviewers required highly specific documentation and nomenclature. In each of the charts identified by the NSQIP reviewers, it was stated specifically that the patient had a SSI, whereas the surgeons were more likely to infer the presence of an SSI on the basis of clinical descriptors.

Discussion

Since the Institute of Medicine report in 1999 demonstrated an alarming rate of medical errors, there has been a tremendous focus on quality improvement nationally. Additionally, as value-based purchasing is introduced as part of the Affordable Care Act, it is increasingly imperative that we work consistently toward quality improvement. Within colorectal surgery, SSI is a central focus of quality improvement efforts, given that it is the most common morbidity. As such, it is imperative that we assure the highest standards in quality from institutions and surgeons. However, this is reliant on a foundation built on an accurate and reliable definition of SSI and collection of the data.

This study demonstrates that the definition of SSI in colon and rectal surgery is subjective and difficult to discern from the medical record. We report substantial discrepancies in defining SSI among three attending surgeons and the highly trained NSQIP clinical nurse reviewers. Surgeons were more likely to consider an incision that was opened purposely in the post-operative period to be an SSI, whereas NSQIP reviewers required specific documentation and nomenclature before making this diagnosis.

This potential discrepancy in SSI based on methodology and definition has important implications for the care of surgical patients. Could this phenomenon explain why smaller single-institutional studies have demonstrated better SSI rates with the Surgical Care Improvement Project (SCIP) measures, whereas national data have failed to demonstrate any meaningful improvement? In our prior efforts with SCIP at the UVA, we were able to demonstrate a significant improvement in our institutional SSI rate, from 25.6% to 15.9%, following institution of SCIP measures in colorectal surgery patients [11]. Berenguer et al. [12] obtained similar results. However, Hawn et al. [13] demonstrated no effect on quality improvement as evident in the National Veterans' Affairs data from 2005 to 2009. Perhaps at the institutional level, the definition of SSI is constant and reliable, enabling the identification of improvements from quality control measures, whereas the national data from numerous institutions with potentially different definitions of SSI can lead to a dilution of the effect of such quality improvement measures. As such, the inability to define SSI reliably has the potential to impede the study of SSI significantly and negatively impact national efforts aimed at prevention.

Incidentally, the collective incidence of SSI as reported by the surgeons in this review was significantly higher than the 15.9% reported after institution of SCIP measures previously at our institution [11]. It is important to point out that this represented only four months' worth of data, and there was a significant portion of patients with Crohn disease and diverticulitis in the current series. Thus, 7% of the patients had contaminated or dirty incisions, which likely contributed to the higher rate of infection.

Public reporting of data

These data also raise concern about public reporting of SSI data as a quality indicator. As the demand increases for public reporting of these data, it is imperative that we have accurate definitions. Those institutions with SSI rates in the 20% range are considered “high outliers” within NSQIP. However, as one takes a closer look at the literature, it becomes clear that in many studies where the incisions are being tracked prospectively, the incidence of SSI after colon and rectal surgery approaches 20% [8,9,14]. Furthermore, in the current era, there are substantial patient, physician, and institutional pressures that tell against accurate reporting. As a result, surgeons have begun wicking the incision in high-risk patients so that it is considered an open wound and thus cannot be classified as a SSI by CMS. This action is going to have implications for the patient and healthcare system, given the morbidity and higher costs associated with an open incision. Although there is no doubt that transparency in quality outcomes is important and can improve patient care significantly, it must be founded on sound, accurate, and reliable data.

Alternative definition of SSI

Another implication may be that ultimately, it does not matter if the patient meets the criteria for a SSI. If the patient has an open incision that takes weeks to heal, it is morbid for the patient and costly for the healthcare system. There is substantial morbidity incurred by care of the open site, regardless of whether the patient satisfies the strict CDC criteria for a SSI. Given the discrepancy in defining SSI in colorectal surgery patients, it may be more logical to track the percentage of all incisions that fail to heal primarily without intervention rather than to rely on poorly reproducible and subjective definitions. Another alternative is the ASEPSIS score. Originally developed in cardiac surgery patients by Wilson et al. [15] in 1986, the ASEPSIS score attempts to overcome the reliance on subjective interpretation and is designed to relate the appearance of the incision to the clinical consequences. The ASEPSIS score is calculated by assigning a score based on the presence of erythema, serous exudate, purulent exudate, and deep tissue separation according to the percentage of the site affected by each process. There are additional points assigned for any intervention such as antibiotic treatment, drainage of pus under local or general anesthesia, isolation of bacteria from the incision, and an inpatient stay longer than 14 days. A score >20 points defines a SSI. A score of 21–30 designates a minor SSI, 31–40 a moderate SSI, and >40 a severe SSI. The score describes the incision quantitatively and qualitatively and may better discriminate between an SSI and a simple seroma. Although it is cumbersome to collect and calculate, the ASEPSIS score is less subjective and has the potential to demonstrate significantly less inter-rater variability than the CDC criteria.

A prospective study was undertaken in a Turkish university hospital of 153 patients undergoing elective or emergency colorectal surgery evaluating the ASEPSIS method for wound surveillance. The authors report a 17.6% incidence of SSI using an ASEPSIS score >20 points indicative of infection. Interestingly, the investigators found that 47.7% of all incisions developed some delay in healing. Given that almost 50% had some problem with healing, it is easy to see how there could be wide variation in the definition of SSI if specific, objective measures are not used [16].

An additional advantage that the ASEPSIS score has over the CDC definition is the ability to assess the severity of infection objectively. With the current CDC definition, a SSI is coded the same way regardless of the length of the wound affected. Although the CDC classification of superficial vs. deep offers some discrimination, this class is based entirely on the presence of fascial involvement. Because the treatment of superficial infections in colorectal surgery requires opening of the incision in the majority of patients, this confers significant morbidity and cost regardless of whether the fascia is involved (particularly in our increasingly obese society). Furthermore, it may be difficult, particularly in the obese patient, to assess definitively whether the fascia is involved when opening the wound at the bedside. Therefore, this introduces an additional opportunity for error in diagnosing SSI based on CDC definitions, which has major implications given that CMS has elected not to include superficial SSI in reporting data through NHSN.

Limitations

This study is limited to a single institution's data, and it could be that the problem we identified is unique to the UVA. However, our nurse reviewers are highly trained SCRs, having been part of the original 12 SCRs who help originate NSQIP in the private sector. Furthermore, our data are on par with national averages. In addition, even among the three highly trained attending surgeons, there was significant inter-rater variability. Yet, to be certain, further studies at other institutions are warranted.

Conclusions

Based on these preliminary data from our institution, there is poor reliability between clinical reviewers in defining SSI based on CDC criteria, a finding that has serious implications. The interpretation of clinical trials may be jeopardized if we cannot define SSI accurately. Furthermore, on the basis of the current CDC definitions and infection tracking strategies, these data suggest that the institutional incidence of SSI may not be a reliable quality measure by which to compare institutions. Improved methods for defining SSI should be sought and implemented if these data are made publicly available and tied to performance measures.

Footnotes

Author Disclosure Statement

No competing financial interests exist for any of the authors.

References

Horan

, Gaynes

, Martone

, et al. CDC definitions of nosocomial surgical site infections, 1992: A modification of CDC definitions of surgical wound infections. Infect Control Hosp Epidemiol, 1992; 13:606–608.

Horan

, Culver

, Gaynes

, et al. Nosocomial infections in surgical patients in the United States, January 1986–June 1992. National Nosocomial Infections Surveillance (NNIS) System. Infect Control Hosp Epidemiol, 1993; 14:73–80.

Smith

, Bohl

, McElearney

, et al. Wound infection after elective colorectal resection. Ann Surg, 2004; 239:599–605.

Wick

, Hirose

, Shore

, et al. Surgical site infections and cost in obese patients undergoing colorectal surgery. Arch Surg, 2011; 146:1068–1072.

Kirkland

, Briggs

, Trivette

, et al. The impact of surgical-site infections in the 1990s: Attributable mortality, excess length of hospitalization, and extra costs. Infect Control Hosp Epidemiol, 1999; 20:725–730.

Hedrick

, Sawyer

, Friel

, Stukenborg

. A method for estimating the risk of surgical site infection in patients with abdominal colorectal procedures. Dis Colon Rectum, 2013; 56:627–637.

Kiran

, El-Gazzaz

, Vogel

, Remzi

. Laparoscopic approach significantly reduces surgical site infections after colorectal surgery: Data from National Surgical Quality Improvement Program. J Am Coll Surg, 2010; 211:232–238.

Anthony

, Murray

, Sum-Ping

, et al. Evaluating an evidence-based bundle for preventing surgical site infection: A randomized trial. Arch Surg, 2011; 146:263–269.

Krieger

, Davis

, Sanchez

, et al. The use of silver nylon in preventing surgical site infections following colon and rectal surgery. Dis Colon Rectum, 2011; 54:1014–1019.

10.

Itani

, Wilson

, Awad

, et al. Ertapenem versus cefotetan prophylaxis in elective colorectal surgery. N Engl J Med, 2006; 355:2640–2651.

11.

Hedrick

, Heckman

, Smith

, et al. Efficacy of protocol implementation on incidence of wound infection in colorectal operations. J Am Coll Surg, 2007; 205:432–438.

12.

Berenguer

, Ochsner

Jr , Lord

, Senkowski

. Improving surgical site infections: Using National Surgical Quality Improvement Program data to institute Surgical Care Improvement Project protocols in improving surgical outcomes. J Am Coll Surg, 2010; 210:737–741.

13.

Hawn

, Vick

, Richman

, et al. Surgical site infection prevention: Time to move beyond the Surgical Care Improvement Program. Ann Surg, 2011; 254:494–499.

14.

de Lalla

. Antimicrobial prophylaxis in colorectal surgery: Focus on ertapenem. Ther Clin Risk Manag, 2009; 5:829–839.

15.

Wilson

, Treasure

, Sturridge

, Gruneberg

. A scoring method (ASEPSIS) for postoperative wound infections for use in clinical trials of antibiotic prophylaxis. Lancet, 1986; 1:311–313.

16.

Aktaş

, Topaloğlu

, Çalik

, et al. Wound surveillance with ASEPSIS in colorectal surgery. Turkish J Surg, 2012; 28:175–181.