Abstract
Abstract
Background:
Colon surgical site infections (SSIs) are being utilized increasingly as a quality measure for hospital reimbursement and public reporting. The Centers for Medicare and Medicaid Services (CMS) now require reporting of colon SSI, which is entered through the U.S. Centers for Disease Control and Prevention's National Healthcare Safety Network (NHSN). However, the CMS's model for determining expected SSIs uses different risk adjustment variables than does NHSN. We hypothesize that CMS's colon SSI model will predict lower expected infection rates than will NHSN.
Methods:
Colon SSI data were reported prospectively to NHSN from 2012–2014 for the six Fairview Hospitals (1,789 colon procedures). We compared expected quarterly SSIs and standardized infection ratios (SIRs) generated by CMS's risk-adjustment model (age and American Society of Anesthesiologist [ASA] classification) vs. NHSN's (age, ASA classification, procedure duration, endoscope [including laparoscope] use, medical school affiliation, hospital bed number, and incision class).
Results:
The patients with more complex colon SSIs were more likely to be male (60% vs. 44%; p = 0.011), to have contaminated/dirty incisions (21% vs. 10%; p = 0.005), and to have longer operations (235 min vs. 156 min; p < 0.001) and were more likely to be at a medical school-affiliated hospital (53% vs. 40%; p = 0.032). For Fairview Hospitals combined, CMS calculated a lower number of expected quarterly SSIs than did the NHSN (4.58 vs. 5.09 SSIs/quarter; p = 0.002). This difference persisted in a university hospital (727 procedures; 2.08 vs. 2.33; p = 0.002) and a smaller, community-based hospital (565 procedures; 1.31 vs. 1.42; p = 0.002). There were two quarters in which CMS identified Fairview's SIR as an outlier for complex colon SSIs (p = 0.05 and 0.04), whereas NHSN did not (p = 0.06 and 0.06).
Conclusion:
The CMS's current risk-adjustment model using age and ASA classification predicts lower rates of expected colon SSIs than does NHSN. This may lead to financial penalties because of the use of limited risk factors. Further efforts at elucidating appropriate risk adjustment measures without unnecessarily burdening hospitals with expensive data collection are necessary.
S
In 2012, NHSN instituted mandatory reporting of inpatient colon and hysterectomy (HYST) SSIs for acute care hospitals. Data submitted to the CDC also are reported to the Centers for Medicare and Medicaid Services (CMS), although using a different set of criteria for counting and risk-adjusting observed and expected infections. This quality measure was posted to the public on the Hospital Compare website beginning in late 2012 and demonstrates how a hospital measures up to the national standard for SSIs and other outcomes [6]. In the fiscal year 2016, this measure will be implemented in CMS's Hospital Value-Based Purchasing (VBP) program under the Affordable Care Act of 2010 based on data already collected [6]. Under this program, a percentage of previously allocated funds (determined according to the patient's diagnosis related-group or DRG) are paid only if the hospital achieves certain benchmarks of achievement or improvement. An estimated $1.4 billion potentially is at stake for VBP-participating U.S. hospitals for the fiscal year 2015 [7]. Also, in 2016, colon and HYST SSIs will be added to CMS's Hospital Acquired Condition (HAC) Reduction Program under Domain 2, healthcare-associated infections. Under this program, CMS reduces payments to hospitals ranking among the lowest-performing quartile of these measures [7]. An SSI after colon surgery is linked to these reimbursements with ever-increasing weight. By fiscal year 2017, value-based surgical outcome measures will account for 2% of the VBP score and an even larger portion of the HAC Reduction Program [6,8]. The CMS already estimates approximately 724 hospitals will see a decrease in payments as a result of FY2015 HAC results [7].
The objective of this study was to compare the CMS and NHSN expected complex colon SSI rates and surveillance methods. These two SSI reporting models were chosen for comparison given their mandatory status and integrated relation and the reputation and reimbursement consequences of these results. Despite using the same definition for complex SSI and an integrated data entry portal, these discrete models use different risk adjustment variables to determine a hospital's expected colon SSI rate, with the above-described financial consequences dependent on the CMS model. Additionally, CMS uses other methods of surveillance for detection of observed infections. We hypothesized that the CMS complex colon SSI risk adjustment model would predict lower infection rates per quarter than did the NHSN model, given its use of a smaller set of variables.
Patients and Methods
The University of Minnesota Institutional Review Board approved this study.
We utilized prospectively reported data on all inpatient colon procedures performed within the Fairview Hospital System (six hospitals; 1,789 procedures) from January 1, 2012, to December 31, 2014. These data were reported to NHSN (NHSN/CDC, which then sends them to CMS to satisfy the mandatory reporting requirement using a separate model that uses only two of the following listed risk factors) along with gender, age, anesthesia type, American Society of Anesthesiologists (ASA) classification, emergency procedure (yes/no), procedure duration, endoscope [including laparoscope] use, medical school affiliation, hospital bed number, incision classification, inpatient status, admission date, event date, and number of post-operative surgical infections for this time period. Beginning in 2013, NHSN also required reporting of body mass index (BMI) and the presence or absence of diabetes. However, given the lack of complete data, these latter items were not included in our analysis.
Complex SSIs for both systems were classified according to the CDC's definitions and include deep soft tissue and organ/space infections. This includes infections involving the fascia/muscle or any part of the body deep to this that is opened or manipulated during the procedure [9]. Complex SSI does not include cellulitis or superficial incisional infections involving the skin or subcutaneous tissue. Complex colon SSIs are counted only for surgical sites with primary skin closure or those using wicking, drains, or other material in a primarily closed incision [9]. If two procedures are performed through the same incision, the SSI is attributed to the highest assigned-risk procedure according to the NHSN Principal Operative Procedure Category Selection List [9]. Colon surgery moved from sixth to second on this list in 2013, with liver transplant now being the highest assigned risk procedure.
Data sources
CMS Surveillance (Observed Infections)
Data obtained through CMS's surveillance of SSI include infections detected during primary admission, readmission to the initial hospital, readmission to an outside hospital, or infection identified at follow-up visit or through post-discharge surveillance. Observed infection rates were recorded for patients older than 18 years undergoing inpatient surgery. Surveillance is initiated on postoperative day 1; therefore, the detection period ends 31 days from the operative day [10].
NHSN Surveillance (Observed Infections)
Data obtained through NHSN's surveillance include complex SSIs found during primary admission or readmission to the hospital where the procedure was performed within 30 days of inpatient surgery. Observed infection rates were recorded for patients older than 1 day and less than 109 years undergoing inpatient surgery, although for this study, a standardized infection ratio (SIR) was calculated only for those more than 18 years old. Surveillance is initiated on the operative day; therefore, the detection period ends 30 days after the operative day [9].
CMS Risk Adjustment (Expected Infections)
The CMS risk adjustment includes increasing age and increasing ASA classification. These variables were chosen in collaboration with the American College of Surgeons and released in 2012. Expected infections are calculated on the basis of the number of colon procedures performed during a quarter and were risk adjusted according to the variables listed [10].
NHSN Risk Adjustment (Expected Infections)
The NHSN's risk adjustment variables were implemented in 2012 after a large study of voluntarily reported data from more than 1,900 hospitals to NHSN from 2006–2008. That analysis found that the following factors confer a greater risk of complex colon SSI: Age ≤75, ASA classification >2, increasing procedure duration in increments of 10 min, non-endoscopic/laparoscopic procedure, no medical school affiliation, hospital bed size >200, and site classification as contaminated or dirty [11]. Expected infections were calculated using a logistic regression model described previously by NHSN using the risk factors described [12]. This value is used to calculate the probability of an SSI for each patient and then summed to give the total expected complex colon SSIs for the group of patients having colon procedures during that quarter.
Statistical analyses
Data preparation and analyses were performed using SPSS version 22 (IBM, Armonk, NY). Comparisons of expected quarterly infection rates were performed using related-samples Wilcoxon signed-rank tests. A χ2 test was used to compare demographics between patients with a complex colon SSI and those without (includes no infection and superficial infection). Expected infection counts for both CMS and NHSN are calculated by NHSN's web-based application [12]. Within each model, NHSN also divides the observed infection count by the expected infection count, giving an O:E ratio known as the Standardized Infection Ratio. If the SIR was >1, and the calculated SIR confidence interval did not include 1, the hospital was identified as an outlier for this quality measure.
Results
The six Fairview hospitals range in size from 52 to 882 beds (Table 1). Two hospitals had greater than 200 beds, and one hospital was affiliated with a medical school. The median number of inpatient colon procedures performed in 2014 at Fairview Hospitals was 67 (range 14–254 procedures). In 2014, 68,162 total procedures were performed at all the hospitals.
2014 National Healthcare Safety Network database.
Fairview database.
UMMC = University of Minnesota Medical Center.
A complex colon SSI was identified in 4% of the patients reported from the Fairview system. The median time to identification of complex SSI was 13.5 days. A superficial colon SSI was identified in an additional 4% of patients, with a median time to the event of 12 days.
Demographics for all patients undergoing colon procedures, including a breakdown across hospitals, are illustrated in Table 2. In aggregate, they had the following NHSN risk factors: 89% underwent open (non-laparoscopic) procedures, 59% were treated at a hospital that had no medical school affiliation, 82% were age ≤75 years, 50% had an ASA classification >2, and 11% had contaminated/dirty incisions. The median procedure duration was 159 minutes for all patients undergoing colon procedures.
Hospitals with fewer than 200 beds were the Lakes, Ridges, Northland, and Range hospitals.
ASA = American Society of Anesthesiologists; CO = contaminated; D = dirty; SSI = surgical site infection; UMMC =University of Minnesota Medical Center.
In comparing the patients with complex colon SSIs with those without, more patients were male (60% vs. 44%; p = 0.011), more had an incision class of contaminated/dirty (21% vs. 10%; p = 0.005), they had a longer duration of surgery (235 min vs. 156 min; p < 0.001), and more were operated on in a medical school-affiliated hospital (53% vs. 40%; p = 0.032). There were no significant differences between the two groups in NHSN identified risk factors of percentage of open procedures, age ≤75 years, or ASA >2, or those associated with a hospital bed size >200 (Table 3).
ASA = American Society of Anesthesiologists; CO = contaminated; D = dirty.
Differences in observed infection numbers attributable to differing 30-day post-operative surveillance methods were noted between CMS and NHSN (Fig. 1). During the study period, NHSN detected 67 complex colon SSIs, whereas CMS detected 70. The NHSN's missing SSIs were detected through CMS's post-discharge surveillance in addition to two readmissions to an outside hospital.

National Healthcare Safety Network vs. Centers for Medicare and Medicaid Services observed complex colon surgical site infections, 2012–2014.
The observed and expected complex colon SSI rates per quarter for all Fairview Hospitals across the study period are shown in Fig. 2. Each quarter, CMS calculated a lower expected infection number than did NHSN. The CMS's median expected SSI rate was significantly lower than NHSN's at 4.58 vs. 5.09 infections per quarter, respectively (p = 0.002; (Fig. 3). This difference was again observed when comparing rates within a large-volume, medical school-affiliated hospital (727 procedures; 2.08 vs. 2.34 infections per quarter; p = 0.002) and a smaller-volume, non-medical school-affiliated hospital (565 procedures; 1.31 vs. 1.42 infections per quarter; p = 0.002).

The Centers for Medicaid and Medicare Services risk adjustment model predicts consistently lower expected complex colon (COLO) surgical site infections (SSIs) than does the National Healthcare Safety Network model.

Median expected quarterly complex colon surgical site infection rates across institutions: Centers for Medicaid and Medicare Services vs. National Healthcare Safety Network. UMMC = University of Minnesota Medical Center, *p = 0.002.
In examining the quarterly SIRs calculated by NHSN for each model (observed divided by expected infections), CMS was consistently greater than NHSN throughout the study period (Fig. 4). In addition, Fairview had two quarters in which CMS identified the SIR as >1 and an outlier (p = 0.05 and 0.04). Conversely, NHSN's SIR for those same quarters did not reach statistical significance (p = 0.06 and 0.06, respectively). Therefore, Fairview would not have been identified as an outlier using this model.

The Centers for Medicaid and Medicare Services (CMS) calculate a higher SIR per quarter from 2012–2014 than does the National Healthcare Safety Network (NHSN). CI = confidence interval. *p = 0.05 for CMS SIR, ‡ p = 0.04 for CMS SIR,
Discussion
The modern paradigm shift from volume- to value-based performance assessment for hospital reimbursement and reputation has pushed SSIs to the forefront of surgical outcome assessment. Unfortunately, given the burden of high-volume data collection, numerous systems are confronted with the challenge of attempting to capture complete data or an accurate representation of the population and to identify risk factors for infection in a dynamic environment. The limitations and downstream effects of risk factors identified through statistical modeling also must be considered. A model is only as strong as its data, the integrity of which can be influenced by the motivation for its input. Including risk factors in pay-for-performance models also can impact patient selection, efforts to minimize the perceived risk of SSI, and peri-operative documentation, all of which can affect the accuracy of measurement of this quality indicator.
Our study demonstrated that the CMS reporting model expected a lower complex colon SSI rate per quarter for Fairview Hospitals than did the NHSN reporting model. The use of age and ASA classification alone for the CMS report appears to underestimate the expected number of colon SSIs. Factors that put patients at a known greater risk for post-operative SSI are excluded from the CMS reporting model. Additionally, using only two risk variables increases the impact each variable has on the overall risk adjustment model. This could be problematic in the CMS model, given previously documented concerns regarding the inter-rater reliability of ASA classification among anesthesia providers that has been demonstrated in several studies [13,14,15]. Additionally, decreasing, not increasing, age was demonstrated to confer a greater risk for colon SSI in a 2011 study [11]. This outright contradiction between the two models is disconcerting, as is the use of only a few variables to risk-adjust an expected number of infections for a diverse case mix and complex patient population.
Development of a colon SSI clearly is multifactorial. Bacterial inoculum and virulence are pitted against innumerable environmental factors at the surgical site in addition to the host response [16]. A recent study examined the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) data for more than 27,000 patients from 305 hospitals to determine risk factors associated with superficial and deep/organ-space SSI separately [17]. The investigators found that regular smoking within 1 year prior to surgery; a post-operative diagnosis of fistula, inflammatory bowel disease, or obstruction/perforation; BMI >35; operative time above the fifth decile; disseminated cancer; and radiation therapy were associated with an increased odds ratio of deep/organ-space SSI. Additionally, laparoscopic surgery, age >55 years, and totally dependent functional status were associated with a significantly lower risk-adjusted odds ratio for deep SSI. Numerous other researchers also have attempted to elucidate factors that are clinically or statistically associated with increasing SSI risk for patients. However, no changes have been made yet to the CMS risk adjustment model [16,18,19,20].
The NHSN reporting model is not without controversy either. With a c-index reported by Mu et al. of 0.61, this model fit is hardly better than chance. As noted by the authors of that paper, the c-indices potentially remain low because of the lack of procedure-specific risk factors [11]. This model also is based on voluntarily reported variables of convenience from almost 10 years ago, several of which are considered surrogate markers for patient populations and complexity, such as hospital bed number and medical school affiliation. As discussed by the authors of that study, historical modeling for public reporting has limited function in the changing quality measure environment [11]. Changes over time in diagnosis-specific treatment approach, pre-operative bowel preparation, antibiotic stewardship, peri-operative scrub selection, site care, peri-operative glucose control, and quality improvement bundles have the potential to alter a patient's risk of infection. Thus, risk factors important in 2006 may no longer be relevant almost a decade later. Further, databases are subject to keying errors, coding discrepancies, and misinterpretation of clinical documentation. Numerous reports of discrepancies in incision classification have been made, including as much as a 63% discordance rate for pediatric general surgery procedures [21,22,23]. Of the seven NHSN risk factors, we demonstrated in our study that only two were higher in the complex colon SSI group: Wound class and procedure duration. The remaining risk factors were not significantly greater in the SSI group, and in fact, medical school affiliation was greater in the SSI group. Mu et al. found that no medical school affiliation confers a greater risk of infection [11]. Additionally, Young et al. reported that half of the NHSN variables were not supported or were even contradicted by the literature as conferring a greater risk of SSI [18]. Taken cumulatively, the inability to achieve an unrealistic benchmark because of the unreliable quality metrics may impact both quality improvement efforts and patient selection at institutions that are being penalized unfairly.
As we have demonstrated in Figure 4, seemingly small differences in expected infection rates can result in statistically significant consequences in the SIR. This, in turn, can have a powerful impact on the overall assessment of a hospital's performance. This is reflected in public reporting and reimbursement for patient care. As CMS continues to shift toward increasing the weight of value-based performance, this impact will continue to grow as SSI is incorporated as a larger percentage of the VBP and HAC Reduction Program, potentially affecting several millions of dollars for patient care in a hospital system such as the one in our study. Additionally, discrepancies found between these two closely integrated systems, CMS and NHSN, calls into question the accuracy of both risk adjustment and SSI monitoring models. Discrepancies between SSI monitoring systems also have been noted between NHSN and the NSQIP data, with dissimilar observed SSI rates and no correlation between the O:E ratios determined by each model [24]. Although some redundancy within a hospital can provide opportunities for auditing and information sharing, inconsistency in duplicated efforts of data collection merely become confusing for clinicians, infection monitoring specialists, and administrators seeking to implement initiatives for SSI reduction.
The limitations of this study should be taken into consideration. The use of a single hospital system may introduce a sampling bias, which may not represent demographically different populations. However, this study does include hospitals ranging from 52–882 beds in an effort to capture a diverse patient mix. Hospital characteristics such as medical school affiliation, number of beds, surgical infection surveillance staff, and discrepancies in assignment of procedure codes, incision class, and ASA class may affect the results of the study. This was a retrospective review of prospectively recorded data and may be subject to selection bias. Finally, we were unable to include hospitals with <200 beds in the subgroup analysis because of the small numbers of colon procedures performed at each institution. When the expected quarterly colon SSI is calculated to be <1, this means the case numbers are too low for NHSN to calculate a precise SIR [12].
Conclusion
The CMS and NHSN risk adjustment reporting models predict different expected complex colon infection rates, leading to significant differences in the SIR. This may induce inappropriate penalties for hospitals and clinicians. Variables used by each model have been called into question by several studies, and NHSN's use of decreasing age vs. CMS's use of increasing age is frankly contradictory despite the integrated relation of these organizations. Improvement and unification of risk adjustment and SSI surveillance is imperative to ensure our hospitals are being fairly assessed and that quality improvement efforts are being focused in a clinically impactful direction. Additionally, pay-for-performance measures should be implemented carefully, as they can have unintended downstream consequences, such as exclusion of severely ill or high-risk patients or delay of treatment resulting in worse outcomes and higher cost of care.
Further efforts at elucidating appropriate risk adjustment measures without unnecessarily burdening hospitals with expensive data collection are vital to efforts at quality improvement in U.S. hospitals. At a minimum, reassessment of existing models should be performed on a regular basis with modern data. Consideration should be given to empiric and literature based and patient and institutional factors as new models are developed.
Footnotes
Author Disclosure Statement
No competing financial interests exist for any of the authors.
