Comparison of Surveillance of Surgical Site Infections by a National Surveillance Program and by Institutional Audit

Patients and Methods

Results

Data from the national surveillance program Swissnoso were available for 301 patients. One hundred fifteen of these patients (38%) were not part of the ERAS program and were thus excluded; two-thirds of these patients (n = 77) underwent emergency surgery. Therefore, 186 patients (62%) were eligible for the present analysis. Of those, 18 patients (10%) were excluded because of refusal to participate and 29 (15%) were lost to follow-up by the EIAS surveillance system because patients did not attend control visits at the hospital (foreign/out of state). Of these 29 patients, four developed SSI before discharge and were included in the sample. The final sample consisted of 143 patients (Fig. 1).

OPEN IN VIEWER

FIG. 1.

Patient flow.

Demographic and surgical details and LOS are displayed in Table 2 for the patients included in the analysis and as comparison for the 25 patients lost to follow-up by EIAS. Note that none of the characteristics of the excluded patients differed from the sample.

Overall, SSI rate was 15% according to Swissnoso assessment and 14% according to EIAS. Of note, among the 25 patients excluded because of insufficient information provided by EIAS, Swissnoso recorded three SSI (12%).

Cross-tabulation of coded infections according to the two surveillance systems is displayed in Table 3. Inter-observer reliability was high (Cohen k value of 0.801, p < 0.001). Compared with Swissnoso, EIAS recording had a sensitivity of 81% (95% confidence interval [CI] = 58.09%–94.55%) and a specificity of 97.5% (95% CI = 92.98%–99.49%) for detection of SSI. Positive and negative predictive values were 85% (95% CI = 64.52%–94.64) and 96.8% (95% CI = 92.49%–98.63%), respectively, yielding an accuracy of 95.1% (95% CI = 90.17%–98.01%).

Discrepant findings between both surveillance systems were found in seven patients. Four additional SSI were described according to Swissnoso but not EIAS, and three additional SSI coded in EIAS were not in the Swissnoso database. Details including SSI criteria and potential explanations for erroneous documentation in EIAS are summarized in Table 4. EIAS failed to declare four SSI (patients 1–4), whereas two infections at distant sites were erroneously declared as SSI (patients 5–6). Swissnoso, however, failed to declare the remaining patient as SSI (patient 7), even though SSI was confirmed through chart review.

Because EIAS follow-up assessment of SSI relied on post-operative consultations rather than the standard assessment 30 days after surgery, we conducted an additional analysis of the days between operation and post-operative consultation. Dates for follow-up were available for 126 patients; for the 13 patients followed by other wards or hospitals, no dates were provided. On average, the post-operative consultations took place 46.06 days after the surgery (standard deviation [SD] = 13.38).

Discussion

The present study revealed concordance in SSI assessment between both surveillance systems, Swissnoso and EIAS, yielding good inter-observer reliability. Both systems may thus be considered reliable tools for surveillance of large numbers of surgeries. Whereas EIAS provides more in-depth information on type of infection, fewer patients were lost to follow-up by Swissnoso, because EIAS is solely based on patient information available at the hospital.

Surgical site infection is the most common complication after colorectal surgery with rates of up to 30% [17] and is a major concern for colorectal surgery [10]. Infection prevention bundles including fostering of a cohesive environment, standardization, and reduction in operative variance have been identified as useful tools to decrease SSI [18,19]. However, to assess SSI rates reliably, the quality of follow-up and surveillance is fundamental [3]. Low-quality surveillance might result in falsely low SSI rates, creating drawbacks for patients, caregivers, and policymakers.

Although results yield an overall good inter-rater reliability between the two surveillance systems, four SSIs were missed and three were erroneously coded as SSI in the EIAS database. Of note, all discrepant coding concerned superficial but no deep or organ/space SSI. Detailed chart review of patients with discrepant coding revealed erroneous documentation by ERAS in two cases because EIAS does not include patient follow-up through phone calls. Systematic phone calls as performed by Swissnoso allowed for more complete surveillance. Note, however, that patient reports about SSI have been found to be biased and unreliable [6,20]. The other misdiagnosed cases were mainly because of diverging interpretation of clinical symptoms, logistical reasons, or coding mistakes of non-procedure as procedure-related infections, leaving room for improvement in data interpretation and documentation in the EIAS system.

To date, a large variety of surveillance methods has been described [21], but studies assessing comparative reliability of the different tools are still scarce [6,9]. A systematic review across assessment tools revealed problems in compliance with surgeon or patient questionnaires [6], and patients' and surgeons' assessment yielding divergent results. A recent study comparing assessment of complications by EIAS to the National Surgical Quality Improvement Program (NSQIP) methodology stresses the importance of accounting for differences in coding definitions [9). In addition, the different methodologies used complicated the identification of a gold standard [6]. Nevertheless, most reports agree on the need for standardized definitions as basis for a valid approach to survey post-discharge SSI. In the United States, the tracking and reporting of SSI has become an important element for quality improvement purposes. The majority of U.S. hospitals participate in the National Health Safety Network (NHSN) to monitor and report SSI through reporting guidelines [22]. The American College of Surgeons National Surgical Quality Improvement Project (ACS-NSQIP) on the other hand uses trained clinical abstractors and well-defined abstraction guidelines [23]. More recently, a machine learning algorithm has been tested for real-time SSI monitoring, potentially leading to improved surveillance [24].

Both Swissnoso and EIAS are based on standardized definitions of SSI and the assessment process. However, recent analyses revealed a wide variation of SSI surveillance quality in Switzerland, with quality of chart review and accuracy of data collection identified as main areas for improvement [3]. It may be more difficult to establish a high-quality data collection in smaller, rural or non-academic centers, with the risk of less accurate detection of SSI, stressing the importance of standardized training and continuous quality management [25].

Nevertheless, Swissnoso is considered an established SSI surveillance tool due to its National scale and its CDC recognized methodology. Validation of EIAS as SSI surveillance tool is of particular importance since EIAS provides data on perioperative care in general, with SSI being only one among numerous aspects evaluated by the ERAS protocol [9]. The present results are encouraging, considering that both studied surveillance systems were completely independent from each other regarding coding, definitions, assessment methods, and staff.

The study has limitations. First, although data collection was prospective in both surveillance systems, the data analysis for the current study was performed retrospectively, leading to inherent limitations. In particular, the EIAS surveillance system included only patients whose care followed the ERAS protocol guidelines. Patients not formally included in the program, mostly emergencies, were not included in the comparison. Thus, the results cannot be generalized for the surveillance of emergency cases. Second, the cohort was rather small, and data were collected regarding a single type of procedure in one hospital only. Further research is needed to replicate and extend the current findings.

Because both Swissnoso and EIAS rely on detailed protocols to assess SSI and offer standardized training programs for study nurses in charge of the surveillance, bias in the use of the protocol is expected to be minimal. However, mission and methodological approach differ considerably between the two systems, and different definitions of codes may lead to bias [9]. Beyond the accuracy, further criteria may be considered when elaborating recommendations. First, the goal of the surveillance must be considered. Swissnoso offered a comprehensive and systematic surveillance, standardized on the national scale, allowing for comparisons across hospitals. ERAS Interactive Audit System did not offer this possibility, although national and international comparisons may become possible as more hospitals participate in the ERAS program. Second, the assessment relies on resources. Swissnoso surveillance needs specific additional resources in terms of healthcare professionals dedicated to the collection of data not otherwise available, including the patient interview. ERAS Interactive Audit System is based solely on in-hospital and outpatient assessment by treating physicians but not dedicated surveillance specialists and information from patient charts, however, at the cost of a less systematic follow-up and thus a slightly less accurate surveillance.

In conclusion, the present study demonstrated the reliability of EIAS in assessing SSI, providing that patients' follow-up consultations can actually be assessed. Where a nationwide comprehensive surveillance program does not exist, assessment through ERAS data might thus represent a valid alternative.