External Validation of the Ventral Hernia Risk Score for Prediction of Surgical Site Infections

Abstract

Background:

Previously, we reported that the Ventral Hernia Risk Score (VHRS) was more accurate in a Veterans Affairs (VA) population in predicting surgical site infection (SSI) after open ventral hernia repair (VHR) compared with other models such as the Ventral Hernia Working Group (VHWG) model. The VHRS was developed using single-center data and stratifies SSI risk into five groups based on concomitant hernia repair, skin flaps created, American Society of Anesthesiologists (ASA) score ≥3, body mass index ≥40 kg/m², and incision class 4. The purpose of this study was to validate the VHRS for other hospitals.

Methods:

A prospective database of all open VHRs performed at three institutions from 2009–2011 was utilized. All 436 patients with a follow-up of at least 1 mo were included. The U.S. Centers for Disease Control and Prevention (CDC) definition of SSI was utilized. Each patient was assigned a VHRS, VHWG, and CDC incision classification. Receiver-operating characteristic curves were used to assess predictive accuracy, and the areas under the curve (AUCs) were compared for the three risk-stratification systems.

Results:

The median follow-up was 20 mos (range 1–49 mos). During this time, 111 patients (25.5%) developed a SSI. The AUC of the VHRS (0.73; 95% confidence interval [CI] 0.67–0.78) was greater than that of the VHWG (0.66; 95% CI 0.60–0.72; p<0.01) and the CDC incision class (0.68; 95% CI 0.61–0.74; p<0.05).

Conclusions:

The VHRS provides a novel, internally and externally validated score for a patient's likelihood of developing a SSI after open VHR. Elevating skin flaps, ASA score ≥3, concomitant procedures, morbid obesity, and incision class all independently predicted SSI. It remains to be determined if pre-operative patient selection and risk reduction, surgical techniques, and post-operative management can improve outcomes in the highest-risk patients. The VHRS provides a starting point for key stakeholders to discuss the management of ventral hernias.

Ventral hernia repairs are among the most common general surgery procedures, with more than 365,000 performed in the United States each year [1]. Complications, particularly surgical site infections (SSIs), are common after a repair. Surgical site infections are not only associated with a host of other complications, such as recurrence and need for re-operation, but also significantly increase costs [1,2].

Stratifying patients' risk of SSI may provide valuable information for patient selection, counseling, risk modification, and surgical management. Several classification systems have been developed to quantify patient and operative factors that predict poor outcomes. The Ventral Hernia Working Group (VHWG) and the U.S. Centers for Disease Control and Prevention (CDC) incision class are models for stratifying patient risk for SSI [3,4]. Previously, we reported that the Ventral Hernia Risk Score (VHRS) was more accurate in a Veterans Affairs (VA) population in predicting SSI after open ventral hernia repair (VHR) than were other models such as the VHWG. The VHRS was developed using single-center data and stratifies SSI risk into five groups on the basis of concomitant hernia repair, creation of skin flaps, American Society of Anesthesiologists (ASA) score ≥3 points, body mass index (BMI) ≥40 kg/m², and incision class 4 or (dirty) [5]. We hypothesized that the VHRS is predictive of SSI in an external validation set of patients who have undergone open VHR.

Patients and Methods

We utilized a prospective database of all open VHRs performed at three institutions from 2009 to 2011. All three institutions were academic, tertiary-care referral centers where VHRs were performed by surgeons with a special interest in ventral hernia surgery. All patients followed for at least 1 mo were included in the analysis.

Patient demographics and clinical, pre-operative, and intra-operative data were collected. Demographic data consisted of race and gender. The institution where the surgery was performed was included. Clinical factors included BMI, history of diabetes mellitus, history of chronic obstructive pulmonary disease (COPD), current smoker, immunosuppression, abdominal aortic aneurysm, prostate disease, serum albumin concentration, and ASA score. Hernia factors included recurrence, previous SSI of the same incision, and VHWG score. Intra-operative data included operative duration, appropriate pre-operative antibiotic administration, concomitant abdominal surgery, incision class, mesh utilization, type of mesh applied (if used), location of mesh placement, fascial closure, fascial release, and creation of skin flaps. Immunosuppression was defined as use of any corticosteroid, chemotherapy, or other immunosuppressive medication. Pre-operative antibiotic administration was defined as appropriate dose, type, and administration of antibiotic(s) as specified by the Surgical Care Improvement Project (SCIP). Concomitant surgery was defined as any hernia repair during a procedure for another surgical indication (e.g., cholecystectomy, colectomy, panniculectomy) through the same incision or intra-operative enterotomy/bowel resection. Location of mesh placement included onlay, inlay, or underlay (including intra-peritoneal, pre-peritoneal, and retro-rectus) locations. Fascial release was defined as any procedure involving incision of the fascia to achieve greater medial release, including component separation. Skin flaps were defined as any dissection of skin and subcutaneous tissue off the fascia without complete excision of the dissected tissue.

The primary outcome was the accuracy of the three models in predicting SSI. To determine the presence of SSI, we used the guidelines described by the CDC [6]. The previously identified model was assessed with this validation patient cohort. Akaike information criterion (AIC), goodness of fit, and Hosmer-Lemeshow test results were assessed. The accuracy of VHRS was then compared with that of the VHWG and CDC incision class. Receiver-operating characteristic curves (ROCs) were used to assess predictive accuracy, and the areas under the curves (AUCs) were compared for the models. The test described by Delong et al. was used to test the ROC curves for statistically significant differences [7].

Other outcomes measured were re-operation or recurrence at any point in the post-operative period. Re-operation was defined as any post-operative procedure that involved the mesh, fascia, or intra-peritoneal cavity. Recurrence was identified by clinical assessment. If there was clinical uncertainty and symptoms, a computed tomography scan was performed, and the imaging results were included in the assessment.

Differences between the two groups (those who developed SSI and those who did not) were compared using the Pearson χ² or Fisher exact test. Continuous data were analyzed using the two-tailed t-test or Mann-Whitney U test for parametric and nonparametric data, respectively. Comparisons between the results at different institutions were accomplished using analysis of variance and the Pearson χ² test. All statistical analyses were performed using SPSS software version 21 (IBM Corp., Armonk, NY).

Results

A total of 436 patients underwent ventral hernia repair. Eleven patients were excluded for failure to be followed up for at least 30 d. There were multiple significant differences in patient characteristics among institutions (Table 1).

Table 1.

Comparison of Institutions

	Hospital 1 (n=166)	Hospital 2 (n=139)	Hospital 3 (n=131)	p
Ethnicity (%)				<0.01
White	121 (72.9)	133 (95.7)	121 (92.4)
Non-white	45 (27.1)	6 (4.3)	10 (7.6)
Male (%)	158 (95.2)	47 (33.8)	61 (46.6)	<0.01
Clinical factors
Mean BMI (SD)	30 (5.8)	35 (7.4)	35 (11.5)	<0.01
DM (%)	51 (30.7)	39 (28.1)	49 (37.4)	0.24
Active smoker (%)	47 (28.3)	41 (29.5)	21 (16.0)	0.02
Recurrent (%)	22 (13.3)	90 (64.7)	73 (55.7)	<0.01
Prior VHR (%)	21 (12.7)	90 (64.7)	66 (50.4)	<0.01
Outcomes (%)
Recurrence	17 (10.2)	12 (8.6)	15 (11.5)	0.73
Surgical site infection	40 (24.1)	31 (22.3)	40 (30.5)	0.01
VHR factors (%)
Concomitant hernia repair	34 (20.5)	10 (7.2)	43 (32.8)	<0.01
Skin flaps created	42 (25.3)	91 (65.5)	29 (22.1)	<0.01
ASA score ≥3	116 (69.9)	94 (67.6)	99 (75.6)	0.33
BMI ≥40	8 (4.8)	27 (19.7)	34 (28.6)	<0.01
Incision class 4	5 (3.0)	16 (11.5)	19 (14.5)	<0.01

ASA=American Society of Anesthesiologists; BMI=body mass index; DM=diabetes mellitus; SD=standard deviation; VHR=ventral hernia repair.

The overall SSI rate was 25.5% (n=111). Median (range) of follow-up was 20 (1–49) mos. The rates of recurrence and re-operation were 10.1% (n=44) and 13.5% (n=59), respectively. The rates of re-operation (27.9% vs. 8.6%; p<0.01) and hernia recurrence (23.4% vs. 5.6%; p<0.01) were significantly higher among patients with SSI compared with those without SSI. Clinical factors associated with SSI included elevated BMI, diabetes mellitus, immunosuppression, and ASA score ≥3 (Table 2). Significant surgical history included a recurrent VHR, number of prior VHRs, and history of SSI. Significant intra-operative factors included concomitant surgery, mesh placement, incision class, longer operative time, fascial release, and skin flap creation (Table 3).

Table 2.

Baseline Characteristics of Patients Who Developed an SSI Versus Those Who Did Not

	No SSI (n=325)	SSI (n=111)	p
Race (%)			0.88
White	280 (86.2)	95 (85.6)
Non-white	45 (13.8)	16 (14.4)
Male (%)	213 (65.5)	53 (47.7)	<0.01
Institution			0.62
1	128 (39.4)	38 (34.2)
2	101 (31.1)	38 (34.2)
3	96 (29.5)	35 (31.5)
Clinical factors
Mean BMI (standard deviation [SD])	32.2 (8.3)	35.7 (8.7)	<0.01
Diabetes mellitus (%)	90 (27.7)	49 (44.1)	<0.01
COPD (%)	76 (23.4)	20 (18.0)	0.29
Active smoker (%)	72 (22.2)	37 (33.3)	0.02
Immunosuppressed (%)	12 (3.7)	3 (2.7)	0.77
AAA (%)	6 (1.8)	2 (1.8)	1.00
Prostate disease (%)	37 (11.4)	12 (10.8)	1.00
Mean albumin (SD)	3.7 (0.6)	3.4 (0.8)	0.06
ASA (%)			<0.01
1	4 (1.2)	0 (0.0)
2	108 (33.2)	15 (13.5)
3	194 (59.7)	87 (78.4)
4	19 (5.8)	9 (8.1)
Surgical history
Recurrent hernia (%)	124 (38.2)	61 (55.0)	<0.01
Prior SSI (%)	37 (16.5)	22 (30.1)	0.02
VHWG score (%)			<0.01
1	56 (17.2)	8 (7.2)
2	207 (63.7)	52 (46.8)
3	45 (13.8)	28 (25.2)
4	17 (5.2)	23 (20.7)

AAA=abdominal aortic aneurysm; ASA=American Society of Anesthesiologists; BMI=body mass index; COPD=chronic obstructive pulmonary disease; SSI=surgical site infection; VHWG=Ventral Hernia Working Group.

Table 3.

Surgical Data of Patients Who Developed an SSI Versus Those Who Did Not

	No SSI (n=325)	SSI (n=111)	p
Median operative time, min (range)	1,466 (142.0)	186 (130.0)	<0.01
Pre-operative antibiotic (%)	305 (93.8)	108 (97.3)	0.22
Incision class (%)			<0.01
1	235 (72.3)	47 (42.3)
2	56 (17.2)	24 (21.6)
3	17 (5.2)	17 (15.3)
4	17 (5.2)	23 (20.7)
Concomitant procedure (%)	57 (17.5)	30 (27.0)	0.04
Mesh (%)	215 (66.2)	88 (79.3)	0.01
Type of mesh (%)			<0.01
Biologic	114 (35.1)	67 (60.4)
Synthetic	97 (29.8)	20 (18.0)
Location of mesh (%)			<0.01
Onlay	8 (3.7)	7 (10.0)
Underlay	100 (45.7)	37 (52.9)
Inlay	1 (0.5)	3 (4.3)
Primary fascial closure (%)	303 (93.2)	105 (94.6)	0.82
Fascial release (%)	86 (26.2)	48 (43.2)	<0.01
Skin flaps (%)	97 (29.8)	65 (58.6)	<0.01

As with the development cohort from the VA population, the VHRS was strongly associated with the rates of SSI in the validation cohort consisting of the other institutions (Tables 4 and 5). The model demonstrated little loss of information (AIC=87.47) and an adequate goodness of fit (Hosmer-Lemeshow test p=0.097) [4]. All variables used to define the VHRS were found to be independent predictors of SSI (Tables 4 and 5) [4]. A ROC curve (Fig. 1) demonstrated that the VHRS had an AUC of 0.73 (95% CI 0.67–0.78) compared with the VHWG (AUC=0.66; 95% CI 0.60–0.72; p=0.01) and CDC incision class (AUC=0.68; 95% CI 0.61–0.74; p<0.05). The VHRS AUCs for hospitals 1, 2, and 3 were 0.75, 0.68, and 0.73, respectively.

FIG. 1.

Receiver operator characteristic curve for accuracy of Ventral Hernia Working Group score (VHWG), Ventral Hernia Risk Score (VHRS), and U.S. Centers for Disease Control and Prevention (CDC) wound classification in predicting surgical site infection.

Table 4.

Ventral Hernia Risk Score for Surgical Site Infection

	Validation Cohort		Development Cohort
Variable	OR	95% CI	OR	95% CI
Concomitant hernia repair	1.7	>1.0–2.9	2.1	1.4–3.3
Skin flaps created	3.3	2.1–5.2	2.3	1.6–3.5
ASA score ≥3	3.4	1.9–6.1	2.1	1.4–3.2
BMI ≥40	2.4	1.4–4.1	3.2	1.7–5.9
Wound class 4	4.7	2.4–9.3	6.8	3.2–15.4

Validation model characteristics

Akaike information criterion: 87.47

Goodness of fit: 25.27

Hosmer-Lemeshow test: Step 1, χ² 10.72, degrees of freedom 6, significance 0.097.

ASA=American Society of Anesthesiologists; BMI=body mass index; CI=confidence interval; OR=odds ratio.

Table 5.

Surgical Site Infection Rate by Ventral Hernia Risk Score

Group	Points	Rate of SSI Validation Cohort (%)	Rate of SSI Development Cohort (%)
1	0	4	8
2	2–3	18	19
3	4	29	29
4	5–10	40	54
5	11–16	63	82

Discussion

In this study, we validated externally the VHRS and its predictive capacity for SSI after open VHR among patients at other academic medical centers with a large volume of operations. The original development cohort for the score was a VA population of largely older, male, high-risk patients. The current three patient cohorts were heterogeneous, with more female and younger patients, and diverse populations. In all four patient cohorts, compared with the VHWG and CDC classifications, the VHRS had greater predictive accuracy. The five factors in the test—incision class, morbid obesity, concomitant procedures, ASA class, and elevation of skin flaps—all have strong biologic plausibility to be associated with SSI and in other studies have been associated with incisional complications.

Incision class and extent of contamination are strong predictors of incisional complications, as has been proved repeatedly. However, with ventral hernias, incision class fails to tell the entire story. For example, class I patients have a much higher rate of incisional complications than other patients in other classes [8]. This may be secondary to previous infections leading to latent bacteria, impaired blood flow as a result of multiple prior incisions, high-risk patient type [9], long surgical times, or implantation of foreign material. In addition, concomitant procedures through the same incision increase the rate of SSI, in particular mesh infection [8].

Morbid obesity and elevated ASA class are widely considered risk factors for complications of surgery [10 –12]. Thus, ASA class ≥3 was an independent risk factor for SSI. This measure is useful to the VHRS scoring system, as it takes into account several other risk factors for SSI, as well as their severity. For example, in our cohort, smoking and diabetes mellitus were both associated with SSI development on univariable analysis. However, they were not independent variables in the VHRS model [3,13 –15]. Including variables such as diabetes mellitus, COPD, or smoking fails to take into account disease severity. For example, in this study, a patient with well-controlled diabetes mellitus with a glycosylated hemoglobin concentration of 6.0 g/dL would be classified the same as a patient with poorly controlled diabetes with a glycosylated hemoglobin concentration of 12.0 g/dL. These diseases are accounted for by ASA along with the severity of the co-morbidity.

We also found that both raising skin flaps and fascial release increased the risk of SSI. Both of these findings may be secondary to the increase in dead space (with or without seroma or hematoma formation), which would allow for growth of bacteria [16]. Raising skin flaps also may increase the risk of devascularizing tissue, causing necrosis. Several newer techniques, including laparoscopic component separation, perforator-sparing skin flaps, and transverus abdominis release avoid development of skin flaps or devascularization of the skin flaps and may decrease the rate of SSI [16,17]. It was common practice for surgeons to utilize drains when developing skin flaps; in addition, drains were applied routinely for implantation of biologic mesh. Drains were not an independent factor in the development of SSI in our prior study.

Our data demonstrated poorer outcomes for those who had an SSI, including higher rates of recurrence and re-operation. As several components of the VHRS are modifiable patient factors, our scoring system may help identify patients who could benefit from pre-operative interventions. In high-risk patients, it also may be possible to modify the surgical technique to lower the risk of infection such as by performing an interval hernia repair or avoiding skin flaps [5,18,19]. A situation where interval repair may be warranted is during bariatric surgery. If a ventral hernia is identified but does not prevent safe completion of the primary operation (bariatric surgery), repairing the hernia at a later date when there is no contamination and the patient has lost substantial weight may be beneficial [5,18,19].

Several factors limit this study. First, all participants were enrolled in academic institutions, and VHR was performed almost exclusively by specialists. These two factors may limit the generalizability of our findings. Second, a number of variables such as incision class or SSI can be variable in interpretation. Although we used the CDC definition of SSI, certain components allow subjective interpretation such as having “at least one of the following signs or symptoms: pain, tenderness, localized swelling, redness, or heat” or diagnosis “by the surgeon or attending physician or other designee.” Third, although this was a prospectively developed database, not all pertinent data points were captured. For example, there was no measure of surgeon skill or relevant patient laboratory data such as serum albumin or glycosylated hemoglobin concentrations. Finally, in identifying SSI, our follow-up was limited to 30 d. Additional SSIs, in particular, deep incisional SSIs or mesh infections, could occur after 30 d. The CDC definition of SSIs includes all implant infections up to 1 y.

Conclusions

We present a prospectively validated score for patients undergoing VHR for SSI. The accuracy of the VHRS for SSI exceeds that of our data published previously, as well as existing risk-assessment tools. The VHRS can be used to identify patients at higher risk for SSI, who might benefit from interval ventral hernia repair, optimization of co-morbidities, patient counseling as to outcome estimates, and modification of surgical techniques. Further study is needed to determine the generalizability of the VHRS to other patient populations (such as non-academic hospitals) and the utility of the VHRS to identify factors that would improve outcomes (such as pre-operative risk reduction programs).

Author Disclosure Statement

Dr. Liang: This work was supported by the Center for Clinical and Translational Sciences, which is funded by National Institutes of Health Clinical and Translational Award UL1 TR000371 and KL2 TR000370 from the National Center for Advancing Translational Sciences. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Center for Research Resources or the National Institutes of Health.

Dr. Roth: Bard, Lifecell, MTF consultant.

Dr. Martindale: Lifecell consultant, Covidien consultant.

References

Poulose

, Shelton

, Phillips

, et al. Epidemiology and cost of ventral hernia repair: Making the case for hernia research. Hernia, 2012; 16:179–183.

Reynolds

, Davenport

, Korosec

, Roth

. Financial implications of ventral hernia repair: A hospital cost analysis. J Gastrointest Surg, 2013; 17:159–167.

Breuing

, Butler

, Ferzoco

, et al.; Ventral Hernia Working Group. Incisional ventral hernias: Review of the literature and recommendations regarding the grading and technique of repair. Surgery, 2010; 148:544–558.

Kanters

, Kirpata

, Blatnik

, et al. Modified hernia grading scale to stratify surgical site occurrence after open ventral hernia repairs. J Am Coll Surg, 2012; 215:787–793.

Berger

, Li

, Hicks

, et al. Development and validation of a risk-stratification score for surgical site occurrence and surgical site infection after open ventral hernia repair. J Am Coll Surg, 2013; 217:974–982.

Alexander

, Solomkin

, Edwards

. Updated recommendations for control of surgical site infections. Ann Surg, 2011; 253:1082–1093.

DeLong

, DeLong

, Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 1988; 44:837–845.

Houck

, Rypins

, Sarfeh

, et al. Repair of incisional hernia. Surg Gynecol Obstet, 1989; 169:397–399.

Lau

, Kim

, Haigh

, Teirian

. Obesity increases the odds of acquiring and incarcerating noninguinal abdominal wall hernias. Am Surg, 2012; 78:1118–1121.

10.

Manilich

, Vogel

, Kiran

, et al. Key factors associated with postoperative complications in patients undergoing colorectal surgery. Dis Colon Rectum, 2013; 56:64–71.

11.

Olsen

, Higham-Kessler

, Kokoe

, et al. Developing a risk stratification model for surgical site infection after abdominal hysterectomy. Infect Control Hosp Epidemiol, 2009; 30:1077–1083.

12.

Wick

, Hirose

, Shore

, et al. Surgical site infections and cost in obese patients undergoing colorectal surgery. Arch Surg, 2011; 146:1068–1072.

13.

Gaynes

, Culver

, Horan

, et al. Surgical site infection (SSI) rates in the United States, 1992–1998: The National Nosocomial Infections Surveillance System Basic SSI Risk Index. Clin Infect Dis, 2001; 33(Suppl 2):S69–S77.

14.

Farrow

, Awad

, Berger

, et al. More than 150 consecutive open umbilical hernia repairs in a major Veterans Administration Medical Center. Am J Surg, 2008; 196:647–651.

15.

Tang

, Chen

, Wang

, et al. Risk factors for surgical site infection after elective resection of the colon and rectum: A single-center prospective study of 2,809 consecutive patients. Ann Surg, 2001; 234:181–189.

16.

Chang

, Mathes

. Comparison of the effect of bacterial inoculation in musculocutaneous and random-pattern flaps. Plast Reconstr Surg, 1982; 70:1–10.

17.

Pauli

, Rosen

. Open ventral hernia repair with component separation. Surg Clin North Am, 2013; 93:1111–1133.

18.

Liang

, Kao

. Reply to PMID 24051068. J Am Coll Surg., 2014; 218:1076–1077.

19.

Brahmbatt

, Carter

, Hicks

, et al. Identifying risk factors for surgical site complications after laparoscopic ventral hernia repair: Evaluation of the Ventral Hernia Working Group grading system. Surg Infect, 2014; 15:187–193.