Modernizing Surgical Quality: A Novel Approach to Improving Detection of Surgical Site Infections in the Veteran Population

Abstract

Introduction:

Surgical site infections (SSIs) are an important quality measure. Identifying SSIs often relies upon a time-intensive manual review of a sample of common surgical cases. In this study, we sought to develop a predictive model for SSI identification using antibiotic pharmacy data extracted from the electronic medical record (EMR).

Methods:

A retrospective analysis was performed on all surgeries at a Veteran Affair’s Medical Center between January 9, 2020 and January 9, 2022. Patients receiving outpatient antibiotics within 30 days of their surgery were identified, and chart review was performed to detect instances of SSI as defined by VA Surgery Quality Improvement Program criteria. Binomial logistic regression was used to select variables to include in the model, which was trained using k-fold cross validation.

Results:

Of the 8,253 surgeries performed during the study period, patients in 793 (9.6%) cases were prescribed outpatient antibiotics within 30 days of their procedure; SSI was diagnosed in 128 (1.6%) patients. Logistic regression identified time from surgery to antibiotic prescription, ordering location of the prescription, length of prescription, type of antibiotic, and operating service as important variables to include in the model. On testing, the final model demonstrated good predictive value with c-statistic of 0.81 (confidence interval: 0.71–0.90). Hosmer–Lemeshow testing demonstrated good fit of the model with p value of 0.97.

Conclusion:

We propose a model that uses readily attainable data from the EMR to identify SSI occurrences. In conjunction with local case-by-case reporting, this tool can improve the accuracy and efficiency of SSI identification.

Introduction

Surgical site infections (SSIs) occur in 0.5%–3% of patients undergoing surgery, add an average increased hospital length of stay by 7–11 days, and account for $3.5 billion to $10 billion in added US healthcare costs annually.^1–4 As such, an institution’s SSI rate is an important surgical quality metric used for benchmarking performance for accreditation and reimbursement.⁵ Surveillance and reporting of SSIs is an important function of surgical quality programs, and many institutions use dedicated staff to find and review cases of SSI. However, the manual curation of this data is time consuming and costly; the Agency for Healthcare Research and Quality estimates that 33 hours of chart review is needed to identify one incidence of SSI.⁶

The implementation of electronic medical records (EMRs) has allowed for automated data abstraction and utilization of computer algorithms in identifying SSIs. Much of the pre-existing research has focused on models trained on administrative claims data, namely International Classification of Diseases (ICD) codes, to capture instances of SSI.^7–8 Identifying SSIs through ICD coding can be problematic, as it relies on accurate coding and documentation of a patient encounter and can be biased by its primary goal of hospital reimbursement. While modeling claims data can provide an accurate reflection of overall SSI rate and trends, such models are inaccurate when identifying complications in individual patients.^9–10

Since chart review and claims data both rely on documentation, we sought a novel method to identify SSIs. We hypothesized that outpatient antibiotic prescription data from an EMR can be leveraged by a machine learning algorithm to accurately estimate the occurrence of an SSI.

Methods

Following institutional review board approval, we performed a retrospective analysis on all patients who received an outpatient prescription for an oral antibiotic within 30 days of a surgical procedure done at a U.S. Veteran Affairs Medical Center between September 2020 and September 2022. We excluded patients who did not receive antibiotics within 30 days of surgery and those who had undergone an endoscopic (such as a colonoscopy, esophagogastroduodenoscopy, hysteroscopy, and cystoscopy), dental, or an isolated anesthesia pain procedure. For patients meeting inclusion criteria, data were abstracted from the Corporate Data Warehouse including demographics, surgery date, operating service, wound classification, patient demographics, type of antibiotic, reason for treatment, date and length of prescription, and the type of encounter from which the prescription was ordered (hospital, clinic, emergency department, etc.).

The primary aim of the study was to develop a machine learning model to accurately identify the occurrence of an SSI using variables that can be readily extracted from surgery and antibiotic prescription data residing in the EMR without manual chart review. Identifying patient-level risk factors for SSI was beyond the scope of this study. Chart review was conducted on the included patients to determine the incidence and type of SSI as defined by the VA Surgical Quality and Improvement Program (VASQIP) wound definitions (see Supplementary Data S1). Logistic regression was performed to identify suitable variables for inclusion in the model, and we included variables with p values < 0.2. The regression model was trained with k-fold cross validation with k = 10 slices on a random 80% subset of the data and then tested on the remaining 20%. The c-statistic, or the area under the curve of the receiver operating characteristic (ROC) curve was calculated, as was the probability threshold for optimum sensitivity and specificity of our model. Hosmer–Lemeshow testing was used to determine goodness of fit. Missing values were not replaced during analysis. All statistics and machine learning computations were performed using R Studio with the caret and pROC packages (Version 3.5.3, Posit, Boston, MA).^11–12 The Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis guidelines were followed for standard reporting of this study. The completed checklist for the article is available in Supplementary Table S1.

As a secondary aim, we tested our assumption that antibiotic data would capture most of our institution’s SSIs. We compared with the collected antibiotic data to SSI reporting data collected by our institution’s quality and peer review teams to identify possible areas of deficiency in our model and opportunities for further quality improvement. The surgical quality team at our institution regularly monitors readmissions after surgery and encourages self-reporting of SSIs by providers.

Results

There were 8253 eligible cases over the study period for which 793 (9.6%) involved an outpatient antibiotic prescription within 30 days. Chart review confirmed SSI in 128 (1.6%) cases, of which 101 (78.9%) were superficial, 15 (11.7%) were deep incisional, and 12 (9.4%) were organ space SSIs. An additional 97 (1.2%) cases receiving antibiotics were because of a surgical wound concern but did not meet the VASQIP defined SSI criteria (Fig. 1). The patient cohort receiving antibiotics within 30 days of surgery had median age of 65 (interquartile range 52–72), was 90% male, and 61% white. The majority of the involved procedures were inpatient and done by general, orthopedic, urologic, or podiatric surgery teams (Table 1).

FIG. 1.

Study consort diagram.

Table 1.

Cases Receiving Antibiotics Within 30 Days of Surgery

Total cases	793
Sex
Female	83 (10%)
Male	710 (90%)
Race
Asian	38 (5%)
African American	132 (17%)
Caucasian	486 (61%)
Hispanic	103 (13%)
Other	17 (2%)
Unknown	17 (2%)
Median Age (years) (IQR)	65 (52–72)
Outpatient surgery	332 (42%)
Operating service
Cardiac	28 (4%)
Otolaryngology	49 (6%)
General	141 (18%)
Gynecology	17 (2%)
Neurosurgery	18 (2%)
Ophthalmology	56 (7%)
Orthopedics	93 (12%)
Orthopedics hand	47 (6%)
Plastic	54 (7%)
Podiatry	118 (15%)
Thoracic	8 (1%)
Urology	111 (14%)
Vascular	53 (7%)
Surgery Wound Class
Clean	360 (45%)
Clean contaminated	187 (24%)
Contaminated	92 (12%)
Dirty/Infected	152 (19%)
Median days to Rx (IQR)	9.5 (4.7–16.4)
Median Rx length (days) (IQR)	7 (6–14)
Location of Rx Order
Hospital discharge	392 (49%)
Emergency Department	91 (12%)
Surgery Clinic	211 (27%)
Primary Care Clinic	30 (4%)
Other clinic	69 (9%)
Type of antibiotic
Amoxicillin/clavulanic acid	169 (21%)
Trimethoprim/sulfamethoxazole	105 (13%)
Cephalexin	116 (15%)
Ciprofloxacin	55 (7%)
Clindamycin	31 (4%)
Doxycycline	148 (19%)
Levofloxacin	38 (5%)
Amoxicillin	32 (4%)
Other	99 (12%)
Diagnosis for antibiotic
Surgical site infection	128 (16%)
Other surgical wound concern	97 (12%)
Other medical diagnosis	568 (72%)

IQR, interquartile range; Rx, antibiotic prescription; Abx, antibiotic.

Using multivariable binomial logistic regression, six variables were selected for inclusion in our model: the time from surgery to the time of antibiotic prescription, the length and clinical setting of the prescription, the type of antibiotic, the wound class of the procedure, and the operating service. Increased time from surgery, a prescription from the operating service’s clinic, clean wound class, and the general surgery and orthopedics hand services were most predictive of the antibiotic being prescribed for an SSI. The final model after training and testing had a c-statistic of 0.81 (confidence interval: 0.71–0.90) (Table 2). A probability threshold of 11.9% yielded the optimum combination of sensitivity and specificity of 94.4% and 53.6%, respectively (Fig. 2), as well as a negative and positive predictive value of 98.7% and 20.7%, respectively. Hosmer–Lemeshow testing demonstrated good fit of the model with p value of 0.974.

FIG. 2.

Receiver operating characteristic curve for the model; ROC, receiver operating characteristic; SSI, surgical site infection. Point illustrated indicates probability threshold yielding the optimal combination of specificity and sensitivity.

Table 2.

Surgical Site Infection Model Final Parameters

	Coefficient	Standard error	z value	p	95% confidenceinterval
Intercept	−1.628	0.612	−2.660	0.008	−2.883	−0.468
Time to Rx	0.070	0.017	4.124	<0.001	0.037	0.104
Rx length	−0.012	0.011	−1.123	0.262	−0.037	0.007
Location of Rx
Hospital Discharge	—	—	—	—	—	—
ED	0.539	0.421	1.279	0.201	−0.297	1.360
Surgery Clinic	1.109	0.345	3.211	0.001	0.439	1.796
PCP Clinic	−16.910	1223.138	−0.014	0.989	—	51.588
Other Clinic	−1.516	0.704	−2.152	0.031	−3.094	−0.262
Type of Abx
Other Abx	—	—	—	—	—	—
Amoxicillin/clavulanic acid	0.605	0.503	1.203	0.229	−0.338	1.656
Trimethoprim/sulfamethoxazole	0.635	0.585	1.084	0.278	−0.490	1.825
Cephalexin	0.108	0.549	0.197	0.844	−0.940	1.233
Ciprofloxacin	0.092	0.701	0.132	0.895	−1.329	1.459
Clindamycin	0.436	0.711	0.613	0.540	−0.965	1.845
Doxycycline	0.386	0.539	0.715	0.474	−0.636	1.497
Levofloxacin	−0.501	0.882	−0.568	0.570	−2.517	1.100
Amoxicillin	−16.574	1164.796	−0.014	0.989	−497.478	−443.243
Operating Service
General	—	—	—	—	—	—
Podiatry	−1.419	0.486	−2.921	0.003	−2.398	−0.487
Urology	−1.752	0.504	−3.475	0.001	−2.789	−0.798
Orthopedic	−0.637	0.461	−1.381	0.167	−1.556	0.256
Ophthalmology	−3.786	1.089	−3.477	0.001	−6.739	−2.039
Plastic	−0.636	0.523	−1.216	0.224	−1.699	0.366
Vascular	−0.730	0.520	−1.404	0.160	−1.795	0.258
Otolaryngology	−1.683	0.650	−2.589	0.010	−3.083	−0.490
Orthopedic hand	0.205	0.516	0.398	0.691	−0.815	1.215
Other	−1.467	0.520	−2.823	0.005	−2.549	−0.492
Wound Class
Clean	—	—	—	—	—	—
Clean contaminated	−0.592	0.354	−1.674	0.094	−1.302	0.090
Contaminated	−1.446	0.528	−2.741	0.006	−2.564	−0.470
Dirty/Infected	−0.347	0.388	−0.893	0.372	−1.122	0.406

Rx, antibiotic prescription; ED, Emergency Department; PCP, primary care provider, Abx, antibiotic.

The combination of SSIs identified by antibiotic prescription review and surgical quality reports represented all known cases of SSI at our institution during the study period. There were 146 occurrences of SSIs, giving an overall incidence of 1.8%. Antibiotic data outperformed quality reporting along, capturing 88% of SSIs compared with to 33% (Table 3a). While superior to quality reporting in detecting superficial (95% vs. 19%) and organ space infections (71% vs. 41%), it was inferior in the detection of deep incisional infections (65% vs. 91%). There were 18 (12%) cases of SSI that were not captured by antibiotic data. Most of the missed cases were not detected because of the prescription of intravenous (IV) antibiotics, the prescription occurring outside of 30 days post procedure, treatment at an outside hospital, or treatment completed entirely in an inpatient setting prior to discharge (i.e., no outpatient antibiotic prescription) (Table 3b).

Table 3.

Performance of Outpatient Antibiotic Prescription Data in Detecting Surgical Site Infections

Table 3a: Comparison of antibiotic and quality reporting data
	All SSIs^a	Superficial	Deep incisional	Organ space
Total	146	106	23	17
30-day Abx data	128 (88%)	101 (95%)	15 (65%)	12 (71%)
QI reporting data	48 (33%)	20 (19%)	21 (91%)	7 (41%)

Table 3b: Analysis of cases missed by antibiotic data^b
	All SSIs	Superficial	Deep incisional	Organ space
All missed SSI	18	5	8	5
Treated at OSH	3 (17%)	1	1	1
Inpatient only	2 (11%)	0	2	0
IV Abx	6 (33%)	0	3	3
No Abx	1 (6%)	1	0	0
After 30 days	6 (33%)	3	2	1

Unique cases of SSIs identified on review of antibiotic data and quality team reports, representing all known SSIs during the study period.

Eighteen cases of SSIs were captured by the quality team but were not captured by the 30-day antibiotic data.

SSI, surgical site infection; Abx, antibiotic; QI, quality improvement; IV, intravenous.

Discussion

We hypothesized that outpatient antibiotic data after surgery can be leveraged in a machine learning algorithm to retrospectively identify incidences of SSIs. We demonstrated that quality reporting alone is insufficient in capturing a true estimate of SSI rate and that the addition of antibiotic data analysis improves SSI identification. Furthermore, our model accurately predicts SSIs derived from antibiotic data with high sensitivity and moderate specificity.

Previous studies on SSI models using structural data have relied on administrative claims data, which is reliant on documentation and accurate reporting. We have provided a novel and alternative approach using operative and antibiotic prescription data, which can be automatically extracted from the EMR and is objective and not documentation based. It provides a highly sensitive way to screen patients for SSIs and can decrease the burden of chart review. However, analyzing antibiotic data alone is not sufficient in capturing all cases, particularly deep incisional and organ space infections. These types of SSI are likely to merit readmission, reoperation, and/or prolonged hospital length of stay and should trigger review by a dedicated surgical quality team.

Our study does have limitations. The demographics of the veteran surgical population differ from the general population, particularly in that it is predominately male. This may limit the study’s generalizability to other patient groups. The VA medical system provides longitudinal and comprehensive care for veterans; such care may not be provided at other types of institutions where postoperative patients may present for care of suspected SSI to a hospital system different from where they underwent their index operation. In addition, other systems may exhibit different prescribing practices in terms of types of antibiotics prescribed and locations in which patients are evaluated. Even within the VA health system, differences in local station characteristics, such as rurality, may limit follow-up. Thus, our results from an urban VA medical center may not be generalizable to other centers. The outpatient antibiotic data did not account for IV antibiotics or antibiotics prescribed outside of the 30-day time window which contributes to missing cases, particularly deep incisional and organ space infections.

Using antibiotic prescription data in conjunction with a surgical quality team and self-reporting gives a more accurate picture of an institution’s SSI Rate. Our model of prescription data offers a more efficient way to screen patients by decreasing the burden of manual chart review by quality and safety officers. Increased specificity in similar models can be improved by incorporating natural language processing of provider notes, which has been shown to increase accuracy of SSI models and will be a target for future research.^7,13,14

Conclusion

Postoperative antibiotic data can be leveraged in machine-learning algorithms to accurately predict occurrences of SSIs without manual chart review. The addition of such algorithms to pre-existing quality reporting systems can greatly enhance the identification of SSIs and support the development of quality and safety initiatives.

Footnotes

Acknowledgments

This material is the result of work supported with resources and the use of facilities at the Jennifer Moreno Department of Veterans Affairs Medical Center in San Diego, California.

The contents do not represent the views of the U.S. Department of Veterans Affairs or the United States Government.

Authors’ Contribution

L.P.: Conceptualization, methodology, software, formal analysis, data curation, writing—original draft, and writing—review and editing; T.O.: methodology, formal analysis, and writing—review and editing; W.A.: conceptualization, methodology, and writing—review and editing; B.P.: conceptualization, methodology, writing—review and editing, supervision.

Author Disclosure Statement

The authors have no conflicts of interest and no funding source to declare.

Funding Information

None.

Supplementary Material

References

Seidelman

, Mantyh

, Anderson

. Surgical site infection prevention: A review. JAMA, 2023; 329(3):244–252; doi: 10.1001/jama.2022.24075

Zimlichman

, Henderson

, Tamir

, et al. Health care-associated infections: A meta-analysis of costs and financial impact on the US health care system. JAMA Intern Med, 2013; 173(22):2039–2046; doi: 10.1001/jamainternmed.2013.9763

O’Hara

, Thom

, Preas

. Update to the centers for disease control and prevention and the healthcare infection control practices advisory committee guideline for the prevention of surgical site infection (2017): A summary, review, and strategies for implementation. Am J Infect Control, 2018; 46(6):602–609; doi: 10.1016/j.ajic.2018.01.018

Badia

, Casey

, Petrosillo

, et al. Impact of surgical site infection on healthcare costs and patient outcomes: A systematic review in six European countries. J Hosp Infect, 2017; 96(1):1–15; doi: 10.1016/j.jhin.2017.03.004

Joint Commission. “The Joint Commission’s implementation guide for NPSG. 07.05. 01 on surgical site infections: the SSI change project”. Illinois: The Joint Commission (2013).

Price

, Savitz

. “Improving the measurement of surgical site infection risk stratification/outcome detection.” Final report (Prepared by Denver Health and its partners under Contract No. 290 2006 00 20). AHRQ Publication, 12(2012): p. 46.

, Khair

, Yang

, et al. Performance of machine learning algorithms for surgical site infection case detection and prediction: A systematic review and meta-analysis. Ann Med Surg (Lond), 2022; 84:104956; doi: 10.1016/j.amsu.2022.104956

Petrosyan

, Thavorn

, Smith

, et al. Predicting postoperative surgical site infection with administrative data: A random forests algorithm. BMC Med Res Methodol, 2021; 21(1):179; doi: 10.1186/s12874-021-01369-9

Sherman

, Heydon

, St John

, et al. Administrative data fail to accurately identify cases of healthcare-associated infection. Infect Control Hosp Epidemiol, 2006; 27(4):332–337; doi: 10.1086/502684

10.

Kulaylat

, Engbrecht

, Rocourt

, et al. Measuring surgical site infections in children: Comparing clinical, electronic, and administrative data. J Am Coll Surg, 2016; 222(5):823–830; doi: 10.1016/j.jamcollsurg.2016.01.004

11.

Kuhn

. Building predictive models in R using the caret package. J Stat Soft, 2008; 28(5):1–26; doi: 10.18637/jss.v028.i05

12.

Robin

, Turck

, Hainard

, et al. pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 2011; 12(1):77–78; doi: 10.1186/1471-2105-12-77

13.

Thirukumaran

, Zaman

, Rubery

, et al. Natural language processing for the identification of surgical site infections in orthopaedics. J Bone Joint Surg Am, 2019; 101(24):2167–2174; doi: 10.2106/JBJS.19.00661

14.

da Silva

, Ten Caten

, Dos Santos

, et al. Predicting the occurrence of surgical site infections using text mining and machine learning. PLoS One, 2019; 14(12):e0226272; doi: 10.1371/journal.pone.0226272

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.03 MB

0.06 MB

0.00 MB