Revisiting the Surgical Infection Society Delphi Analysis of the Research Agenda in Surgical Infections: Advancements and Unanswered Questions

Abstract

Background:

In 2006, the Surgical Infection Society (SIS) used a modified Delphi process to enlist SIS member-experts to identify 15 research priorities in the field of surgical infectious diseases; it was intended to serve as a research road map for the next one to two decades. We sought to evaluate the progress made in each of these priority areas.

Hypothesis:

We examined the progress achieved with respect to the 15 research areas identified by the Delphi process at that time, hypothesizing that advances in knowledge would be achieved in most domains, if not all.

Methods:

Surgical Infection Society members were surveyed to determine whether each priority area question had been satisfactorily answered in the last 14 years; to assess the quality of evidence answering each question (1–3 scale); and to delineate whether there is a current unmet need for continued research in each area. Randomized controlled trials (RCTs) regarding these initiatives were also identified via literature search and their citations in the literature were tabulated.

Results:

Sixty-six members of the SIS responded to the survey. Thirteen of 15 research priority areas saw an increase in data perceived to be available as adjudged by experts, as well an increase in the number of RCTs addressing that topic. However, there were only six questions that were deemed by experts to be answered sufficiently, primarily regarding antibiotic duration for certain conditions and the impact of glycemic control on infection. The questions that remained unanswered related to nosocomial infections, sepsis/septic shock, prevention of SSI, and antimicrobial pharmacokinetics. For a majority of the questions that experts believed were not answered sufficiently (8/9), respondents opined that continued research into these areas was warranted.

Conclusion:

Whereas 40% (6/15) of the research questions prioritized by the SIS in 2006 were answered to the satisfaction of member-experts, there are many questions that remain unanswered despite an increase of available data. Revisiting these research priorities highlights advancements made in the field of surgical infections, but also helps identify the areas that would benefit from continued study. That a majority of questions remain unanswered underscores an opportunity for member-experts to collaborate on SIS-managed or -endorsed RCTs.

Hospital-acquired infections (HAI) are a major source of morbidity, increased hospital cost, and mortality in the United States [1]. As the number of surgical procedures continues to increase, surgical site infections (SSI) remain a persistent cause of HAI, however, as many as 50% may be preventable [2]. Targeted, evidence-based interventions have the potential to reduce surgical infections, hospital cost, and improve outcomes as has been documented in colorectal surgery [3].

However, surgical patients are uniquely challenging to study because of inherent hetergeogeneity [4], variability of care processes among individual institutions, certain populations being likely to be lost to follow-up, and to the pervasive challenge of designing appropriate controls. This is compounded by the expense and complexity of well-designed clinical trials, to which the reticence of potential subjects to enroll in such trials is a contributor [5]. To help guide the application of limited resources to the most compelling research questions, Nathens et al. [6] solicited experts in the field of surgical infections to identify the most important research questions using the Delphi approach to achieve consensus.

Nathens et al. [6] identified 15 research questions related to a variety of topics (e.g., sepsis, SSI, nosocomial infections, pancreatitis, drug pharmacokinetics), identified those randomized controlled trials (RCTs) related to the topics of interest, and examined the perceptions of experts in the field at the time. The average cost of an SSI was estimated to be approximately $3,000 with an annual cost of $1.5 billion [7]. More modern estimates place the annual cost of HAI at closer to $10 billion with SSI comprising the largest portion at 33.7% [8]. By publishing a consensus assessment, Nathens et al. [6] hoped to focus research resources to the areas of study that experts deemed to be of greatest need. Although the estimated incidence of surgical infections may have increased over time (1.2% in 1999 vs. 2% in 2013), what progress has been made in the last 15 years [7,8]? Studies are few and may be difficult to compare [7]. With this in mind, we re-examined these questions in a contemporary context by assembling another group of experts (with little overlap), and soliciting opinions on the progress made, the quality and relevance of research performed, and the current importance of each research question 14 years later.

Methods

This survey of the SIS membership was designed by the Scientific Studies Committee of the Surgical Infection Society (SSC-SIS) as part of a research initiative revisiting the work of Nathens et al. [6], begun in 2004 and published in 2006. Web-based survey design and study data collection was hosted by Google Forms (www.google.com/forms; Alphabet, Inc., Mountain View, CA) from May 1, 2018 through March 31, 2019. A link to the survey was distributed to all members of the SIS via e-mail, with periodic e-mail reminders to encourage participation. The survey had two goals: to determine the long-term perception of the success of the 2006 research agenda and to re-define the research agenda in surgical infections in 2020 and beyond.

The survey instrument collected identifiers of all respondents to ensure there were no duplicate respondents, and so that participants could be contacted for follow-up if necessary regarding the second goal of the survey (the new Delphi consensus to generate the updated research agenda). Anonymity among participants was maintained. Demographic information for each participant was also solicited, including practice region, academic affiliation, years of practice post-training, field of medicine including subspecialty, and leadership positions at his/her institution, SIS annual meeting attendance, and duration of SIS membership. Each participant was also asked whether they participated in the original SIS Delphi process [6].

The 15 research questions (Table 1) identified as important by expert consensus in 2006 were provided verbatim, followed by a series of questions for this analysis [6]. These questions determined whether or not the respondent believed that the research question had been answered, the importance of continued research on the topic (five-point Likert scale: unimportant to very important) and how they perceived the current quality of research on the question (three-point Likert scale: very little data to substantial data). Separately, we identified by literature search 134 RCTs responsive to the research questions, and determined how many times each had been cited in the peer-reviewed literature (Science Citation Index, Tompson Reuters, Toronto, ON).

Table 1.

Comparison of Perceived Importance, Perceived Data Availability, and Number of Randomized, Controlled Trials That Address or Partially Address the Fifteen Questions Identified in the Original Surgical Infection Societyt Delphi between 2019 and 2006

		2019 Version					2006 Version
Original rank	Question	Question answered(% yes)	% Ranking important/very important	Mean score	Data availability	Number of RCTs	% Ranking important/very important	Mean score	Data availability	Number of RCTs
1	Does strict glycemic control compared with standard care reduce the risk of surgical site infection in patients undergoing abdominal surgery?	58%	43%	3.25	2.44	18	86%	4.12	1.67	1
2	Does a fixed, short term of antimicrobial therapy (e.g., 5 d) in patients with intra-abdominal infections affect the infection risk of microbiologic or clinical cure compared with patients receiving therapy for a duration determined by clinical parameters (resolution of temperature and WBC)?	66%	62%	3.71	2.54	6	60%	3.79	1.7	1
3	Is there any difference in antimicrobial resistance, ICU length of stay, and mortality in patients with suspected infection nosocomial infections who receive early, empiric antimicrobial therapy compared with an approach in which antimicrobial is withheld until culture data are available?	38%	75%	4.05	2.05	1	56%	3.77	1.91	0
4	Will control of blood glucose in the high normal range (100–150 mg/dL) in infected surgical ICU patients achieve equivalent reduction in mortality compared with intensive insulin therapy to maintain blood glucose between 80 and 110 mg/dL?	70%	37%	2.93	2.65	12	63%	3.72	1.51	0
5	Is there any survival benefit to raising blood pressure pharmacologically in adult patients with septic shock with inotropes/vasopressors compared with volume loading and tolerance of lower MAP?	34%	64%	3.76	2.09	0	53%	3.53	1.6	0
6	Does avoidance of systemic hypothermia and increased arterial oxygenation in the early post-operative period reduce the risk of surgical site infections compared with standard care?	53%	58%	3.47	2.23	15	47%	3.51	1.98	4
7	Does resuscitation with hypertonic saline reduce the incidence of organ dysfunction, nosocomial infection, or mortality in patients with septic or traumatic shock?	39%	37%	2.95	1.97	17	49%	3.51	1.74	9
8	Does routine antimicrobial prophylaxis with a 2- to 4-wk course of a carbapenem compared with placebo affect the risk of infected necrotizing pancreatitis, need for operative intervention, antimicrobial resistance, and mortality among those admitted with severe necrotizing pancreatitis?	52%	36%	2.71	2.21	9	49%	3.49	1.95	6
9	Is there a difference in the rate of surgical site infections, non-union, cost, and antimicrobial resistance in patients with open fractures treated with 24 vs. 72 h of antimicrobial prophylaxis? Separate strata for heavily contaminated wounds, where 72 h would be compared with 7 d.	28%	61%	3.67	1.98	6	56%	3.47	1.7	1
10	Does antioxidant therapy improve mortality or reduce subsequent organ dysfunction compared with placebo in patients with sepsis?	13%	54%	3.55	1.66	27	47%	3.35	1.72	6
11	Does an increase in antimicrobial dose or frequency reduce the incidence of surgical site infections or cure rates (for pharmacokinetics established infections) in the obese patient?	22%	72%	3.89	1.69	5	47%	3.35	1.47	1
12	Is there any difference in ventilator days, ICU LOS, costs, antimicrobial resistance, mortality, and clinical or associated microbiologic cure rates in patients with VAP receiving 7 vs. 14 d of antimicrobial therapy?	58%	51%	3.31	2.42	6	47%	3.33	1.98	3
13	Does rotation of empiric antimicrobial regimens in the ICU affect clinical cure rates or the emergence of resistance infection organisms compared with standard practice? If so, what is the optimal interval?	31%	56%	3.56	2.02	1	42%	3.3	1.95	0
14	Does routine fever suppression affect clinical or microbiologic cure rates in septic surgical patients?	22%	56%	3.44	1.79	5	47%	3.3	1.58	1
15	Is there any difference in costs, sensitivity, and specificity of BAL, mini-BAL, sputum sample, CXR, and chest CT as associated diagnostic tests in those with suspected VAP?	36%	54%	3.44	2.13	6	44%	3.3	2.3	2

WBC = white blood cell count; RCTs = randomized controlled trials; ICU = intensive care unit; MAP = mean arterial pressure; LOS = length of stay; VAP = ventilator-associated pneumonia; BAL = bronchoalveolar lavage; CXR = chest radiograph; CT = computed tomography.

Experts were asked to not review the literature prior to responding so that perceptions could be compared to existing published data. Published studies were identified using a focused PubMed search of articles published between 2004 and August 2019 related to the identified research questions. The search was limited to RCTs. Any RCTs identified by Nathens et al. [6] were also included. Each question on the survey was considered and reported individually (i.e., the methodology was reiterated 14 times to address each of the 15 questions). Parameters were reported as either means with standard deviations (SD) or counts with percent, for continuous and categorical parameters, respectively. Linear relations between parameters were assessed using Pearson's correlation coefficient. All statistical analysis was performed in R 3.5.1 (www.r-project.org; Vienna, Austria, The R Foundation for Statistical Computing).

Results

Sixty-six survey respondents were included in the final analysis; two responses were excluded because they provided only demographic information. The majority of respondents were located within the United States (83%; n = 53), identified as academic clinical faculty (75%; n = 48), and identified their field as either general surgery (27%; n = 17) or acute care surgery (63%; n = 40). In addition, 29 respondents (45%) reported having held a leadership position within their organization during his or her career, and 26 respondents (41%) reported having more than 15 years of post-training experience. Although a majority of individuals did not participate in the original survey of Nathens et al. [6], there were nine individuals who did (14%).

The list of the top 15 questions identified in the original SIS Delphi study (in rank order) is summarized in Table 1, along with a summary of responses about the perceived importance of continued research in the topic, the perceived data availability, and the number of RCTs that either completely or partially addressed the research question. Of the 134 RCTs that were identified as being related to these research questions (Supplemental Table S1), 13% listed a author/contributor who is or was a member of SIS previously. Collectively, these 134 RCTs had been cited by other publications 30,504 times (mean 231 ± 610, median 82.5 [interquartile range, 3–203]). The mean impact factor (IF) for the journals at the time of publication was 10 ± 14, although an IF was not available for 26 of the publications (19%).

With the exception of questions 10 and 15, every question was perceived to have new available data between 2004 and 2019 (Fig. 1). Whereas the number of RCTs addressing each of these topics has increased since 2004, there was, perhaps surprisingly, no correlation between the number of RCTs and the mean available data score (r² = 0.002, p = 0.882) reported by respondents (Fig. 2). There was also an increase in the percentage of respondents who reported a question being important/very important between 2004 and 2019, with the exception of questions 1, 4, 7, and 8 (Fig. 3).

FIG. 1.

Difference in perceived available data between 2006 and 2019.

FIG. 2.

Scatterplot of relation between perceived available data and the number of randomized controlled trials (RCTs) addressing the topic with linear trend line. Model statistics: 2006 study (p = 0.447; r² = 0.045), current study (p = 0.882; r² = 0.002).

FIG. 3.

Difference in percentage of respondents who believe that a question is important/very important between 2006 and 2019.

For only a few questions did a majority of respondents believe that they had been answered fully in the last 15 years (Table 1). In fact, only question 2 (regarding antibiotic duration in intra-abdominal infections) and question 4 (regarding strict glycemic control in the intensive care unit) had more than two-thirds of respondents who believed that the question had been answered.

Although there was also no correlation between the percentage of respondents who believed that a question had been answered and the number of new RCTs related to the topic (r² = 0.006, p = 0.775), there was a modest correlation between the mean number of citations for all RCTs related to the question and the perceived amount of available data (r² = 0.264, p = 0.035). However, the most notable correlation was between the perceived mean available data and the percentage of respondents who believed that a question had been answered (Fig. 4, r² = 0.948, p < 0.001).

FIG. 4.

Scatterplot of relation between perceived available data and the percentage of respondents who believe that a question has been sufficiently answered. Linear model statistics: p < 0.001, r² = 0.938.

Discussion

The Delphi method is predicated on the belief that forecasts (or decisions) made by a structured group of individuals are more accurate than those from unstructured groups [9]. Experts answer questionnaires in two or more rounds. After each round, a facilitator provides an anonymous summary of the experts' forecasts from the previous round as well as the reasons they provided for their judgments. Thus, experts are encouraged to revise their earlier answers in light of others' replies. During this process the range of answers should decrease; the group converges towards the correct answer. Key components include anonymity of the participants (maintained herein), structuring of information flow (design of the survey instruments), regular feedback (through multiple iterations), and the role of the facilitator (SSC-SIS). The Delphi method can be used most successfully in forecasting single scalar indicators, such as the discrete research questions again posed herein, but it must be realized that in science and technology forecasting, the degree of uncertainty is high. Thus exact, always-correct predictions are impossible, and error is to be expected. A particular weakness of the Delphi method is that future developments are not always predicted correctly by expert consensus. This shortcoming concerning the issue of ignorance is important. If panelists are misinformed about a topic, the use of the Delphi method may only add confidence to their ignorance [10].

This study re-examined the surgical infection research questions identified by Nathens et al. [6] by utilizing a largely new group of experts who were asked about the importance of each question, their perception of the current available research, and whether the question had been answered sufficiently over the ensuing period. Whereas nearly every question had an increase in both the perceived available data and the number of RCTs related to the topic, the experts believed a minority of questions had been answered sufficiently.

When these questions were originally identified, there was only one question that the experts believed had more than “some” data available (question 15). By contrast, nearly every question re-posed by this survey had an increase in the perceived available data. On the surface, this would seem to correlate well with the nearly four-fold increase in the number of RCTs overall, but when questions were considered individually there was no correlation between perceived available data and the number of RCTs related to the topic, which could imply that the experts were unsatisfied by the extant data. Nathens et al. [6] made a similar observation during the original Delphi process, and hypothesized that methodological limitations or conflicting results between trials prevented the data from answering the question at hand. Whereas this hypothesis may still hold true, it is also possible that experts may inform their perceptions by experience and expertise, or reports less rigorous than RCTs, rather than solely with data from RCTs.

Despite the increase in available data, only 40% (6/15) of questions had a majority of the experts believing that the question had been answered sufficiently. As one might expect, the number of experts who believed a question had been answered correlated with the perceived available data rather than the number of RCTs, further suggesting that it is not RCTs alone that influence decision making. By way of example, a well-designed meta-analysis, [11] or published guidelines from a trusted society [12] may have influenced experts to opine that the optimal duration of antibiotic therapy in ventilator-associated pneumonia has been determined (e.g., question 12). Alternatively, a single, widely promulgated trial, such as the Study to Optizime Peritoneal Infection Therappy (STOP-IT) trial of the duration of antibiotic therapy for complicated intra-andominal infection with achieved source control [13] may be sufficient to convince experts that the question has been answered (e.g, question 2). This is supported by the correlation we found between the mean number of citations among RCTs related to a topic and the perceived available data by experts.

A surprising aspect of the study was the reported percentage of experts who claimed to not know whether or not a particular question had been answered. For 10 of the 15 studies, at least 20% of the experts answered that they did not know whether or not the question had been answered. Whether that might be because of lack of awareness, dissatisfaction with extant data, or some other factor is unknown. The distribution of the original survey that solicited these research questions was targeted to experts in the field, whereas this particular survey was sent to the active SIS membership. This could have resulted in a less-experienced cohort of experts, and therefore more uncertainty. Conversely, a well-read cohort might also have responded with skepticsm to a question because of the contradictions that are inherent in multiple studies on a topic. Regardless of the cause, the percentage of participants did correlate inversely with the perceived available data score, suggesting that an increase in the amount of perceived available data reduced the uncertainty about whether or not a question had been answered (p = 0.003, R² = 0.457) (Supplemental Fig. S1). Perhaps this uncertainty could be addressed by improving (i.e., changing) the ways research information is disseminated by societies such as SIS.

Evidence-based medicine applies decision theory to clinical diagnosis [14]. The Delphi process incorporates elements of both. Expert diagnostic problem solving depends on the perceived difficulty of the question, knowledge of the strategic options available, and clinical/scientific knowledge (both experience and evidence). Final conclusions thus depend both on prior belief and strength of the evidence [15]. Conclusions reached by Bayesian heueristics and clinical intuition may conflict; it is the task of the expert to perceive, parse, and resolve if possible the conflict.

Limitations of this study include the issues inherent with survey-based design, including selection and reporting bias, and those of the Delphi process itself. It is also possible that the members who completed the survey were not truly representative of experts in the field, athough we believe that distribution to the active SIS membership allowed for appropriate sampling of expert opinion. Finally, some of the perceived available data trends may have been limited by the use of a three-point scale instead of the standard five-point Likert scale. Ultimately the three-point scale was used to allow comparisons with the original Delphi survey of Nathens et al. [6]. There seemed to be a bias among respondents toward reporting “some data” as opposed to “no data” or “substantial data.” It is possible that more nuanced opinions could have been elicited by using a more granular scale.

The last 15 years have brought substantial progress in surgical infection research, both as perceived and as measured by the number of published RCTs relative to the research questions that were identified 15 years ago by experts in surgical infections. That only 13.4% of identified RCTs had meaningful contributions from SIS members present or past (i.e., sufficient to merit co-authorship) underscores the opportunity for surgeon clinician-scientists to participate more widely and increase their reach and influence in the field, which could be facilitated by the SSC-SIS or the SIS Foundation for Research and Education. This study also demonstrated a possible gap between perceived available data compared with the amount of available data in the form of RCTs, which warrants further investigation. Novel methods of communication, such as utilization of social media platforms, are showing promise for increasing the readership of important research [16]. Perhaps our traditional approaches to data dissemination are in need of modification and innovation.

Footnotes

Acknowledgments

We would like the thank the additional members of the SSC-SIS (Christopher Guidry, Rishi Rattan, April Mendoza, Tina Mele) for their support of the project.

Scientific Studies Committee of the Surgical Infection Society

Haytham M.A. Kaafarani, Massachusetts General Hospital and Harvard Medical School, Boston, MA; Sebastian Schubl, University of California, Irvine, Orange, CA; Philip S. Barie, New York-Presbyterian Hospital/Weill Cornell Medical Center, New York, NY; Rishi Rattan, DeWitt Daughtry Family Department of Surgery, University of Miami, Miami, Fl; Carolyn Twomey, Medical Affairs and Research, IrriMax Corporation, Richmond, VA; Gareth Morris-Stiff, Case Western Reserve University, Cleveland, OH; Leo Andrew Benedict, St. Luke's Hospital of Kansas City, Kansas City, MO; Tina S. Mele, University of Western Ontario, Ontario, Canada; April Mendoza, Massachusetts General Hospital, Boston, MA; Christopher A. Guidry, University of Kansas Medical Center, Kansas City, KS; David Machado-Aranda, Michigan Medicine, Ann Arbor, MI, Dennis Y. Kim, Harbor-UCLA, West Carson, CA; Philip Efron MD, University Florida College of Medicine, Gainesville, FL, Jeffrey Upperman, Vanderbilt University Medical Center, Nashville, TN.

Funding Information

There are no funding sources to disclose for this article.

Author Disclosure Statement

None of the authors listed on this article have financial disclosures to make related to this project.

Supplementary Material

References

Scott

. The direct medical costs of healthcare-associated infections in

U.S

. hospitals and the benefits of prevention. 2009. https://stacks.cdc.gov/view/cdc/11550 (Last accessed March 5, 2020).

Berrios-Torres

, Umscheid

, Bratzler

, et al. Centers for Disease Control and Prevention guideline for the prevention of surgical site infection, 2017. JAMA Surg, 2017; 152:784–791.

Cima

, Dankbar

, Lovely

, et al. Colorectal surgery surgical site infection reduction program: A national surgical quality improvement program-driven multidisciplinary single-institution experience. J Am Coll Surg, 2013; 216:23–33.

Barie

PS.

Outcomes of surgical sepsis. Surg Infect, 2018; 19:230–235.

Barie

PS.

Oh Lord! I've got those clinical research blues. Surg Infect, 2004; 5:327–342.

Nathens

, Cook

, Machiedo

, et al. Defining the research agenda for surgical infection: A consensus of experts using the Delphi approach. Surg Infect, 2006; 7:101–110.

Kirkland

, Briggs

, Trivette

, et al. The impact of surgical-site infections in the 1990s: Attributable mortality, excess length of hospitalization, and extra costs. Infect Control Hosp Epidemiol, 1999; 20:725–730.

Zimlichman

, Henderson

, Tamir

, et al. Health care-associated infections: A meta-analysis of costs and financial impact on the US health care system. JAMA Intern Med, 2013; 173:2039–2046.

Linstone

, Turoff

The Delphi technique. Techniques and applications Available from: web.njit.edu/∼turoff/pubs/delphibook/delphibook.pdf (Last accessed March 4, 2020).

10.

Rowe

, Wright

. Expert opinions in forecasting. Role of the Delphi technique. In: Armstrong JS (ed): Principles of Forecasting: A Handbook for Researchers and Practitioners. New York, NY: Springer Science+Business Media,, 2001:125–144.

11.

Pugh

, Grant

, Cooke

, et al. Short-course versus prolonged-course antibiotic therapy for hospital-acquired pneumonia in critically ill adults. Cochrane Database Syst Rev, 2015:CD007577.

12.

Kalil

, Metersky

, Klompas

, et al. Management of adults with hospital-acquired and ventilator-associated pneumonia: 2016 clinical practice guidelines by the Infectious Diseases Society of America and the American Thoracic Society. Clin Infect Dis, 2016; 63:e61–e111.

13.

Sawyer

, Claridge

, Nathens

, et al. Trial of short-course antimicrobial therapy for intraabdominal infection. N Engl J Med, 2015; 372:1996–2005.

14.

Elstein

, Schwartz

. Clinical problem solving and diagnostic decision making: Selective review of the cognitive literature. BMJ, 2002; 324:729–732.

15.

Klein

Sources of Power. Cambridge, MA: MIT Press, 1998.

16.

Thoma

, Murray

, Huang

SYM

, et al. The impact of social media promotion with infographics and podcasts on re search dissemination and readership. CJEM, 2018; 20:300–306.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.04 MB

0.05 MB

0.00 MB