Contextual Validation of the Prediction of Postoperative Complications of Colorectal Surgery by the “ ACS NSQIP ® Risk Calculator ” in a Tunisian Center

Abstract

Context:

Models for predicting individual risks of surgical complications are advantageous for operative decision making and the nature of postoperative management procedures.

Objective:

Validate the “ACS NSQIP^® Risk Calculator” in the prediction of postoperative complications during colorectal cancer surgery, operated during the years 2015 to 2019.

Methods:

this is a prognostic validation study of the “ACS NSQIP^®” applied retrospectively to patients operated on for colorectal cancer in the surgical department of Farhat Hached hospital, during the 2015 and 2019 5-year term. Three levels of adjustment. Discrimination and calibration were carried out mainly by ROC curves (AUC ⩾ 0.8).

Results:

In this study, 129 patients were included with a sex ratio of 1.22 and a median age of 62 years. The most common operative procedure was low segmental colectomy with colorectal anastomosis. Thirty-seven patients (28.7%) had at least one postoperative complication. The prediction and cuts-off points values of mortality (AUC = 0.858; CI_95% [0.570-0.960]; Cuts-off points = 1.8%), cardiac complications (AUC = 0.824; CI_95% [0.658-0.990]; Cuts-off points = 1.8%), thromboembolic complications (AUC = 0.802; CI_95% [0.617-0.987]; Cuts-off point = 3.1%), and renal insufficiency (AUC = 0.802; CI_95% [ 0.623-0.981]; Cuts-off point = 1.2%) were adjusted according to level 1 of the calculator.

Conclusion:

This work contextualized the prediction of postoperative complications in colorectal surgery in the university general surgery department of Farhat Hached in Sousse (Tunisia), making it possible to improve the quality and safety of surgical care. The application of the Tunisian mini calculator is recommended as well as the generalization of validation following the development of a generic calculator for all operating procedures.

Keywords

General surgery colorectal surgery colorectal tumors health care quality postoperative complications -mortality morbidity prognosis ROC curve Tunisia

Introduction

Postoperative complications of colorectal surgery are still a major concern for patients and surgeons; Despite improvements in surgical technique, anesthesia and perioperative care.¹ Increased by the disposition of patient and the underlying disease, these postoperative complications remain a problem in colorectal surgery² whose frequency continues to increase annually in the world and in Tunisia.³ This type of cancer is the third most common cancer for men after prostate and lung cancer and the second most common after breast cancer for women with 1.93 million new cases per year in 2020.^4,5

The surgical treatment of colorectal cancers is endowed with a significant morbidity and mortality with a rate of 27% of postoperative complications.^2,6 Surgical decisions are traditionally based on a clinical, biological, and radiological preoperative assessment and are based on the individual experience of the surgeon and the arguments of the literature, often in the form of monocentric studies with a low level of evidence.^6,7

To reach consensus on preoperative risk assessment, the American College of Surgery (ACS), in the National Surgical Quality Improvement Program (NSQIP), has developed a surgical risk calculator called ACS NSQIP^® Risk Calculator. This decision-making tool, based on reliable multi-institutional clinical data,⁶ provides to estimate the risks of most surgical operations. ACS NSQIP^® Risk Calculator helps clinicians and patients make decisions based on its specific results. This calculated knowledge of risks is a benefit for better management of high-risk patients and an improvement in the quality, safety and cost of care.⁸

To date, no Tunisian publication has reported results on the use of this calculator. However, the contextual validation of this calculator will allow us to improve the quality of treatment of patients, and to predict post-operative results in Tunisia. It will be a decision support, especially in patients requiring specific management, of scheduled colorectal cancer surgery.

The objective of this article is to validate the “ACS NSQIP^® Risk Calculator” in the prediction of post-operative complications during colorectal cancer surgery in the general surgery department of the “Farhat Hached” Hospital (Sousse—Tunisia).

Population and Methods

This work is part of a preliminary monocentric analysis of the “20/20/20” database of the project “Observatory of Surgical Morbidity in Central-East Tunisia (OMC-CET),” from the Research Laboratory LR19SP01: “Measurement and Support for Hospital Performance,” from the Ministries of Health and Higher Education and Scientific Research, in Tunisia. This is a validation study of a prognostic instrument, involving all patients who have had elective surgery for a colorectal tumor. The data of patients operated on for colorectal cancer were collected, during the period from 01/01/2015 to 31/12/2019, at the “Farhat Hached” hospital in Sousse (Tunisia).

Patients were recruited by consulting the “colon and rectum” coded admission register, the operating report notebooks, and the department’s computer databases containing the discharge conclusions. This study included all patients undergoing elective surgery for colon or rectal cancer in the surgical unit of the “Farhat Hached” Hospital in Sousse from 2015 to 2019. Patients with non-operated colorectal cancers, patients with night surgery for complicated colorectal cancer, and patients with an unusable medical record (not archived, not found, missing or inappropriate data) were not included in this study.

From these data sources, information was collected on the data collection file, developed by the project team, 30 days before and 30 days after surgery, concerning 3 main sections:

1. Identification of the patient, health problem and intervention: surname and first name, number of the medical file, date and place of birth, sex, nationality, address and telephone number, date of the decision to operate, date and mode of admission to the hospital, reason for hospitalization and main preoperative diagnosis with the significant associated diagnoses, date of the surgical intervention, duration of the operation, intraoperative diagnosis, description of the surgical procedure, rank of senior surgeon and assistant, rank of senior anesthesiologist and assistant, operative room exit mode and date, and hospital exit mode.

2. Preoperative predictive factors (30 days before the operation): sex, age group (⩽64 years old/65-74 years old/75-84 years old/⩾85 years old), functional status of the patient (independent/partially dependent / totally dependent) and ASA class (American Society of Anesthesiologists physical Status classification system), mode of admission, use of corticosteroids and immunosuppressants, presence of ascites, presence and degree of systemic sepsis in the 48 hours before the intervention, mechanical ventilation in the 48 hours before the surgery, neoplastic context with its nature and stage (metastatic or not), medical history: type of diabetes, arterial hypertension, presence of congestive heart failure preceding the intervention, degree of dyspnea, smoking, presence of chronic obstructive pulmonary disease, if the patient is on dialysis and the presence of preoperative acute renal failure, weight, height, body mass index, and nutritional status of the patient.

3. Postoperative complications (new events appearing during the 30 postoperative days), medical (pneumonia, prolonged use of mechanical ventilation for more than 48 hours, cardiac complications: myocardial infarction and cardiac arrest, unexpected intubation, urinary tract infection, sepsis, thromboembolic events, progressive or acute renal failure) or surgical (surgical, incisional or visceral site infection, Clostridium difficile colitis, Prolonged ileus, anastomotic leaks, postoperative readmission, return to the operating room, hemorrhage per or postoperative, bedsore) and mortality.

The “ACS NSQIP^® Risk calculator” for postoperative complications was systematically applied.⁹ It estimated the patient’s individual risk according to 3 levels of the calculator (level 1: no adjustment necessary, level 2: risk somewhat higher than estimated, level 3: risk significantly higher than expected) as well as the average risk of each complication, calculated on the population used to construct the logistic regression models, as a reference to the individual risks.⁶

The individual patient characteristics listed in the calculator form were manually entered into the “ACS NSQIP^® Risk calculator,” including the code CPT (Current Procedural Terminology).¹⁰

For the question “Are there other potential appropriate treatment options?,” the answer “None” was always selected. Next, the three levels of adjustment were noted in “Surgeon Adjustment of Risks.” The patient’s estimated personal risks for each post-operative adverse event were recorded as a percentage.

Data entry was done on SPSS for a preliminary analysis of the “20/20/20” database. The demographic data and the clinical characteristics of the patients were summarized by the mean and the standard deviation or the median and the Inter Quartile Intervals (IQI) for the quantitative variables and in percentage for the categorical variables.

The performance of the ACS NSQIP^® Risk Calculator in predicting risk was evaluated using 2 metrics for categorical outcomes, the C-statistic and the Brier score. The C-statistic is a test that represents the Area Under the Curve (AUC) of the ROC “Receiver Operating Characteristic” curve allowing discrimination. This curve is constructed from the values of sensitivity and (1—specificity) for different thresholds of the variable to be tested. Discrimination was rated as poor (AUC of 0.6-0.69), adequate (AUC of 0.7-0.79), good (AUC of 0.8-0.89), or excellent (AUC of 0.9-1.0).^11-13 The curves were constructed based on the predicted values according to the real state of the patient postoperatively (complication present or absent). Thereafter, the cuts-off points of the ROC curves¹⁴ were calculated, with the¹⁴ score.¹⁵ The Brier score, used to evaluate discrimination and calibration, is defined as the mean square difference between the predicted probabilities and the value assigned by binary coding, that is, 0 for “absent event” and 1 for “present event.” It has the advantage of being influenced by both discrimination and calibration. A value of 0 indicates a perfect prediction model. Values between 0 and 0.25 were considered informative.^11,12 The concordance between the C-statistic and the Brier score was assessed by a composite index “performance score” or “ Brier Skill Score ” (BSS) calculated according to formula 1—(Brier score/AUC).¹⁶ For each complication, the 3 values given by the adjustment of the calculator were tested. During this study, the validation of the discrimination was retained, through the ROC curve, if the AUC was greater than 0.8 corresponding to a good prediction. The Brier score greater than 0.25 indicates that there was a bad calibration.

Results

During the 2015 to 2019 5-year period, a total of 186 patients were operated on in the general and digestive surgery department of the Farhat Hached hospital in Sousse (Tunisia) for colorectal cancer, including 24 operated during nightshift and 33 presenting a missing data. Thus, the study population was composed of 129 patients whose sex ratio was 1.22 and the median age was 62 years, with extremes [23-88 years] and an IQI of [51-71 years]. The median postoperative length of stay was 7 days with an IQI of [5-9 days] and extremes of [2-30 days].

Demographic and clinical characteristics of patients

“Scheduled” admissions were the dominant mode of entry into the general surgery department of the CHU Farhat Hached in Sousse, with a rate of 64% (Table 1). Tumors of the middle and lower rectum were the most frequent (35%). The most applied colorectal cancer surgical procedures were low segmental colectomy with colorectal anastomosis (18%), right colectomy with ileo-transverse anastomosis (16%) and anterior rectal resection with ileo-rectal anastomosis (12%) (Table 1).

Table 1.

Demographic, clinical and surgical characteristics of the 129 patients operated on for colorectal cancer, at the general surgery department of CHU Farhat Hached in Sousse between the years 2015 and 2019.

	Male	Feminine	Total
	n = 71 (%)	n = 58 (%)	N = 129 (%)
Home governorates
Sousse	34 (47.9)	34 (58.6)	68 (52.7)
Monastir	10 (14.1)	7 (12.1)	17 (13.2)
Kasserine	10 (14.1)	3 (5.2)	13 (10.1)
Kairouan	7 (9.9)	5 (8.6)	12 (9.3)
Mahdia	5 (7.0)	2 (3.4)	7 (5.4)
Sidi Bouzid	2 (2.8)	3 (5.2)	5 (3.9)
Others	3 (4.2)	4 (6.8)	7 (5.4)
Input methods
Admission programed by the service	47 (66.2)	35 (60.3)	82 (63.6)
External consultations	15 (21.1)	15 (25.9)	30 (23.3)
Emergencies	5 (7.0)	6 (10.3)	11 (8.5)
Transfer from another service	3 (4.2)	1 (1.7)	4 (3.1)
Transfer from another hospital	1 (1.4)	1 (1.7)	2 (1.6)
Tumor sites
Mid/lower rectal tumor	25 (35.2)	20 (34.5)	45 (34.9)
Left colonic tumor	15 (21.1)	20 (34.5)	35 (27.1)
Right colonic tumor	17 (23.9)	10 (17.2)	27 (20.9)
Upper rectum tumor	8 (11.3)	4 (6.9)	12 (9.3)
Diffuse colonic polyposis	6 (8.5)	4 (6.9)	10 (7.8)
Operating procedures
Lower segmental colectomy with colorectal anastomosis	8 (11.3)	15 (25.9)	23 (17.8)
Right colectomy with ileotransverse anastomosis	16 (22.5)	5 (8.6)	21 (16.3)
Anterior rectal resection with colorectal anastomosis	8 (11.3)	7 (12.1)	15 (11.6)
Total colectomy with ileorectal anastomosis	5 (7.0)	6 (10.3)	11 (8.5)
Anterior resection of the rectum with colo-anal anastomosis	5 (7.0)	5 (8.6)	10 (7.8)
Anterior resection of the rectum with protected colo-anal anastomosis	6 (8.5)	3 (5.2)	9 (7.0)
Abdomino-perineal amputation with left iliac colostomy	4 (5.6)	3 (5.2)	7 (5.4)
Total colon proctectomy with ileo anal anastomosis	5 (7.0)	1 (1.7)	6 (4.7)
Others	14 (19.7)	13 (22.4)	27 (20.9)

Postoperative complications

Out of a total of 129 patients, 37 (29%) had at least one postoperative complication mainly of the type: sepsis, surgical site infection and anastomotic leak (Table 2). Patients who had serious complications involving their vital prognosis represented 16%. Death has been reported for 7 operated patients.

Table 2.

Distribution, by sex, of postoperative complications of colorectal cancer surgery observed in patients operated on at the surgery department of CHU Farhat Hached in Sousse, between the years 2015 and 2019. n (%).

	Male n = 71 (%)	Female n = 58 (%)	Total N = 129 (%)
Any complications	24 (33.8)	13 (22.4)	37 (28.7)
Sepsis	18 (25.4)	11 (19.0)	29 (22.5)
Surgical site infection	15 (21.1)	12 (20.7)	27 (20.9)
Transfusion	13 (18.3)	9 (15.5)	22 (17.1)
Serious complications	14 (19.7)	7 (12.0)	21 (16.2)
Anastomotic leak	11 (15.5)	8 (13.8)	19 (14.7)
Return to the operating room	11 (15.5)	6 (10.3)	17 (13.2)
Prolonged ileus	7 (9.9)	9 (15.5)	16 (12.4)
Renal failure	10 (14.1)	3 (5.2)	13 (10.1)
Readmission	10 (14.1)	3 (5.2)	13 (10.1)
Unplanned intubation	7 (9.9)	3 (5.2)	10 (7.8)
Urinary tract infection	3 (4.2)	6 (10.3)	9 (7.0)
Ventilation >48 h	4 (5.6)	3 (5.2)	7 (5.4)
Pneumonia	4 (5.6)	3 (5.2)	7 (5.4)
Death	6 (8.5)	1 (1.7)	7 (5.4)
Cardiac complication	4 (5.6)	1 (1.7)	5 (3.9)
Thromboembolic event	4 (5.6)	0 (0.0)	4 (3.1)

Preoperative predictive factors

Of the 20 predictive factors of postoperative complications researched, the “under 65” age group was predominant (60%) (Table 3). Arterial hypertension and diabetes were present in 27% and 20% of patients respectively. Patients classified as ASA II represented 52% and the cancers were metastatic in 16% of cases. Cardiac events occurred during the last 30 preoperative days in 7% of patients.

Table 3.

Distribution, according to predictive factors and according to sex, of the 129 patients operated on for colorectal cancers in the general surgery department of the CHU Farhat Hached in Sousse (Tunisia), between the years 2015 and 2019. n (%).

	Male n = 71 (%)	Female n = 58 (%)	Total N = 129 (%)
Age (years)
⩽64	41 (57.7)	36 (62.1)	77 (59.7)
65-74	19 (26.8)	11 (19.0)	30 (23.3)
75-84	9 (12.7)	9 (15.5)	18 (14.0)
⩾85	2 (2.8)	2 (3.4)	4 (3.1)
Body mass index (Kg/ m²⁾
Regular <25	26 (36.6)	16 (27.6)	42 (32.6)
Overweight 25-29	19 (26.8)	22 (37.8)	41 (31.8)
Obesity ⩾30	3 (4.2)	5 (8.6)	8 (6.2)
Missing values	23 (32.4)	15 (25.9)	38 (29.5)
Functional independence
Independent	67 (94.4)	52 (89.7)	119 (92.2)
Partially dependent	3 (4.2)	6 (10.3)	9 (7.0)
Totally dependent	1 (1.4)	0 (0.0)	1 (0.8)
History of COPD*
Yes	6 (8.5)	0 (0.0)	6 (4.7)
ASA Score^†
I	19 (27.1)	21 (36.2)	40 (31.2)
II	40 (57.1)	26 (44.8)	66 (51.6)
III	11 (15.7)	11 (19.0)	22 (17.2)
Chronic use of corticosteroids
Yes	4 (5.6)	1 (1.7)	5 (3.9)
Metastatic cancer
Yes	10 (14.1)	11 (19.0)	21 (16.3)
Diabetes
Under OAD^‡	14 (19.7)	9 (15.5)	23 (17.8)
On insulin	2 (2.8)	1 (1.7)	3 (2.3)
High blood pressure
Yes	16 (22.5)	19 (32.8)	35 (27.1)
Dyspnea
At rest	1 (1.4)	0 (0.0)	1 (0.8)
With moderate effort	5 (7.0)	5 (8.6)	10 (7.8)
Acute renal failure
Yes	1 (1.4)	0 (0.0)	1 (0.8)
Ascites
Yes	2 (2.8)	1 (1.7)	3 (2.3)
Sepsis (last 48 h)
Sepsis	4 (5.6)	3 (5.2)	7 (5.4)
Smoking
Yes	17 (23.9)	2 (3.4)	19 (14.7)
Cardiac event
Yes	5 (7.0)	4 (6.9)	9 (7.0)
Anemia
Yes	21 (29.6)	10 (17.2)	31 (24.0)
Anticoagulation
Yes	2 (2.8)	3 (5.2)	5 (3.9)
Platelet aggregation inhibitor
Yes	7 (9.9)	4 (6.9)	11 (8.5)
Alcoholism
Yes	4 (5.6)	0 (0.0)	0 (0.0)

Chronic obstructive pulmonary disease.

†

American Society of Anesthesiologists.

‡

Orals antidiabetics.

Validation of the “ACS NSQIP^® Risk calculator”

The best prediction of postoperative complications was observed for level 1 of adjustment of the “ACS NSQIP^® Risk calculator” (Table 4 and Figure 1). The discriminant performance was good for the prediction of death with an AUC of 0.858 (CI_95% [0.757-0.960], P < .001), cardiac complications with an AUC of 0.824 (CI_95% [0.658-0.990], P < .028), thromboembolic events with an AUC of 0.802 (CI_95% [0.617-0.987], P < .040) and for renal failure with an AUC of 0.802 (CI_95% [0.623-0.981], P < .040). The cuts-off points representing the probabilities above which the risk of having complications are mentioned in Table 5.

Table 4.

Comparison of the different Areas Under the Curve (AUC) for the 3 adjustment levels of the “ACS NSQIP^® Risk Calculator” and their Brier scores for postoperative complications of patients operated on for colorectal cancers in the surgery department of University Hospital of Farhat Hached in Sousse (Tunisia), between the years 2015 and 2019.

Complications	AUC Lvl 1	CI_95%*		Brier score Lvl 1	AUC Lvl 2	CI _95%*		Brier score Lvl 2	AUC Lvl 3	CI _95%*		Brier score Lvl 3
Complications	AUC Lvl 1	Inf.	Sup.	Brier score Lvl 1	AUC Lvl 2	Inf.	Sup.	Brier score Lvl 2	AUC Lvl 3	Inf.	Sup.	Brier score Lvl 3
Death	0.858	0.757	0.960	0.044	0.743	0.601	0.885	0.050	0.618	0.450	0.885	0.083
Cardiac complications	0.824	0.658	0.990	0.030	0.757	0.558	0.957	0.029	0.673	0.468	0.877	0.031
Renal failure	0.802	0.623	0.981	0.089	0.690	0.408	0.972	0.086	0.665	0.395	0.935	0.085
Thromboembolic event	0.802	0.617	0.987	0.029	0.688	0.453	0.923	0.028	0.471	0.290	0.651	0.030
Surgical site infection	0.691	0.565	0.818	0.177	0.688	0.552	0.823	0.166	0.686	0.539	0.833	0.162
Serious complications	0.650	0.505	0.795	0.127	0.623	0.476	0.769	0.138	0.592	0.441	0.743	0.165
Readmission	0.643	0.505	0.780	0.091	0.632	0.492	0.771	0.094	0.543	0.382	0.704	0.102
Pneumonia	0.619	0.454	0.784	0.053	0.468	0.300	0.637	0.052	0.44	0.295	0.585	0.056
Sepsis	0.541	0.409	0.672	0.202	0.534	0.406	0.662	0.193	0.527	0.401	0.654	0.188
All complications	0.533	0.418	0.647	0.209	0.513	0.400	0.627	0.208	0.481	0.369	0.592	0.230
Anastomotic release	0.466	0.328	0.604	0.148	0.460	0.321	0.598	0.148	0.499	0.342	0.656	0.145
Return to the operating room	0.450	0.288	0.611	0.118	0.493	0.327	0.659	0.112	0.477	0.311	0.643	0.111
Urinary tract infection	0.442	0.243	0.640	0.067	0.387	0.211	0.563	0.067	0.313	0.176	0.450	0.067
Colonic ileus	0.378	0.205	0.552	0.127	0.387	0.213	0.561	0.147	0.345	0.174	0.516	0.187

Abbreviations: Inf., inferior; Lvl, risk calculator adjustment level; Sup, superior.

95% Confidence interval.

Bold entries for signifcants results.

Figure 1.

ROC curves for postoperative complications of colorectal cancer surgery in the general surgery department of Farhat Hached in Sousse, Tunisia between the years 2015 and 2019, for level 1 adjustment of the “ACS NSQIP^® Risk Calculator.”

Table 5.

Classification of complications according to the Brier skill score for level 1 adjustment of the “ACS NSQIP^® Risk Calculator.”

	AUC*	Brier	BSS^†
Thromboembolic event	0.802	0.029	0.964
Cardiac complication	0.824	0.030	0.964
Death	0.858	0.044	0.949
Pneumonia	0.619	0.053	0.914
Renal failure	0.802	0.089	0.889
Readmission	0.643	0.091	0.858
Urinary tract infection	0.442	0.067	0.848
Serious complications	0.650	0.127	0.805
Surgical site infections	0.691	0.177	0.744
Return to the operating room	0.450	0.118	0.738
Anastomotic leak	0.466	0.148	0.682
Prolonged ileus	0.378	0.127	0.664
Sepsis	0.541	0.202	0.627
Any complications	0.533	0.209	0.608

Area Under the Curve.

†

Brier Skill Score _ _

Bold entries for significant results.

Discussion

During surgery, a thorough understanding of the potential risks of surgery¹⁷ facilitates rational treatment decisions and informed patient consent. The application of statistical postoperative risk assessment models in surgical environments different from those in which they were developed requires prior validation.¹⁸ This study focused on one of the prediction models adopted in the United States of America, not yet currently used in Tunisia and the Maghreb countries, and therefore needs to be contextually validated. The choice of colorectal oncological surgery was motivated by the permanent increase in its incidence in third-line reference hospitals (university hospitals), second-line (regional hospitals) and in private medicine, in Tunisia. During colorectal surgery, understanding the complications will make it possible to change the operating and perioperative procedures,^8,19 based on the ROC curve, the Brier score, and its performance score.

Despite the validity of the ACS NSQIP^® risk calculator for the major complications reported in this study, certain limitations should be noted: the use of retrospective recruitment (missing data), the small sample size and adverse events and the monocentric nature making it difficult to generalize its use. However, this work is part of a large multicenteric study for all operative procedures adapted to surgical practice at Tunisian university hospitals. This work remains a pioneer in the contextual evaluation of the postoperative risk calculator, having developed cuts-off specific points.

Addition to the “ACS NSQIP^® Risk calculator,” many institutions have created postoperative risk assessment and prediction tools to aid decision-making, such as the “Cleveland Clinic Foundation colorectal cancer model ” (CCF-AFC), “the Physiological and Operative Severity Score for enUmeration of Mortality and morbidity” (POSSUM), and “Colorectal preOperative Surgical Score” (CrOSS), only assessing mortality risk or are difficult and complex to use at the bedside.^20,21 This calculator has been validated in the American context (country with a strong economic base and the best conditions of care for patients), hence the need for its contextualization in middle-income countries, including Tunisia.⁶

We therefore concluded, based on the discrimination by the ROC curves and the calibration by the Brier score and its skill score, to the validation of the reliability and reproducibility of this score for level 1 of adjustment of the calculator for cardiac complications, thromboembolic events, renal complications, and mortality.

Mortality prediction

The prediction of death in this study was consistent with that predicted by the risk calculator with an AUC of 0.858 (CI_95% [0.757-0.960] with P < .001) and a Brier score of 0.044. These results were consistent with those in the literature. A single-center American study found a good prediction of mortality with an AUC of 0.969 (CI_95% [0.932-1.000]) for a P < .001 and a Brier score of 0.014.²² Another Chinese study reported an AUC of 0.827 and a Brier score of 0.022.²³ While a New York study found an AUC of 0.723 without evaluating the Brier score.²⁴ Note that the mortality data, on which the risk calculator was designed, involved 88 334 colorectal procedures, the AUC found was 0.919 and a Brier index of 0.029.⁶

Prediction of cardiac complications

The prediction of cardiac complications in this study was consistent with that predicted by the calculator with an AUC of 0.824 (CI_95% [0.658-0.990], P < .028) and a Brier score of 0.030. The AUC result was close to that reported by the original US study.⁶

Prediction of thromboembolic complications

The prediction of the results of this study was consistent with the result given by the calculator with an AUC higher than the Bilimoria et al⁶ data (an AUC of 0.738 for colorectal procedures vs an AUC of 0.802). However, the prediction of this complication according to the Chinese study was lower (AUC of 0.607).²³

Prediction of kidney failure

In this study, the prediction was consistent with those predicted by the calculator with an AUC of 0.802 and a Brier score of 0.089. Similar results were reported by Bilimoria et al.⁶ However, the Chinese study²³ did not conclude with good precision with an AUC of 0.628. Table 6 summarizes the data from the literature.

Table 6.

Comparison between the different results of ASCs and Brier indices found in the biomedical literature.

Complications	CHU Farhat Hached n = 129		Hyde et al²² n = 288		Keller et al²⁴ n = 355		Hu et al²³ n = 18 544		Bilimoria et al⁶ n = 88 334
Complications	AUC* (CI_95%)^†	Brier	AUC* (CI_95%)^†	Brier	AUC* (CI_95%)^†	Brier	AUC*	Brier	AUC*	Brier
Mortality	0.858 (0.757-0.960)	0.044	0.969 (0.932-1.000)	0.0139	0.723 (0.453-0.992)	—-	0.827	0.02216	0.919	0.0291
Cardiac complication	0.824 (0.658-0.990)	0.03	—-		—-	—-	0.727	0.00454	0.844	0.0171
Renal failure	0.802 (0.623-0.981)	0.089	—-		—-	—-	0.628	0.00754	0.82	0.0162
Thromboembolic event	0.802 (0.617-0.967)	0.029	0.992 (0.982-1.000)	0.0098	—-	—-	0.607	0.0158	0.738	0.0218
Surgical site infection	0.691 (0.565-0.818)	0.177	0.568 (0.454-0.681)	0.0721	—-	—-	0.595	0.06913	0.671	0.1004
serious complication	0.650 (0.505-0.795)	0.127	0.685 (0.598-0.772)	0.144	0.544 (0.452-0.637)	—-	—-	—-	—-	—-
Readmission	0.643 (0.505-0.780)	0.091	0.516 (0.393-0.638)	0.0925	0.569 (0.468-0.670)	—-	—-	—-	—-	—-
Pneumonia	0.619 (0.454-0.784)	0.053	—-	—-	—-	—-	0.719	0.02792	0.819	0.0296
Sepsis	0.541 (0.409-0.672)	0.202	—-	—-	—-	—-	0.627	0.04057
Any complications	0.533 (0.418-0.647)	0.209	0.680 (0.598-0.761)	0.1535	0.566 (0.503-0.629)	—-	—-	—-	0.726	0.1598
Anastomotic release	0.466 (0.328-0.604)	0.148	—-	—-	—-	—-	—-	—-	—-	—-
Return to the operating room	0.450 (0.288-0.611)	0.118	0.635 (0.488-0.781)	0.0651	0.513 (0.341-0.686)	—-	0.601	0.05301	—-	—-
Urinary tract infection	0.442 (0.243-0.640)	0.067	—-	—-	—-	—-	—-	—-	0.727	0.0302
Colonic ileus	0.378 (0.205-0.552)	0.127	—-	—-	—-	—-	—-	—-	—-	—-

Area Under the Curve.

†

Confidence interval at 95%.

Bold entries for significants results.

For the 4 complications for which compliance was good (mortality, venous thrombosis, renal failure, and cardiac complications), the risk thresholds were calculated, varying between 1.2% and 3.1%. These low risk threshold values predicted by the calculator remind surgeons that the zero risk of complications from colorectal surgery does not exist.

Thus, the postoperative risk calculator for colorectal cancer in Tunisia, adjusted according to national clinical practices, not only has a good prediction of major complications but also has an impact on the quality, safety, and cost of care. The use of these objective indicators would allow the surgeon to avoid relying on his subjective intuition and could improve postoperative results. In practice, we propose the use of the “ACS NSQIP^® Risk Calculator,” adapted to the particularities of the Sousse series, for colorectal cancer surgery in Tunisia. If the patients would have a risk higher than the cuts-off points, additional prevention should be initiated, either surgery by a panel of senior surgeons, reinforced by experts in anesthesia-resuscitation or the referral of patients to a center specialized in oncological surgery. The prevention of cardiac and/or renal complications requires hospitalization in intensive care for close monitoring and the involvement of cardiologists and/or nephrologists in the management of high-risk patients.²⁵ In addition, avoidance of renotoxic drugs, adequate hydration, proximity to a dialysis or hemofiltration unit²⁶ and early removal of bladder sampling are preventive measures for postoperative renal failure and urinary tract infections.²⁷

The ability to reduce the mortality rate associated with complications in colorectal surgery (Failure To Rescue) depends largely on the compliance of services with the standards of good clinical practice, on the human capacity to detect and respond early to the clinical signs of patients with complications.^28,29 The establishment of centers specializing in colorectal surgery will increase the expertise of health personnel in the detection and treatment of complications of colorectal surgery. Training surgeons and anesthesiologists in the use of this contextualized calculator for predicting postoperative complications is essential for the success of its implementation. This original calculator could be part of the checklist when the patient is admitted to the service and when entering the operating room. Such training is a fundamental component of the Master class of safety and quality of care, in surgical specialties. Multicenter research in the Tunisian and North African context remains essential for the generalization of the validation of the model for predicting postoperative complications in general surgery: a pillar of the strategy to support quality and safety in care.

Finally, at the end of this study, the model of the “ACS NSQIP^® Risk Calculator” for the prediction of postoperative complications, has been validated in the Tunisian context, in the field of colorectal cancer surgery, for the following items: mortality, cardiac and thromboembolic complications, and renal failure, with AUC greater than 0.8 and Brier indices less than 0.1, adjusting according to level 1. Cuts-off points with sensitivity and specificity greater than 75%, helped in the application of a contextualized flowchart for cancer surgery colorectals in Tunisia (Box 1). Hence the interest of the generalization of the contextual validation of this calculator with all the procedures of general surgery, in the different levels of the health system of the countries of the Greater Maghreb.

Box 1.

Risk assessment grid for patients with colorectal tumors in the general surgery department of CHU Farhat Hached in Sousse (Tunisia).

Footnotes

Funding:

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Declaration Of Conflicting Interests:

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Author Contributions

MBA: Collecting Data, writing Manuscript.

MAE: Study design, revising the manusript.

AM: collecting Data.

ABA: Stady design, revising the manuscript.

Data Availability Statement

Data related to this manuscript is available on the hand of corresponding author and will be obtained if requested.

Ethical Statements and Informed Consent Information

The study was conducted after getting ethical clearance from the ethical clearance committee of Farhat Hached university hospital. Data were collected after getting consent from all patients. All patient treated in our department are agree to using their information in retrospectives studies.

References

Varadhan

Neal

Dejong

Fearon

Ljungqvist

Lobo

DN.

The enhanced recovery after surgery (ERAS) pathway for patients undergoing major elective open colorectal surgery: a meta-analysis of randomized controlled trials. Clin Nutr. 2010;29:434-440.

Park

Kim

YW.

The American Society of Anesthesiologists score influences on postoperative complications and total hospital charges after laparoscopic colorectal cancer surgery. Medicine. 2018;97:e0653.

Arfaoui Toumi

Kriaa Ben Mahmoud

Khiari

, et al. [Epidemiological study, pathologic evaluation and prognostic factors of colorectal mucinous vs non-mucinous adenocarcinoma (a series of 196 patients)]. Tunis Med. 2010;88:12-17.

World Health Organization. Global cancer epidemiology 2021. https://www.who.int/fr/news-room/fact-sheets/detail/cancer

World Health Organization. Profile of cancers in Tunisia. https://cdn.who.int/media/docs/default-source/country-profiles/cancer/tun-2020.pdf?

Bilimoria

Liu

Paruch

, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217:833-842.e1.

Tan

Talbott

Leong

Isenberg

Goldstein

SD.

American Society of Anesthesiologists class and Charlson’s comorbidity index as predictors of postoperative colorectal anastomotic leak: a single-institution experience. J Surg Res. 2013;184:115-119.

Sharp

Malizia

Skancke

, et al. A NSQIP analysis of trends in surgical outcomes for rectal cancer: what can we improve upon? Am J Surg. 2020;220:401-407.

American College of Surgery. ACS risk calculator website. https://riskcalculator.facs.org/RiskCalculator/about.html

10.

Thorwarth

WT.

CPT®: an open system that describes all that you do. J Am Coll Radiol. 2008;5:555-560.

11.

Alba

Agoritsas

Walsh

, et al. Discrimination and calibration of clinical prediction models: users’ guides to the medical literature. JAMA. 2017;318:1377.

12.

Steyerberg

Vickers

Cook

, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128-138.

13.

Merkow

Hall

Cohen

, et al. Relevance of the c-statistic when evaluating risk-adjustment models in surgery. J Am Coll Surg. 2012;214:822-830.

14.

Larner

AJ.

Defining ‘optimal’ test cut-off using global test metrics: evidence from a cognitive screening instrument. Neurodegener Dis Manag. 2020;10:223-230.

15.

Fluss

Faraggi

Reiser

Estimation of the Youden index and its associated cutoff point. Biom J. 2005;47:458-472.

16.

Weigel

Liniger

Appenzeller

The discrete Brier and ranked probability skill scores. Mon Weather Rev. 2007;135:118-124.

17.

Knops

Legemate

Goossens

Bossuyt

Ubbink

DT.

Decision aids for patients facing a surgical treatment decision: a systematic review and meta-analysis. Ann Surg. 2013;257:860-866.

18.

Liu

Sharma

Application of the ACS NSQIP surgical risk calculator on a global scale. Am J Surg. 2021;222:876.

19.

Al-Mazrou

Haiqing

Guanying

Kiran

RP.

Sustained positive impact of ACS-NSQIP program on outcomes after colorectal surgery over the last decade. Am J Surg. 2020;219:197-205.

20.

Baré

Alcantara

Gil

, et al. Validity of the CR-POSSUM model in surgery for colorectal cancer in Spain (CCR-CARESS study) and comparison with other models to predict operative mortality. BMC Health Serv Res. 2018;18:49.

21.

Kong

Guest

Stupart

Faragher

Chan

Watters

DA.

Colorectal preOperative surgical score (CrOSS) for mortality in major colorectal surgery. ANZ J Surg. 2015;85:403-407.

22.

Hyde

Valizadeh

Al-Mazrou

Kiran

RP.

ACS-NSQIP risk calculator predicts cohort but not individual risk of complication following colorectal resection. Am J Surg. 2019;218:131-135.

23.

Chen

Lee

, et al. Assessment of the addition of hypoalbuminemia to ACS-NSQIP surgical risk calculator in colorectal cancer. Medicine. 2016;95:e2999.

24.

Keller

Mercadel

Ogola

Steele

SR.

Are we taking a risk with risk assessment tools? Evaluating the relationship between NSQIP and the ACS risk calculator in colorectal surgery. Am J Surg. 2018;216:645-651.

25.

Fields

Dionigi

Scully

, et al. Reduction in cardiac arrhythmias within an enhanced recovery after surgery program in colorectal surgery. J Gastrointest Surg. 2020;24:1158-1164.

26.

Moghadamyeghaneh

Phelan

Carmichael

, et al. Preoperative dehydration increases risk of postoperative acute renal failure in colon and rectal surgery. J Gastrointest Surg. 2014;18:2178-2185.

27.

Farsi

AH.

Risk factors and outcomes of postoperative catheter-associated urinary tract infection in colorectal surgery patients: a retrospective cohort study. Cureus. 2021;13:e15111.

28.

Silber

Williams

Krakauer

Schwartz

JS.

Hospital and patient characteristics associated with death after surgery. A study of adverse occurrence and failure to rescue. Med Care. 1992;30:615-629.

29.