Abstract
Introduction
The Surgical Risk Preoperative Assessment System is a parsimonious, universal surgical risk calculator integrated into our local electronic health record. We determined how many of its eight preoperative risk predictor variables could be automatically obtained from the electronic health record. This has implications for the usability and adoption of Surgical Risk Preoperative Assessment System, serving as an example of use of electronic health record data for populating clinical decision support tools.
Methods
We quantified the availability and accuracy in the electronic health record of the eight Surgical Risk Preoperative Assessment System predictor variables (patient age, American Society of Anesthesiology physical status classification, functional health status, sepsis, work Relative Value Unit, in-/outpatient operation, surgeon specialty, emergency status) at the patient’s preoperative encounter of 5205 patients entered into the American College of Surgeons National Surgical Quality Improvement Program. Accuracy was determined by comparing the electronic health record data to the same patient’s National Surgical Quality Improvement Program data, used as the “gold standard.” Acceptable accuracy was defined as a Kappa statistic or Pearson correlation coefficient ≥0.8 when comparing electronic health record and National Surgical Quality Improvement Program data. Acceptable availability was defined as presence of the variable in the electronic health record at the preoperative encounter ≥95% of the time.
Results
Of the eight predictor variables, six had acceptable accuracy. Only preoperative sepsis and functional health status had Kappa statistics <0.8. However, only patient age and surgeon specialty were ≥95% available in the electronic health record at the preoperative visit.
Conclusions
Processes need to be developed to populate more of the Surgical Risk Preoperative Assessment System preoperative predictor variables in the patient’s electronic health record prior to the preoperative visit to lessen the burden on the busy surgeon and encourage more widespread use of Surgical Risk Preoperative Assessment System.
Introduction
Preoperative prediction of risk of adverse postoperative outcomes has the potential to improve surgical outcomes. It is our belief that a prediction system has to have the following characteristics in order for it to be widely used. It should (1) cover a broad range of operations, (2) be parsimonious in the number of predictor variables, (3) cover a broad range of outcomes, (4) be accurate, and (5) be built into the electronic health record (EHR) for easy use by busy healthcare providers. 1
Over the past several years, the Surgical Risk Preoperative Assessment System (SURPAS) has been developed using a series of risk calculation algorithms for the preoperative prediction of the risk of common adverse outcomes in patients undergoing surgery in nine surgical specialties.2–4 The associated research has focused on reducing the data collection burden by conducting a series of analyses of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) participant use file (PUF), comprised at the time of data on 2.3 million operations performed from 2005 to 2012. Through a factor analysis of the 18 individual ACS NSQIP postoperative adverse outcomes, it was found that the outcomes could be reduced to eight meaningful groups of outcomes. 2 Through a series of stepwise multiple logistic regression analyses, it was found that the eight most important predictor variables accounted for 97.9–99.2% of the total c-index of the full models using all of the 28 non-laboratory NSQIP preoperative predictor variables. 3
The eight SURPAS risk predictors are patient age, American Society of Anesthesiology physical status classification (ASA class), systemic sepsis within 48 h prior to the operation, functional health status of the patient prior to surgery, the complexity of the primary operation as represented by work Relative Value Unit (wRVU), in- or outpatient setting for the operation, specialty of the primary surgeon, and the emergency status of the operation. The SURPAS outcomes are the patient’s risk of eight 30-day postoperative adverse events: mortality, overall morbidity, and six clusters of postoperative complications: infection, pulmonary, cardiac/transfusion, venous thromboembolism, renal, and stroke.
Another universal risk calculator available for use is the ACS Surgical Risk Calculator. 5 That risk calculator is accessed via the Internet and calls for the entry of 21 preoperative variables to obtain the patient’s individual risk compared to national averages for a broad array of adverse postoperative outcomes. Although we have found this calculator to be useful on occasion for individual cases, SURPAS has been developed to be more easily used during the compressed clinical encounter by calling for fewer variables to be entered, being integrated into the EHR, and adding other desirable features such as an automatic preoperative note written to the patient’s EHR record and a printed graphical display of the patient’s risks compared to national averages for the patient to take home. 6 It is hoped that these features of SURPAS will make it more widely used throughout our healthcare system. Examples of the SURPAS input screen, histograms of a patient’s SURPAS risks compared to national averages, and the graphical display of the patient risks compared to national averages given to the patient have been published. 6
To make SURPAS even more user friendly, it was sought to determine how many of the eight preoperative predictor variables could reliably be obtained from the patient’s EHR documentation at the preoperative visit. The primary aim of this study was to better understand the accuracy and availability of the eight SURPAS predictor variables in the EHR. Accuracy was measured by the agreement between the electronically collected values of the eight predictor variables from the EHR and those collected by trained nurse reviewers for the ACS NSQIP. The availability of the predictor variables in the EHR for automated abstraction at the preoperative encounter was determined by comparing the dates of the preoperative encounter and when the data values of the predictor variables were first entered into the EHR.
Methods
Data sources
All surgical cases entered into the ACS NSQIP from the University of Colorado Hospital (UCH) between 4 July 2013 and 20 January 2016 were included. The ACS NSQIP is a national surgical quality improvement program in which trained clinical nurse reviewers abstract preoperative, operative, and postoperative data from a systematic sample of surgical patients at participating hospitals. Abstracted data are then submitted to the ACS for comparison of risk-adjusted outcomes between participating hospitals. Using the patient’s medical record number and date of surgery, two of the coauthors (JAS, JMM) performed queries of the EHR and generated spreadsheets of the required data elements. These corresponding EHR data were linked to each patient’s NSQIP data.
The ACS NSQIP data were considered to be the “gold standard” because: (1) each variable has a published, standardized definition; (2) the data are collected by trained nurses who review the patient’s medical record and synthesize the data to make educated interpretations in consultation with the surgeons to determine accurate values of the variables; and (3) the data are periodically audited.7,8 For some variables (e.g. patient age, ASA class, etc.), very high agreement between EHR and NSQIP values might be expected because the NSQIP nurses obtain these data directly from a single structured field in the EHR. For other variables (e.g. preoperative sepsis and functional health status), the level of agreement might be less because the variables involve multiple fields in the EHR, International Classification of Diseases (ICD) codes, or other records (e.g. clinic notes, laboratory values, etc.).
For the EHR data, age, ASA class, in- or outpatient procedure, and primary surgeon specialty were extracted directly from the local EHR (Epic Systems, Verona, WI) as structured data from single fields. The wRVU was obtained from a look-up table based on the primary operation’s current procedural terminology (CPT) code. 9 Multiple structured data sources within the EHR were examined for the remaining three predictor variables (systemic sepsis, emergency, and patient functional health status). Ascertainment of systemic sepsis was based on existence of ICD-9 codes recorded in the “Patient Problem” list which is updated by members of the medical team. Emergency status was derived from five different EHR fields pertaining to an emergency surgery; the patient was considered “emergency” if any one of the fields was labeled “emergency.” Functional health status was derived from eight separate variables about activities of daily living (ADLs) in the EHR. If independent with all ADLs was answered “yes,” the patient was classified as “independent.” If the patient had deficiencies with all ADLs, they were classified as “totally dependent”; if the patient had deficiencies with some but not all of the ADLs, then they were classified as “partially dependent.” Time stamps for each of the variable values indicating when the item was first entered into the EHR (up to 366 days prior to the date of surgery) were collected to determine if the variable was known at the time of the preoperative encounter.
SURPAS is applicable to both non-emergent and emergent operations. Emergent operations were included even though they might not have a routine preoperative visit, because it was thought that preoperative risks might still be worth calculating and considering for these patients, e.g. if the patient’s risk was very high, the decision might be made not to proceed with emergency surgery but instead offer the patient palliative care. Because emergent surgery was included in SURPAS, it was decided to include them in this study as well.
Data analysis
The study was approved by the Colorado Multiple Institutional Review Board. A frequency distribution of key ACS NSQIP variables was performed to characterize the sample and show the distributions of the eight SURPAS preoperative predictor variables in the study sample. The same was performed for the key EHR variables considered for potential use to populate the eight SURPAS preoperative predictor variables to show the completeness and range of values for these variables in the EHR.
For the categorical SURPAS preoperative predictor variables, agreement between the ACS NSQIP values and the EHR-derived values was measured using a Kappa statistic; for the continuous variables (age and wRVU), agreement was measured using a Pearson correlation coefficient.
Availability of each variable in the EHR at the time of the patient’s preoperative encounter was assessed as follows. For each patient, the “preoperative encounter” in the EHR was defined as the last clinic visit prior to surgery with the operative surgeon listed as provider. If a preoperative clinic visit did not exist in the EHR, a note in the patient’s EHR signed by the operative surgeon prior to surgery was used. Time stamps for each of the EHR-derived SURPAS predictor variables were then used to calculate the time when each variable was entered into the EHR in relation to the preoperative visit or note. An EHR-derived SURPAS predictor variable was considered as potentially useful for SURPAS if: (1) the Kappa statistic or the Pearson correlation coefficient was ≥0.80 and (2) the EHR-derived value was available at the preoperative encounter ≥95% of the time. All analyses were performed using SAS software, version 9.4 (SAS, Inc., Cary, NC).
Results
A total of 5294 patients were entered into the ACS NSQIP database from UCH between 4 July 2013 and 20 January 2016. Patients were excluded if MRN could not be matched between NSQIP and EHR records (n = 58), the patient had a procedure not targeted by the NSQIP Essentials Program and SURPAS (n = 2), or if there were missing data for key NSQIP variables (n = 29). The analytic cohort contained 5205 patients (98.3% of the total of 5294 patients).
Key ACS NSQIP variables for the cohort are presented in supplemental Table 1. Supplemental Table 2 presents frequency distributions for the SURPAS preoperative predictor variables as obtained from the EHR. The EHR indicated a slightly lower percentage of inpatient operations (58.7% versus 60.7%), a slightly higher percentage of emergency operations (7.4% versus 6.2%), a much lower mean wRVU (10.6 ± 5.0 versus 17.1 ± 10.2), and many fewer patients with sepsis (131, 2.5% versus 291, 5.6%), compared to the UCH NSQIP data. The EHR indicated that 25 cases were performed by the transplant surgery service, a specialty not included in the ACS NSQIP. Preoperative sepsis was indicated in 131 patients by 30 different ICD-9 or 31 Healthcare Common Procedure Coding System codes, which includes CPT codes. A CPT code for the primary operation was available in the EHR for only 2150 patients (41.3% of the 5205 total patients; 1989 outpatients and 161 inpatients); a CPT code was not available for 3055 patients (58.7% of the 5205 patients; 2998 inpatients and 57 outpatients). This likely accounts for the mean wRVU being lower in the EHR data compared to the UCH NSQIP data, as most of the CPT codes available in the EHR were for outpatient operations.
A comparison of the EHR to the UCH NSQIP data for the predictor variables is provided in Table 1. For each variable, or category within the variable, the numbers in the first column are the number of cases in the NSQIP dataset; the numbers in the second and third columns are the numbers of cases that are equal to and not equal to the EHR-derived values. So, the percentages in the third column show for which variables, or categories of variables, there is more or less agreement between the NSQIP and EHR data. For example, there is more agreement when a patient’s ASA class is III, versus I–II or IV–V; there is more agreement in the non-emergency cases versus the emergency cases, etc. Six of the SURPAS preoperative predictor variables had acceptable Kappa statistics or Pearson correlation coefficients (≥0.80) between the UCH NSQIP data and the EHR data (Table 1): Age, 0.996; primary surgeon specialty, 0.989; ASA class, 0.977; inpatient/outpatient operation, 0.914; emergency operation, 0.839; and wRVU, 0.806. Two of the SURPAS preoperative predictor variables did not have an acceptable Kappa statistic: functional health status, 0.487 and preoperative systemic sepsis, 0.283.
Comparison of the EHR to the UCH NSQIP data for the predictor variables.
ASA class: American Society of Anesthesiology physical status classification; CI: confidence interval; EHR: electronic health record; NSQIP: National Surgical Quality Improvement Program; UCH: University of Colorado Hospital.
aKappa statistic was calculated using the non-missing data or excluding levels not found in the UCH NSQIP data.
bWork Relative Value Unit was obtained from the current procedure terminology (CPT) codes through a look-up table. A Pearson correlation coefficient was used and was only performed when values were present from both datasets.
First appearance of EHR variables in relationship to preoperative visit (n = 5205).
ASA class: American Society of Anesthesiology physical status classification; EHR: electronic health record; wRVU: work Relative Value Units.
aA total of 463 (8.9%) of the patients did not have a preoperative visit date.
bWork Relative Value Unit was obtained from the current procedure terminology (CPT) codes through a look-up table.
A total of 3981 (75.2%) patients had a preoperative encounter (as denoted by a note within the EHR) within 0–30 days prior to surgery, 842 (15.9%) patients had a preoperative encounter >30 days before the day of surgery, and 471 (8.9%) patients did not have a preoperative encounter recorded in the EHR. The patients with and without a preoperative encounter in the EHR were compared (data not shown). The patients without a preoperative encounter by the operative surgeon in the EHR tended to be younger, Hispanic, uninsured, an emergency case, inpatient, transferred from another acute care hospital, undergoing a general surgical, vascular, or neurosurgical operation, with systemic sepsis, and with a higher ASA class and more comorbidities.
Table 2 presents the first appearance of each of the SURPAS predictor variables in the EHR in relation to the patient’s preoperative encounter. There was a clear pattern in the table that only two variables—age and surgeon specialty—had a significant majority of their values available at the time of the preoperative encounter. ASA class was typically available in the EHR during or after the time of the surgical encounter, as it is entered into the anesthesiology team’s documentation of the operation. At our institution, wRVU, derived from the CPT code, is variably available in the EHR after the documentation of the operation is entered by the surgeon. Surrogates of functional health status, as recorded by eight variables about ADLs, are usually entered by nursing staff at various encounters.
Table 3 summarizes the Kappa statistic or Pearson correlation coefficient between the NSQIP and EHR values for each SURPAS predictor variable, and the percent of cases which had the EHR variable available before the preoperative encounter. Only two variables met our criteria of both a Kappa statistic or Pearson correlation coefficient ≥0.80 and at least 95% of the cases having the EHR variable available before the preoperative encounter—age (0.996 and 95.1% of the cases having the EHR variable available before the preoperative encounter) and primary surgeon specialty (0.989 and 99.8% of the cases).
Summary of Kappa statistic/Pearson correlations and percent of cases having the EHR variables before the preoperative visit for each of the SURPAS predictor variables.
ASA class: American Society of Anesthesiology physical status classification; EHR: electronic health record; NSQIP: National Surgical Quality Improvement Program; UCH: University of Colorado Hospital; wRVU: work Relative Value Units.
For all variables the numerator is the number of patients who have the variable reported in the EHR before the preoperative encounter documentation. For age, ASA class, work RVU, inpatient/outpatient and primary surgeon specialty, the denominator is 5205–463 = 4742 patients who had a preoperative encounter documentation. For systemic sepsis, emergency operation, functional health status prior to surgery, the denominator is the total number of patients reported with systemic sepsis, emergency operation, and functional health status prior to surgery.
aWork Relative Value Unit was obtained from the CPT codes through a look-up table.
Discussion
The purpose of this study was to determine if the eight SURPAS predictor variables could be automatically populated from the patient’s EHR at the preoperative encounter in order to make SURPAS easier for surgeons to use, potentially increasing their use of SURPAS. Local EHR structured data were analyzed to determine their availability at the preoperative encounter and their agreement with corresponding UCH NSQIP values (used as the “gold standard”). Of the eight variables required to predict surgical outcomes, two did not agree well with UCH NSQIP values (presence of sepsis and functional health status), four agreed adequately with UCH NSQIP values but were not reliably populated in the EHR at the preoperative encounter (ASA class, CPT code of the primary operation used to determine wRVU by look-up table, in- versus outpatient procedure, and emergency status), and two (patient age and primary surgeon specialty) had high agreement with UCH NSQIP values and were reliably populated in the EHR at the time of the preoperative encounter. Based on these findings, in order for SURPAS to be used for predicting postoperative adverse outcomes, six variables (presence of sepsis, functional health status, ASA class, CPT code of the primary operation, in-/outpatient procedure, and emergency status) will need to be entered by the provider team at the preoperative encounter.
It is not surprising that some of the variables had very high Kappa statistics/Pearson correlations because they are found in one field in the EHR which is probably used by the NSQIP nurse to populate the same fields in the NSQIP database: age, 0.996; surgeon specialty, 0.989; ASA class, 0.977; and in-/outpatient, 0.914. Two variables had lower but possibly still acceptable Kappa statistics/Pearson correlations: emergency, 0.839; and wRVU, 0.806. The Kappa statistic for emergency was probably lower because a patient’s classification of “emergency” was found in five different fields in the EHR, and they were not always in agreement. The correlation for wRVU was perhaps lower because many CPT codes were missing from the EHR, particularly for inpatient operations, and perhaps those data are not highly reliable in the EHR and the NSQIP nurses used other sources for that information—at our institution CPT codes are generally coded for billing outside of the EHR and not reliably documented in the EHR at the time of the operation. The remaining two variables had unacceptably low Kappa statistics (functional health status, 0.487; and preoperative sepsis, 0.283), probably due to the NSQIP nurse relying on other sources for the information, such as clinic notes from the surgical team or other providers. These two variables were not single fields in the EHR, but had to be “derived” from diagnosis codes (sepsis) or ADL fields from nursing notes (patient functional health status). If the Kappa statistics measuring agreement between the “EHR derived” and NSQIP variables had been high (≥0.80), the “derived” variables could have been used to populate the SURPAS variables, probably with minimal effect on the prediction. However, since the Kappa statistics were low, these variables will need to be put into SURPAS by the healthcare providers.
The six variables that need to be entered into SURPAS are likely fairly easily determined by a medical provider. Functional health status may be obtained at the preoperative encounter during the history and physical examination by asking the patient and/or family a few standardized questions about the patient’s ability to engage in ADLs, or by reviewing other providers’ notes (e.g. primary care physician, physical therapy, or occupational therapy notes). ASA class can likely be accurately estimated by non-anesthesiologist care providers by establishing brief guidelines for them derived from consultation with a focus group of anesthesiologists. The surgical team should be able to rapidly and accurately assess in-/outpatient and emergency operation status. wRVU can be determined by a table look-up within the SURPAS software based on the CPT code and name of the planned primary operation. To facilitate determination of CPT code, the SURPAS tool provides a list of CPT codes and operation names in response to entry of a generic operation name by the surgical team, ordered by the frequency of each CPT code in the large, national ACS NSQIP database. Determining if a patient is suffering from systemic sepsis within 48 h prior to the operation appears more challenging. The low Kappa statistic (0.283) indicates that this variable is difficult to assess, particularly from existing ICD-9 codes in the EHR, and is reliant on the history, physical examination, and additional work-up. In future generations of SURPAS, we will consider eliminating or replacing this variable with another predictor variable that is easier to ascertain, such as CPT-specific risk. Consideration could be given to training a non-surgeon (e.g. Nurse Practitioner, Physician Assistant, Medical Assistant, trainee) to obtain and enter the six variables into SURPAS prior to the preoperative encounter, so that the surgeon can spend her/his time discussing the operation and output from the SURPAS report.
SURPAS is now integrated into the UCHealth system EHR and has been used in over 3000 patients. Several pilot studies have been completed to gather feedback from surgeons and patients, and have been published or are in progress. 10 The incorporation of automated integration of variables into the SURPAS input is aimed at decreasing the burden of data collection by the user, to facilitate rapid use of this clinical decision support tool. SURPAS also produces an automatic preoperative note written to the patient’s EHR record and a printed graphical display of the patient’s risks compared to national averages for the patients to take home. This is in contrast to our experience using the online ACS Surgical Risk Calculator, 11 which calls for the entry of 21 predictor variables, does produce patient risks compared to the national averages, but does not produce an automatic preoperative note written to the patient’s EHR record nor a printed graphical display for the patients to take home. Although the ACS Surgical Risk Calculator uses default values when predictor variables are missing, the effect of missing values of predictor variables on the accuracy of the prediction is unknown.
This study has several strengths and limitations. It is based on parsimonious prediction models developed from the national ACS NSQIP database and is one of the first attempts to study the quality and usability of data automatically extracted from the EHR in order to optimize design of a multi-outcome risk model. Limitations include: (1) non-use of ICD-10 codes might have affected the analysis of the sepsis variable, although they officially started on 1 October 2015, were slow to be implemented at UCH, and the last of the operations in this study were performed on 20 January 2016, so the effect is probably minimal. (2) Although SURPAS has desirable qualities, it is still only currently functional within one healthcare system. The interactive software was developed by a vendor contracted by UCHealth (Agile MD, San Francisco, CA). (3) Medical centers may differ in the maturity of their EHR. Analysis of a more mature EHR, such as that of the Veterans Health Administration or the Kaiser healthcare system, and particularly in a HMO-type setting where there are considerable outpatient data on patients before they undergo surgery, might identify more structured variables that can be automatically obtained from an organization’s EHR. These system-based differences highlight the importance of local data quality validation for implementations relying on automatically extracted clinical data. 12
Conclusion
The EHR provides limited useful surgical risk-predictive data at the preoperative encounter for the SURPAS tool. Further pilot testing of the SURPAS tool will be needed to determine if entry of the six variables by the care provider and discussion of the SURPAS risk estimates with patients and families are feasible to include in the workflow at the preoperative encounter.
Supplemental Material
CRI876489 Supplemental Material - Supplemental material for The Surgical Risk Preoperative Assessment System: Determining which predictor variables can be automatically obtained from the electronic health record
Supplemental material, CRI876489 Supplemental Material for The Surgical Risk Preoperative Assessment System: Determining which predictor variables can be automatically obtained from the electronic health record by Robert A Meguid, Michael R Bronsert, Karl E Hammermeister, David P Kao, Anne Lambert-Kerzner, Jacob A Sinex, Jody M Myers and William G Henderson in Journal of Patient Safety and Risk Management
Footnotes
Authors’ contributions
RAM, AL-K, KEH, WGH, DPK, and MRB guided the overall project. RAM, KEH, WGH, and MRB conducted the quantitative data analyses that provided the background data and the creation of SURPAS. JAS and JMM provided access to and information on architecture of UCHealth Epic data. WGH and RAM were major contributors in writing the manuscript and making revisions with support from KEH, DPK, and AL-K. All authors read and approved the final manuscript.
Availability of data and material
The datasets used and analyzed during the current study are the ACS NSQIP PUF. These data are the property of the American College of Surgeons, and are freely available to faculty and staff at institutions participating in the ACS NSQIP.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by a grant from the Agency for Healthcare Research and Quality (AHRQ) grant 1R21HS024124-02 and 1R21HS024124-01. The Agency for Healthcare Research and Quality had no role in the design of the study and collection, analysis, and interpretation of data or in writing the manuscript. The authors do not derive any financial gain from SURPAS.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
