Abstract
Abbreviations
spinal cord injury pressure ulcer Spinal Cord Injury Pressure Ulcer Scale electronic health record receiver operating characteristic area under the curve positive predictive value negative predictive value Rehabilitation Engineering Research Center University of Pittsburgh Medical Center
Introduction
Pressure ulcers (PrUs) continue to cause various physical, psychological, and economical burdens for individuals with spinal cord injury (SCI). They are one of the most common secondary complications following SCI (Salzberg, 1998). Approximately 34% of individuals with SCI develop at least one PrU in their initial acute hospitalization or inpatient rehabilitation (The Annual Statistical Report for the Model Spinal Cord Injury Care Systems., 2006). With over 200 known risk factors (Byrne, 1996) and treatment costing up to $11 billion annually in the United States (“National Pressure Ulcer Advisory Panel”, 2014), PrU prevention is extremely complex and important. Proper identification of individuals with SCI at risk for PrU development and initiation of appropriate preventive practices are necessary to decrease PrU incidence and prevalence (Ayello, 2002). The evaluation of the psychometric priorities of SCI specific risk assessment scales is one of the top priorities ofthe Veterans Affair’s SCI Quality Enhancement Research Initiative (Henzel, 2011). To identify PrUs at their incipient stage, clinicians need risk assessment tools that are standardized in acute and post-acute facilities.
More than 38 scales have been developed to quantify PrU risk across a broad range of populations, although these tools are not always representative of PrU risk and development in the SCI population (Mortenson, 2008). The most studied and utilized scales such as the Braden, the Norton and the Waterlow have been validated for use in hospital settings and nursing homes, but do not account for the SCI-specific risk factors associated with the loss of mobility and sensation below the level of lesion (DeJong, 2014; Mortenson, 2008). Individuals with SCI are at high risk for PrU development and validation of an appropriate risk assessment scale is crucial (Henzel, 2011). The Pressure Ulcer Assessment Scale for Spinal Cord Injured (referred to as Salzberg Scale or SCIPUS) is a 15-item risk assessment scale created to specifically evaluate risk of pressure ulcer development in the SCI population (Salzberg, 1996). The total scores range from 0 (no risk) to 25, and the original study (Salzberg, 1996; Salzberg, 1999) determined scores >6 as the optimum balance between sensitivity and specificity. Scores between 6 to 8 were categorized as high-risk and scores≥9 were determined as very high-risk category. Although the SCIPUS is customized for the assessment of SCI-specific risk factors, reliability and validity of the scale is not well established (Henzel, 2011; Mortenson, 2008). The objective of this study was to evaluate the predictive validity of the SCIPUS in newly injured individuals with traumatic SCI. The first aim was to determine the optimal risk cutoff scores of the SCIPUS during acute care hospitalization and inpatient rehabilitation to accurately assess risk for varying time points to PrU development. The second aim was to evaluate the psychometric properties of the scale in both settings for the varying time points using the new cutoff scores by computing the sensitivity, specificity and area under the curve (AUC) for the receiver operating characteristic (ROC).
Methods
Study design
This study is a secondary analysis, using an existing database of the Rehabilitation Engineering Research Center (RERC) on SCI. The protocol received approval from the appropriate Institutional Review Board for Human Subjects Research.
Inclusion and exclusion criteria
The Clinical Core of the RERC on SCI collaborated with University of Pittsburgh Medical Center (UPMC) and the University of Pittsburgh Model System on SCI to recruit, screen, and enroll individuals with new, traumatic SCI within 24-72 hours of admission. Demographic and medical information, along with development and stage of PrU outcomes were collected from acute care and inpatient rehabilitation settings. Participants were enrolled from 2008 to 2012. Data were collected on enrolled participants three times per week during acute care hospitalization and once per week during inpatient rehabilitation.
Individuals were eligible to participate in the original study if they were 18 years or older and received acute care treatment at UPMC. Individuals excluded from the RERC study were those with pre-existing diseases (such as autoimmune or demyelinatingdiseases) affecting the inflammatory response to SCI, as well as those with previous SCI and other neurological diseases that affected the motor and sensory function. Data were extracted for this secondary analysis from those participants who did not voluntarily withdraw from original RERC study, and for whom complete information required for the SCIPUS components could be extracted. Based on these criteria, 34 individuals were included for this study. Three individuals were in both the acute and inpatient settings, whereas the others were in either acute hospitalization (n = 20) or inpatient rehabilitation settings (n = 11) – Fig. 1.
Data collection
The SCIPUS consists of 15 items including age, tobacco use or cigarette smoking, residency status (nursing home or hospital), level of activity, degree of mobility, completeness of SCI, urinary incontinence, autonomic dysreflexia, diabetes, comorbidities such as cardiac, pulmonary, renal disease, impaired cognition, hypoalbuminemia and low hematocrit. These 15 variables were extracted along with demographic information, and PrU incidence time points and setting (acute vs. inpatient). The severity of PrUs were noted according to the NPUAP staging and guidelines (Black, 2007). Hematocrit, and albumin levels were retrieved from the electronic health records (EHR), to calculate the SCIPUS score, as they were not previously recorded in the primary RERC study. Access to this information could be retrieved only from the EHR for individuals who did not withdraw from the RERC study. The extracted variables were recorded for each subject assessment. Data for the SCIPUS items for each assessment were recorded based on the expert opinion of one of two nurses who performed assessments and reviewed EHR notation and assessment notes. Once all items were recorded, a total SCIPUS score was calculated.
Acute care hospitalization
This analysis evaluated the predictive validity of the SCIPUS during acute care hospitalization. The initial visit would most likely estimate the development of the PrU upon the following visit, PrU development was evaluated on the consecutive assessment – 2 to 3 days after initial compilation of the SCIPUS score. Eighteen included individuals had a range of one to four assessments each that contributed to 30 encounters for analysis. Evaluating the occurrence of PrUs and administrating a risk assessment scale are usually based on the severity and stability of the individual’s condition, and the care setting the individual is in (Braden, 2005). Since there is no definite consensus (Barton, 2009) on the frequency of administration of a PrU risk assessment tool for individuals in different care settings, a PrU assessment that was 5 to 7 days after the initial SCIPUS administration was also considered to assess the predictive validity of the scale. Twenty-three included individuals with a range of one to seven assessments each contributed 45 encounters. The time points – 2 to 3 days and 5 to 7 days after SCIPUS administration – were chosen to evaluate the predictive validity of the scale since individuals were assessed three times a week during acute care hospitalization as part of the primary study (RERC) protocol. A total of 23 individuals were included in evaluation of the predictive validity of SCIPUS during acute care hospitalization (Fig. 1).
Inpatient rehabilitation
This analysis evaluated the predictive validity of the SCIPUS during inpatient rehabilitation. During inpatient rehabilitation, the individual’s PrU assessment would range from 5 to 7 days after the initial SCIPUS. Twelve included individuals had a range of one to six assessments each that contributed 25 encounters for analysis. Since there is no definite consensus (Barton, 2009) on the frequency of administration of a PrU risk assessment tool for individuals in different care settings, individual’s PrU assessment that was 14 to 21 days after the initial SCIPUS administration, was also considered to calculate the predictive validity of the scale. Fourteen individuals with a range of one to seven assessments contributed 32 encounters. The time points – 5 to 7 days and 14 to 21 days after SCIPUS administration – were chosen to evaluate the predictive validity of the scale since individuals were assessed once a week during inpatient rehabilitation as part of the primary study (RERC) protocol. A total of 14 individuals were included in evaluation of the predictive validity of SCIPUS during inpatient rehabilitation (Fig. 1).
Data analyses
Four separate analyses were performed to evaluate the predictive validity of SCIPUS – two for individuals in acute care hospitalization (one with a skin assessment 2 to 3 days after administration of SCIPUS and one with the assessment 5 to 7 days after administration of SCIPUS) and two for individuals in inpatient rehabilitation (one with skin assessment 5 to 7 days after administration of SCIPUS and one with the assessment 14 to 21 days after administration of SCIPUS). Box plots were used to compare differences in the SCIPUS score at initial assessment in individuals with a new PrUs and without a PrUs at the respective time points. Optimal cut-off scores were calculated using the sensitivity and specificityfor individuals in both the settings for the respective time points. The psychometric properties evaluated were: sensitivity – probability that the test will be positive for individuals who developed PrUs or true positive, specificity – probability that the test will be negative for individuals who did not develop PrUs or true negative, positive predictive value (PPV) – the percentage of individuals above the optimal cutoff scores who developed PrUs, and negative predictive value (NPV) – the percentage of individuals who did not develop PrUs below the optimal cutoff scores. A receiver operating characteristic (ROC) curve was constructed to measure the area under the curve (AUC) – the strength of the SCIPUS’s prognostic validity. AUC of 1.0 indicates a perfect tool, between 0.7 and 0.9 indicates a good to moderate tool and <0.6 indicates less accurate or not informative (Zweig, 1993).
Results
Acute care hospitalization
The demographic information of this group is summarized in Table 1. Eighteen individuals with 30 encounters had skin assessments within 2 to 3 days of SCIPUS administration. Two out of the 30 encounters (6.6%) had an incidence of PrU. The average SCIPUS score for encounters with PrU development compared to no PrU at the 2 to 3 day assessment was 17.5 ± 2.1 vs. 13.0 ± 3.6, p > 0.05 (Fig. 2a). Twenty-three individuals with 45 encounters had skin assessments within 5 to 7 days of SCIPUS administration. Six out of the 45 encounters (13.3%) had an incidence of PrU. The average SCIPUS score for encounters with PrU development compared to no PrU after 5 to 7 day assessment was14.6 ± 3.7 vs. 13.4 ± 3.5, p > 0.05 (Fig. 2a).
Psychometric properties evaluation for individuals in acute care hospitalization
The SCIPUS cut off score of 15 represents best balance of sensitivity (100%) and specificity (75%), with 22.2% PPV and 4% NPV for the 2 to 3 day follow-up skin assessment. This is a nine-point increase from the existing cutoff score at 6 in Salzberg’s original study that determined the high-risk categories. An AUC of 0.85 implies that the SCIPUS is a strong predictor of PrU development using 15 as the discriminating cutoff score between individuals who develop PrUs and those who do not in acute hospitalization (Fig. 3). The SCIPUS’s cut off score of 13 represents 50% sensitivity and 51.3% specificity with 13.6% PPV and 86.9% NPV for the 5 to 7 day assessment. This is a seven-point increase from the originally proposed cutoff score of 6 that determined the high-risk categories. An AUC of 0.59 implies that the SCIPUS is a not a strong predictor of PrU development for this time period.
Inpatient rehabilitation
The demographic information of this group is summarized in Table 2. Twelve individuals with 25 encounters had skin assessments within 5 to 7 days of SCIPUS administration during inpatient rehabilitation. Three out of the 25 (12%) encounters had an incidence of a PrU. The average SCIPUS score for encounters with PrU development compared to no PrU at the 5 to 7 day assessment was 9.6 ± 0.5 vs. 9.9 ± 2.6, p > 0.05 (Fig. 2b). Fourteen individuals with 32 encounters had skin assessments within 14 to 21 days of SCIPUS administration during inpatient rehabilitation. Three out of the 32 encounters (9.4%) had an incidence of PrU. The average SCIPUS score for encounters with PrU development compared to no PrU at the 14 to 21 day assessment was 9.6 ± 0.5 vs. 10.9 ± 3.1 in inpatient rehabilitation, p > 0.05 (Fig. 2b).
Psychometric properties evaluation for individuals in inpatient rehabilitation
The SCIPUS’s cut off score of 9 represents optimal balance of sensitivity (66.7%) and specificity (45.5%), with 14.3% PPV and 0.7% NPV for the 5 to 7 day follow-up skin assessment. The ROC curve showed an AUC of 0.49 indicating the SCIPUS to be completely ineffective in predicting development of PrU for this time period. The SCIPUS’s cut off score of 9 represents 66.7% sensitivity and 37.9% specificity with 10% PPV and 91.7% NPV for the 14 to 21 day assessment. The high NPV score show the scale’s strong ability to decipher that an individual who scored below the determined cut-off score would not develop a new PrU. An AUC of 0.59 implies that the SCIPUS is a not a strong predictor of PrU development for this timeperiod.
Discussion
Although the SCIPUS was developed to predict PrU risk for individuals with SCI, few studies have reported the validity of the scale (Henzel, 2011; Mortenson, 2008). In this study, the scale was tested for varying psychometric properties of time until PrU assessment during acute care hospitalization and inpatient rehabilitation. The results of this study suggest that the SCIPUS is able to predict the risk of PrU development with the optimal cut-off score of 15 in new, traumatic SCI population in acute hospitalization within 2 to 3 days of pressure ulcer occurrence. Salzberg’s original study recommended cut-off scores for a population specifically focused on outpatient care of >6 points for high risk (Salzberg, 1998; Salzberg, 1996), whereas in this study of individuals with new, traumatic SCI, the cut off scores ranged from 9 to15 points across both settings depending on the follow-up assessment time point. Using the original cut-off scores, we determined that the balance in sensitivity and specificity was not optimal. Although individuals with PrUs had a higher SCIPUS score compared to no PrUs (AUC = 0.8) during acute hospitalization at the 2 to 3 day follow-up skin assessment, the scale was not able to predict pressure ulceration (AUC < 0.6) at the 5 to 7 day assessment. SCIPUS administration within 2 to 3 days of PrU development appears to more accurately predict PrU risk in acute hospitalization, possibly due to rapid changes in health as measured by the tool. The scale was also not able to predict pressure ulceration (AUC < 0.6) in individuals during inpatient rehabilitation at the 5 to 7 day, and 14 to 21 day assessment. A few weeks between assessments may have weakened the association of SCIPUS factors and risk to develop PrUs due to confounding variables such as preventive techniques in the hospital or health improvement over time. A recent study reported limited ability of the scale to predict risk to develop PrU during inpatient rehabilitation in individuals with SCI (Delparte, 2015). In that study after administrating the SCIPUS, PrU incidence anytime during the inpatient stay (approximately 95 days for individuals with PrU) was utilized to evaluate the psychometric properties of the scale.
In Salzberg’s original study, the balance of sensitivity and specificity had a higher specificity score in initial hospitalization of 84.2% as compared to sensitivity of 36.8%, which is not favored since it is better to choose a cutoff with higher sensitivity than specificity so that the risk of developing a PrU is not missed (Fletcher, 2012; Mortenson, 2008; Salzberg, 1998; Salzberg, 1999). In this study, the optimal cutoff scores for all time point assessments had higher sensitivity scores; as it is essential not to miss the presence or risk of PrU development (Fletcher, 2012; Källman, 2014).The overall higher cutoff score during acute hospitalization may suggest that these individuals are at a higher risk and are more susceptible to PrU development and may need different weights on risk factors to differentiate from other settings. Inconsistencies in the cutoff scores between the original study and this study could be due to the time since injury, as this can impact the occurrence of secondary complications. The time since injury for Salzberg’s original sample population of 219 individuals was 17.2 ± 12.1 years (Salzberg, 1996); on the other hand in this study, the time since injury for the cohort of 23 individuals in acute care and 14 individuals in inpatient rehabilitation, with new SCI was 30 ± 41 days and 37 ± 18 days, respectively. Following traumatic SCI, a majority of individuals in acute hospitalization are confined to bed or on mechanical ventilation, which reduces their activity and mobility that in turn inflates their SCIPUS score by nearly seven points. As for the population of Salzberg’s original study, only two individuals were confined to a bed and those temporarily hospitalized used mobility and activity data from a baseline state (Salzberg, 1998). Three percent of individuals were in a nursing home or hospital at the time of assessment in Salzberg’s original study, in contrast the population in this study were in a hospital setting (acute or inpatient) that added two points to their SCIPUS score, essentially shifting the average score to a higher cutoff.
Low PPV throughout all the assessments across acute care hospitalization and inpatient rehabilitation can be attributed to the low incidence of PrU development in each category (Portney, 2000). When screening a population with very low incidence, it is inevitable that many people with positive test results will in fact be false positives or a Type II error (Altman, 1994). The incidence of PrU in this study population was 9% to 13%, which is not representative of the general population trends (Krause, 2004). If PrU incidence in this cohort had been higher, it is likely that the PPV would have been higher and the NPV would have been lower. In addition, low PPV may be associated with the effect of preventative measures such as support surface technology, skin assessment or tissue integrity assessment, or pressure-redistribution turning schedules used while individuals are being treated in these settings. Also, the level of a patient’s health may have improved during their stay in the hospital or inpatient facility, which led to decreased risk in development of PrUs (Fletcher, 2012).
Limitations and future work
This is a retrospective study with variability in assessment follow up dates and limited sample size. Due to the limited sample size, all the assessment pairs (SCIPUS score and follow-up PrU assessment) were taken into consideration regardless of whether the encounters came from the same individual, to augment the size of the sample. The same individuals were also included in the analyses for different assessment time points due to multiple assessment encounters. This may have biased the results of the study. Also, the nurses determined the SCIPUS components retrospectively by evaluating the EHR, and this may have led to potential errors in spite of their expertise. A prospective method with consistent follow up assessments may have resulted in more precise scores to for the level of mobility, activity, incontinence, and moisture control rather than relying on notes pertaining to catheterization changes and the subjectivity of nurses to interpret the EHR. Also, the scale could not be evaluated for its ability to predict PrUs following 2 to 3 days of administration during inpatient rehabilitation since the assessments in this setting were performed weekly in the primary study protocol. This follow-up PrU assessment period can be implemented in future in inpatient rehabilitation studies to evaluate the scale’s predictive validity for this time period. In addition, the scale’s weighted risk factors could be modified to accommodate the drastic changes in conditions in individuals with acute SCI, compared to chronic injury.
Conclusion
In this study, higher cut-off scores were obtained for individuals in acute care and inpatient rehabilitation to determine risk for PrU development as compared to Salzberg’s original study. The findings of this study suggest that SCIPUS is most optimal to measure the PrU risk following new, traumatic SCI when used in acute-care hospitalization every 2 to 3 days for reassessment. The results of this study show evidence of weak predictive validity of the scale to predict risk of pressure ulceration over longer time periods.
Conflict of interest
The authors declare that no conflicts exist in connection with this paper.
Footnotes
Acknowledgments
Supported primarily by the National Institute on Disability and Rehabilitation Research (NIDRR), Rehabilitation Engineering Research Center on Spinal Cord Injury (grant no. H133E070024); and by the NIDRR (grant no. H133P110012) and the U.S. Agency for Healthcare Research and Quality (grant no. 1R24HS022134).